Databricks Simplifies S3 Data Connection with Delegated IAM Permissions
Databricks has introduced a streamlined, automated flow for connecting Amazon S3 buckets to Unity Catalog, reducing manual configuration from days to minutes. This new method uses AWS IAM temporary delegation, eliminating the need for complex IAM policies and CloudFormation templates, and directly benefiting users by simplifying a foundational step for data ingestion, pipelines, and analytics. The process is now a few clicks within the Databricks workspace, automatically provisioning necessary storage credentials and external locations with least-privilege permissions. Users lacking sufficient AWS access can request it from their administrator directly within the flow, and the authorization automatically expires after provisioning, enhancing security.
- →Automated S3 external location creation with IAM delegation
- →Enhanced security with temporary, expiring IAM authorization
- →Streamlined user experience for S3 connectivity
- →Enables LTAP architecture by simplifying cloud storage integration
Security (1) ›
- Enhanced security with temporary, expiring IAM authorization
Databricks uses AWS IAM temporary delegation, meaning the authorization granted to Databricks to provision resources on your behalf is time-bounded and automatically expires. The system ensures Databricks never holds standing access to your AWS account, and all provisioning actions are logged in AWS CloudTrail for auditing. This approach aligns with enterprise security standards by creating least-privilege IAM roles.
Features (1) ›
- Automated S3 external location creation with IAM delegation
Databricks now offers a simplified, automated workflow for creating external locations to connect Amazon S3 buckets to Unity Catalog. This new flow leverages AWS IAM temporary delegation, reducing the setup complexity from writing 140-line IAM policies and configuring multiple AWS resources to a few clicks within the Databricks workspace. It automatically handles storage credential provisioning with least-privilege IAM roles and configures the external location, enabling Auto Loader and File Events.
Enhancements (1) ›
- Streamlined user experience for S3 connectivity
The update significantly simplifies connecting S3 data to Databricks, which is crucial for ingestion, pipelines, analytics, and the emerging LTAP architecture. Users previously faced a complex, multi-console setup process; this change allows them to set up secure S3 connections in minutes. A new flow within Catalog Explorer guides users through the process, including the option to request necessary AWS permissions from administrators if they lack them.
Notes (1) ›
- Enables LTAP architecture by simplifying cloud storage integration
The simplified S3 connectivity removes a key infrastructure barrier for adopting the Lake Transactional/Analytical Processing (LTAP) architecture. This allows operational data in cloud storage to be instantly queryable by analytical engines without requiring separate pipelines or complex data replication.
https://www.databricks.com/blog/connect-amazon-s3-data-databricks-delegated-iam-permissions
Related releases
- Building an AI-Forward Healthcare Organization on Data and Governance Databricks Blog ·
- Databricks Accelerates Agentic Media Buying with New Reference Implementation Databricks Blog ·
- NBCUniversal Migrates to Databricks Lakehouse for Scalable Analytics Databricks Blog ·
- Databricks SDK for Python v0.123.0: API changes and breaking updates Databricks Python SDK Releases ·
- Databricks Go SDK v0.167.0 Adds Cloud Provider Auth Fields Databricks Go SDK Releases ·
- Databricks SDK Java v0.138.0 Adds Model Provider Config Fields Databricks Java SDK Releases ·