Databricks on AWS
Partner Solution Deployment Guide
July 2023
Denis Dubeau, Databricks
Shivansh Singh, AWS Quick Start team
| Refer to the GitHub repository to view source files, report bugs, submit feature ideas, and post feedback about this Partner Solution. To comment on the documentation, refer to Feedback. |
This Partner Solution was created by Databricks in collaboration with Amazon Web Services (AWS). Partner Solutions are automated reference deployments that help people deploy popular technologies on AWS according to AWS best practices. If you’re unfamiliar with AWS Partner Solutions, refer to the AWS Partner Solution General Information Guide.
Overview
This Quick Start reference deployment guide provides step-by-step instructions for deploying Databricks workspaces on the AWS Cloud.
Databricks is a unified data-analytics platform for data engineering, machine learning, and collaborative data science. A Data Science & Engineering workspace is a software as a service (SaaS) environment for accessing all your Databricks assets. The workspace organizes objects (that is, notebooks, libraries, and experiments) in directories and provides access to data and computational resources, such as clusters and jobs.
This Quick Start is for IT infrastructure architects, administrators, and DevOps professionals who want to use the Databricks API to create Databricks workspaces on the AWS Cloud. This Quick Start creates a new workspace in your AWS account and sets up the environment for deploying future workspaces.
Costs and licenses
There is no cost to use this Partner Solution, but you will be billed for any AWS services or resources that this Partner Solution deploys. For more information, refer to the AWS Partner Solution General Information Guide.
For cost estimates, see the Databricks AWS Pricing page for product tiers and features.
To launch the Quick Start, you need the following:
-
An AWS account. If you don’t have an account, sign up at https://aws.amazon.com.
-
A Databricks account ID. If you don’t have an account, Contact your Databricks representative to sign up.
Determine if your workspace has the following features enabled. Enabling them requires an account for the E2 version of the platform. If you have questions about availability, Contact your Databricks representative:
-
Customer-managed VPC. Provide your own VPC.
-
Secure cluster connectivity. Network architecture with no VPC open ports and no Databricks runtime worker public IP addresses. In some API operations, this is referred to as No Public IP (NPIP).
-
Customer-managed keys for managed services. To encrypt notebooks in the Databricks-managed control plane, provide keys to AWS Key Management Service (AWS KMS).
Architecture
Deploying this Partner Solution with default parameters builds the following Databricks environment in the AWS Cloud.
As shown in Figure 1, this Partner Solution sets up the following:
-
A highly available architecture that spans at least three Availability Zones.
-
A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.*
-
In the private subnets:
-
Databricks clusters of Amazon Elastic Compute Cloud (Amazon EC2) instances.
-
One or more security groups for secure cluster connectivity.
-
-
In the public subnet:
-
A network address translation (NAT) gateway to allow outbound internet access.
-
-
Amazon CloudWatch for the Databricks workspace instance logs.
-
(Optional) A customer managed AWS KMS key to encrypt notebooks.
-
An Amazon Simple Storage Service (Amazon S3) bucket to store objects, such as cluster logs, notebook revisions, and job results.
-
AWS Security Token Service (AWS STS) for requesting temporary, least-privilege access for users.
-
A VPC endpoint for access to Amazon S3 artifacts and logs.
-
A cross-account AWS Identity and Access Management (IAM) role to deploy clusters in the VPC for the new workspace. Depending on the deployment option you choose, you either use an existing IAM role or create an IAM role during deployment.
* The template that deploys this Partner Solution into an existing VPC skips the components marked by asterisks and prompts you for your existing VPC configuration.
Deployment options
This Partner Solution provides the following deployment options:
Predeployment steps
Prepare your Databricks account
You must have a Databricks E2 account ID to launch this Quick Start. For more information, Contact your Databricks representative.
Deployment steps
-
Sign in to your AWS account, and launch this Partner Solution, as described under Deployment options. The AWS CloudFormation console opens with a prepopulated template.
-
Choose the correct AWS Region, and then choose Next.
-
On the Create stack page, keep the default setting for the template URL, and then choose Next.
-
On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. When you finish reviewing and customizing the parameters, choose Next.
Unless you’re customizing the Partner Solution templates or are instructed otherwise in this guide’s Predeployment section, don’t change the default settings for the following parameters: QSS3BucketName,QSS3BucketRegion, andQSS3KeyPrefix. Changing the values of these parameters will modify code references that point to the Amazon Simple Storage Service (Amazon S3) bucket name and key prefix. For more information, refer to the AWS Partner Solutions Contributor’s Guide. -
On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you finish, choose Next.
-
On the Review page, review and confirm the template settings. Under Capabilities, select all of the check boxes to acknowledge that the template creates AWS Identity and Access Management (IAM) resources that might require the ability to automatically expand macros.
-
Choose Create stack. The stack takes about 15 minutes to deploy.
-
Monitor the stack’s status, and when the status is CREATE_COMPLETE, the Databricks deployment is ready.
-
To view the created resources, choose the Outputs tab.
Postdeployment steps
-
When the status is CREATE_COMPLETE for the AWS CloudFormation stack, check the
WorkspaceStatusoutput key value. It should be RUNNING. For any other value, see Troubleshoot a workspace that failed to deploy. -
Navigate to the workspace URL (for example, deployment-name.cloud.databricks.com), and log in to the web application.
Troubleshooting
For troubleshooting common Partner Solution issues, refer to the AWS Partner Solution General Information Guide and Troubleshooting CloudFormation.
Customer responsibility
After you deploy a Partner Solution, confirm that your resources and services are updated and configured—including any required patches—to meet your security and other needs. For more information, refer to the Shared Responsibility Model.
Feedback
To submit feature ideas and report bugs, use the Issues section of the GitHub repository for this Partner Solution. To submit code, refer to the Partner Solution Contributor’s Guide. To submit feedback on this deployment guide, use the following GitHub links:
Notices
This document is provided for informational purposes only. It represents current AWS product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS products or services, each of which is provided "as is" without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at https://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "as is" basis, without warranties or conditions of any kind, either expressed or implied. Refer to the License for specific language governing permissions and limitations.