In this guide, we'll give you everything you need to get up and running with Dagster Cloud, including:
Creating your Dagster Cloud account
Selecting a deployment type
Deploying your code
Setting up CI/CD
Setting up environment variables and secrets
Let's get started!
Step 1: Create a Dagster Cloud account and organization#
If you don't already have a Dagster Cloud account, sign up for one before continuing.
You'll also be asked to create a Dagster Cloud organization. When your organization is created, a single deployment, named prod, will also be created. Refer to the Managing deployments guide for more information about deployments in Dagster Cloud.
In this step, you'll select the type of deployment you want to use: Serverless or Hybrid.
Type
How it works
May be a good fit if...
Serverless
In a Serverless deployment, Dagster manages the infrastructure needed to run Dagster jobs. This means you don't need to spin up or manage any infrastructure. Refer to the Serverless deployment documentation for more info.
You want to try Dagster Cloud without a lot of time investment
You don't want to focus on infrastructure management
Your Dagster jobs aren't computationally expensive or memory intensive
Hybrid
In a Hybrid deployment, your Dagster code is executed in your infrastructure. The agent runs in your infrastructure, executes your code, and streams metadata about code execution and assets and jobs (for visualization) over HTTPS back to Dagster Cloud. Refer to the Hybrid deployment documentation for more info.
Dagster Cloud supports a variety of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.
You want to use your own infrastructure
You can create and configure an agent in your infrastructure
Select the deployment type you want to use and proceed to the next step. Note: If you find your deployment type isn't right for you, it can be changed at any time.
Now that you've selected a deployment type, the next step is to tell Dagster Cloud the location of your Dagster code. In addition to this, you'll set up Continuous Integration (CI) and Continuous Deployment (CD). Once completed, Branch Deployments and continuous code location deployments will be enabled for your Dagster project.
The steps for accomplishing this vary depending on the deployment type you selected in the previous step:
Click the tab for your deployment type - Serverless or Hybrid - to view what's next.
For Serverless deployments, there are two ways to deploy your code to Dagster Cloud:
Start from a template - Use one of our quickstart templates to get up and running. All templates come with CI/CD already configured and will be cloned to a new GitHub repository.
Import an existing project - Import an existing GitHub repository using our GitHub integration or the dagster-cloud CLI. Note: If using the GitHub integration, Dagster Cloud will automatically set up CI/CD for you.
To import existing Dagster code, you can use Dagster's GitHub app or the dagster-cloud CLI.
Using the GitHub integration
Using the GitHub integration to import an existing GitHub repository also sets up CI/CD for you.
Before you get started, note that the repository must have a dagster_cloud.yaml file in order for Dagster Cloud to deploy it. This file defines the code locations in your Dagster code.
If this file doesn't currently exist, create it in the root of your repository with the following code:
# dagster_cloud.yamllocations:-location_name: my_location_name # the name of the code locationcode_source:package_name: hackernews # the name of the python package associated with the code location
After you've committed the file to the repository, come back to Dagster Cloud to complete the import process:
Click the Import an existing project tab.
Sign in to your GitHub account, if prompted.
In Dagster Cloud, define the following:
Git scope - Select the organization or account that the repository is in.
Using the dagster-cloud CLI requires a recent version of Python 3 and Docker.
To complete this step using the CLI, you can use your own Dagster code or the Dagster starter kit. The starter kit is a template with everything you need to get started using Serverless in Dagster Cloud, including CI/CD configuration and the required dagster_cloud.yaml file.
After you've finished setting up your local project, move on to deploying using the CLI:
To install the dagster-cloud CLI, run:
pip install dagster-cloud
Next, you'll need to authenticate to Dagster Cloud. Run the following command and follow the prompts to log in:
dagster-cloud config setup
After you've successfully authenticated, run the following to deploy your code to Dagster Cloud:
dagster-cloud serverless deploy \
--location-name "<LOCATION_NAME>"\# name of the code loation
--package-name "<PACKAGE_NAME>"\# name of the Python package associated with the code location
To set up Hybrid deployment and deploy your code, you'll need to:
Set up an agent
Configure CI/CD for your project
For most Hybrid deployments - with the exception of those using a local agent - you'll need to create a Docker image containing your Dagster code and then add a code location to Dagster Cloud that references the image.
Step 1: Create a Hybrid agent
The dialog that displays will contain a pre-generated agent token and details about the agents currently supported by Dagster Cloud.
Note: Keep this token handy - you'll need it to set up CI/CD in the next step.
Follow the steps for setting up and deploying your agent:
The dialog will indicate when Dagster Cloud receives an agent heartbeat. Click Continue.
Step 2: Configure CI/CD for your project
Next, you'll configure CI/CD for your Dagster project. How this is accomplished depends on your CI/CD provider:
GitHub - If using GitHub, you can use our GitHub Action workflow to set up CI/CD for your project.
Another CI/CD provider - If you're not using GitHub, you can configure CI/CD using the dagster-cloud CLI.
To set up continuous integration using GitHub Actions, you can use your own Dagster code or the Dagster Cloud Hybrid Quickstart. The quickstart is a template with everything you need to get started using Hybrid deployment in Dagster Cloud.
If not using the template, copy the GitHub workflow files (.github/workflows) from the Hybrid quickstart repository and add them to your repository. This is already done for you if using the quickstart.
In the repository, set the necessary GitHub action secrets:
DAGSTER_CLOUD_API_TOKEN - The Dagster Cloud agent token from the previous section. Refer to the agent tokens documentation for more info.
ORGANIZATION_ID - The organization ID of your Dagster Cloud organization, found in your account's URL. For example, if your organization is found at https://dagster.cloud/pied-piper or https://pied-piper.dagster.cloud/, this would be pied-piper.
Refer to the GitHub docs for more info about GitHub Action secrets.
In both workflow files, uncomment the step that is relevant to your Docker container registry. For example, if using DockerHub, you'd uncomment the DockerHub step in these files.
When finished setting up the repository, navigate back to Dagster Cloud and move on to the next step.
The agent will attempt to load your code. After the code is loaded, the new code location will show a Loaded status and a sample Dagster job will display in Dagster Cloud.
Congrats! At this point, your Dagster Cloud deployment should be up and running. To ensure the external services you use in your Dagster project work correctly, start setting up your environment variables. Using environment variables, you can securely pass in sensitive info like passwords, API tokens, etc.