Setting Dagster Cloud environment variables using agent configuration#

This guide is applicable to Dagster Cloud.

In this guide, we'll walk you through setting environment variables for a Dagster Cloud Hybrid deployment using the Hybrid agent's configuration.

There are two ways to set environment variables:

  • On a per-code location basis, which involves modifying the dagster.yaml file. Note: This approach is functionally the same as setting environment variables using the Dagster Cloud UI. Values will pass through Dagster Cloud.
  • For a full deployment and all the code locations it contains. This approach makes variables available for all code locations in a full Dagster Cloud deployment. As values are pulled from the user cluster, values will bypass Dagster Cloud entirely.

Prerequisites#

To complete the steps in this guide, you'll need:


Setting environment variables for a code location#

Editor, Admin, or Organization Admin permissions are required to modify and redeploy code locations in Dagster Cloud.

Setting environment variables for specific code locations is accomplished by adding them to your agent's configuration in your project's dagster_cloud.yaml file. The container_context property in this file sets the variables in the agent's environment.

Note: This approach is functionally the same as setting environment variables using the Dagster Cloud UI.

How container_context is configured depends on the agent type. Click the tab for your agent type to view instructions.

Amazon ECS agents#

Using the container_context.ecs.env_vars and container_context.ecs.secrets properties, you can configure environment variables and secrets for a specific code location.

# dagster_cloud.yaml

container_context:
  ecs:
    env_vars:
      - DATABASE_NAME=testing
      - DATABASE_PASSWORD
    secrets:
      - name: "MY_API_TOKEN"
        valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::"
      - name: "MY_PASSWORD"
        valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::"
    secrets_tags:
      - "my_tag_name"
PropertyDescription
container_context.ecs.env_varsA list of keys or key-value pairs to include in the task. If a value is not specified, the value will be pulled from the agent task.
In the example above, FOO_ENV_VAR will be set to foo_value and BAR_ENV_VAR will be set to whatever value it has in the agent task.
container_context.ecs.secretsIndividual secrets specified using the same structure as the ECS API.
container_context.ecs.secrets_tagsA list of tag names. Each secret tagged with any of those tag names in AWS Secrets Manager will be included in the launched tasks as environment variables. The name of the environment variable will be the name of the secret, and the value of the environment variable will be the value of the secret.

After you've modified dagster_cloud.yaml, redeploy the code location in Dagster Cloud to apply the changes:

Highlighted Redeploy option in the dropdown menu next to a code location in Dagster Cloud

Setting environment variables for full deployments#

Setting environment variables for a full deployment will make the variables available for all code locations in the full deployment. Using this approach will pull variable values from your user cluster, bypassing Dagster Cloud entirely.

Click the tab for your agent type to view instructions.

Amazon ECS agents#

To make environment variables accessible to a full deployment with an Amazon ECS agent, you'll need to modify the agent's CloudFormation template as follows:

  1. Sign in to your AWS account.

  2. Navigate to CloudFormation and open the stack for the agent.

  3. Click Update.

  4. Click Edit template in designer.

  5. In the section that displays, click View in Designer. The AWS template designer will display.

  6. In the section displaying the template YAML, locate the AgentTaskDefinition section:

    Highlighted AgentTaskDefinition section of the AWS ECS agent CloudFormation template in the AWS Console
  7. In the user_code_launcher.config portion of the AgentTaskDefinition section, add the environment variables as follows:

    user_code_launcher:
      module: dagster_cloud.workspace.ecs
      class: EcsUserCodeLauncher
      config:
        cluster: ${ConfigCluster}
        subnets: [${ConfigSubnet}]
        service_discovery_namespace_id: ${ServiceDiscoveryNamespace}
        execution_role_arn: ${TaskExecutionRole.Arn}
        task_role_arn: ${AgentRole}
        log_group: ${AgentLogGroup}
        env_vars:
          - SNOWFLAKE_USERNAME=dev
          - SNOWFLAKE_PASSWORD       ## pulled from agent environment
    ' > $DAGSTER_HOME/dagster.yaml && cat $DAGSTER_HOME/dagster.yaml && dagster-cloud agent run"
    
  8. When finished, click the Create Stack button:

    Highlighted Create Stack button in the AWS Console
  9. You'll be redirected back to the Update stack wizard, where the new template will be populated. Click Next.

  10. Continue to click Next until you reach the Review page.

  11. Click Submit to update the stack.