Using environment variables with components
info
dg and Dagster Components are under active development. You may encounter feature gaps, and the APIs may change. To report issues or give feedback, please join the #dg-components channel in the Dagster Community Slack.
With dg and components, you can easily configure components depending on the environment in which they are run. To demonstrate this, we'll walk through setting up an example ELT pipeline with a Sling component which reads Snowflake credentials from environment variables.
1. Create a new Dagster components project
First, we'll set up a basic ELT pipeline using Sling in an empty Dagster components project:
dg scaffold project ingestion
cd ingestion && source .venv/bin/activate
We'll install dagster-sling and scaffold an empty Sling connection component:
uv add dagster-sling
dg list plugins
Plugin object cache is invalidated or empty. Building cache...
Using /.../ingestion/.venv/bin/dagster-components
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Plugin ┃ Objects ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Symbol ┃ Summary ┃ Features ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │
│ │ │ dagster.asset │ Create a │ [scaffold-targ… │ │
│ │ │ │ definition for │ │ │
│ │ │ │ how to compute │ │ │
│ │ │ │ an asset. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.asset_check │ Create a │ [scaffold-targ… │ │
│ │ │ │ definition for │ │ │
│ │ │ │ how to execute │ │ │
│ │ │ │ an asset check. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.components.DefinitionsComponent │ An arbitrary set │ [component, │ │
│ │ │ │ of dagster │ scaffold-targe… │ │
│ │ │ │ definitions. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.components.DefsFolderComponent │ A folder which │ [component, │ │
│ │ │ │ may contain │ scaffold-targe… │ │
│ │ │ │ multiple │ │ │
│ │ │ │ submodules, each │ │ │
│ │ │ │ which define │ │ │
│ │ │ │ components. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.components.PipesSubprocessScriptCollectionComponent │ Assets that wrap │ [component, │ │
│ │ │ │ Python scripts │ scaffold-targe… │ │
│ │ │ │ executed with │ │ │
│ │ │ │ Dagster's │ │ │
│ │ │ │ PipesSubprocess… │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.job │ Creates a job │ [scaffold-targ… │ │
│ │ │ │ with the │ │ │
│ │ │ │ specified │ │ │
│ │ │ │ parameters from │ │ │
│ │ │ │ the decorated │ │ │
│ │ │ │ graph/op │ │ │
│ │ │ │ invocation │ │ │
│ │ │ │ function. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.multi_asset │ Create a │ [scaffold-targ… │ │
│ │ │ │ combined │ │ │
│ │ │ │ definition of │ │ │
│ │ │ │ multiple assets │ │ │
│ │ │ │ that are │ │ │
│ │ │ │ computed using │ │ │
│ │ │ │ the same op and │ │ │
│ │ │ │ same │ │ │
│ │ │ │ upstream assets. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.resources │ Symbol for dg │ [scaffold-targ… │ │
│ │ │ │ scaffold to │ │ │
│ │ │ │ target. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.schedule │ Creates a │ [scaffold-targ… │ │
│ │ │ │ schedule │ │ │
│ │ │ │ following the │ │ │
│ │ │ │ provided cron │ │ │
│ │ │ │ schedule and │ │ │
│ │ │ │ requests runs │ │ │
│ │ │ │ for the provided │ │ │
│ │ │ │ job. │ │ │
│ │ ├─────────────────────────────────────────────────────────────┼──────────────────┼─────────────────┤ │
│ │ │ dagster.sensor │ Creates a sensor │ [scaffold-targ… │ │
│ │ │ │ where the │ │ │
│ │ │ │ decorated │ │ │
│ │ │ │ function is used │ │ │
│ │ │ │ as the sensor's │ │ │
│ │ │ │ evaluation │ │ │
│ │ │ │ function. │ │ │
│ │ └─────────────────────────────────────────────────────────────┴──────────────────┴─────────────────┘ │
│ dagster_sling │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Symbol ┃ Summary ┃ Features ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ dagster_sling.SlingReplicationCollectionComponent │ Expose one or more │ [component, │ │
│ │ │ │ Sling replications │ scaffold-target] │ │
│ │ │ │ to Dagster as │ │ │
│ │ │ │ assets. │ │ │
│ │ └───────────────────────────────────────────────────┴──────────────────────┴───────────────────────┘ │
└───────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┘
dg scaffold dagster_sling.SlingReplicationCollectionComponent ingest_to_snowflake
Using /.../ingestion/.venv/bin/dagster-components
2. Use environment variables in a component
Next, we will configure a Sling connection that will sync a local CSV file to a Snowflake database, with credentials provided with environment variables:
curl -O https://raw.githubusercontent.com/dbt-labs/jaffle-shop-classic/refs/heads/main/seeds/raw_customers.csv
src/ingestion/defs/ingest_files/replication.yaml
source: LOCAL
target: SNOWFLAKE
defaults:
mode: full-refresh
object: "{stream_table}"
streams:
file://raw_customers.csv:
object: "sandbox.raw_customers"
We will use the env function to template credentials into Sling configuration in our component.yaml file. Running dg check yaml will highlight that we
need to explicitly encode these environment dependencies at the bottom of the file:
src/ingestion/defs/ingest_files/component.yaml
type: dagster_sling.SlingReplicationCollectionComponent
attributes:
sling:
connections:
- name: SNOWFLAKE
type: snowflake
account: "{{ env('SNOWFLAKE_ACCOUNT') }}"
user: "{{ env('SNOWFLAKE_USER') }}"
password: "{{ env('SNOWFLAKE_PASSWORD') }}"
database: "{{ env('SNOWFLAKE_DATABASE') }}"
replications:
- path: replication.yaml
dg check yaml
/.../ingestion/.../ingestion/defs/ingest_files/component.yaml:1 - requirements.env Component uses environment variables that are not specified in the component file: SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_PASSWORD, SNOWFLAKE_USER
|
1 | type: dagster_sling.SlingReplicationCollectionComponent
| ^ Component uses environment variables that are not specified in the component file: SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_PASSWORD, SNOWFLAKE_USER
2 |
3 | attributes:
4 | sling:
5 | connections:
6 | - name: SNOWFLAKE
7 | type: snowflake
8 | account: "{{ env('SNOWFLAKE_ACCOUNT') }}"
9 | user: "{{ env('SNOWFLAKE_USER') }}"
10 | password: "{{ env('SNOWFLAKE_PASSWORD') }}"
11 | database: "{{ env('SNOWFLAKE_DATABASE') }}"
12 | replications:
13 | - path: replication.yaml
|
After adding the environment dependencies, running dg check yaml again will confirm that the file is valid:
src/ingestion/defs/ingest_files/component.yaml
type: dagster_sling.SlingReplicationCollectionComponent
attributes:
sling:
connections:
- name: SNOWFLAKE
type: snowflake
account: "{{ env('SNOWFLAKE_ACCOUNT') }}"
user: "{{ env('SNOWFLAKE_USER') }}"
password: "{{ env('SNOWFLAKE_PASSWORD') }}"
database: "{{ env('SNOWFLAKE_DATABASE') }}"
replications:
- path: replication.yaml
requirements:
env:
- SNOWFLAKE_ACCOUNT
- SNOWFLAKE_USER
- SNOWFLAKE_PASSWORD
- SNOWFLAKE_DATABASE
dg check yaml
All components validated successfully.
Next, you can invoke dg list env, which shows all environment variables configured or used by components in the project. Here we can see all of the Snowflake credentials we must configure in our shell or .env file in order to run our project:
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ │ ingest_files │
│ SNOWFLAKE_DATABASE │ │ ingest_files │
│ SNOWFLAKE_PASSWORD │ │ ingest_files │
│ SNOWFLAKE_USER │ │ ingest_files │
└────────────────────┴───────┴──────────────┘
You can edit the .env file in your project root to specify environment variables for Dagster to use when running the project locally. You can run dg list env again to see that they are now set:
echo 'SNOWFLAKE_ACCOUNT=...' >> .env
echo 'SNOWFLAKE_USER=...' >> .env
echo 'SNOWFLAKE_PASSWORD=...' >> .env
echo "SNOWFLAKE_DATABASE=sandbox" >> .env
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │
└────────────────────┴───────┴──────────────┘
[Optional] 3. Configure environment variables for components in Dagster+
If you are using Dagster+, you can also use the dg CLI to push environment variables to your deployment so that your code is ready to run in a production setting.
First, you will need to authenticate with Dagster+:
dg plus login
The dg list env command will now automatically show what scopes environment variables are configured for, in your configured deployment:
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃ Dev ┃ Branch ┃ Full ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │ │ │ │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │ │ │ │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │ │ │ │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │ │ │ │
└────────────────────┴───────┴──────────────┴─────┴────────┴──────┘
From here, you can use the dg plus create env command to push environment variables to your deployment. The --from-local-env flag will read the environment variables from your local shell or .env file.
Configure local environment variables
To enable the rest of your team to run your project locally, you can run the following command to push each Snowflake credential to the local scope:
dg plus create env SNOWFLAKE_ACCOUNT --from-local-env --scope local &&
dg plus create env SNOWFLAKE_USER --from-local-env --scope local &&
dg plus create env SNOWFLAKE_PASSWORD --from-local-env --scope local &&
dg plus create env SNOWFLAKE_DATABASE --from-local-env --scope local
Reading environment variable SNOWFLAKE_ACCOUNT from project .env file
Environment variable SNOWFLAKE_ACCOUNT set in local scope for location ingestion in deployment prod
Reading environment variable SNOWFLAKE_USER from project .env file
Environment variable SNOWFLAKE_USER set in local scope for location ingestion in deployment prod
Reading environment variable SNOWFLAKE_PASSWORD from project .env file
Environment variable SNOWFLAKE_PASSWORD set in local scope for location ingestion in deployment prod
Reading environment variable SNOWFLAKE_DATABASE from project .env file
Environment variable SNOWFLAKE_DATABASE set in local scope for location ingestion in deployment prod
Running dg list env again will confirm that the environment variables have been pushed to your deployment:
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃ Dev ┃ Branch ┃ Full ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │ ✓ │ │ │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │ ✓ │ │ │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │ ✓ │ │ │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │ ✓ │ │ │
└────────────────────┴───────┴──────────────┴─────┴────────┴──────┘
Now that local scope environment variables have been pushed to your deployment, your teammates can pull them into a .env file for local use:
dg plus pull env
Environment variables saved to .env
Configure production environment variables
Finally, you can configure a different set of production environment variables for your Dagster+ deployment:
dg plus create env SNOWFLAKE_ACCOUNT ... --scope branch --scope full &&
dg plus create env SNOWFLAKE_USER ... --scope branch --scope full &&
dg plus create env SNOWFLAKE_PASSWORD ... --scope branch --scope full &&
dg plus create env SNOWFLAKE_DATABASE production --scope branch --scope full
Environment variable SNOWFLAKE_ACCOUNT set in branch, full scope for location ingestion in deployment prod
Environment variable SNOWFLAKE_USER set in branch, full scope for location ingestion in deployment prod
Environment variable SNOWFLAKE_PASSWORD set in branch, full scope for location ingestion in deployment prod
Environment variable SNOWFLAKE_DATABASE set in branch, full scope for location ingestion in deployment prod
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃ Dev ┃ Branch ┃ Full ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
└────────────────────┴───────┴──────────────┴─────┴────────┴──────┘