Dagster
Dagster is an orchestration platform for the development, production, and observation of data assets. It helps you define jobs and assets in code, and provides tools to run them reliably and monitor their health. It is for data engineers and platform engineers building data pipelines.
✨ Key features
- Define data assets and jobs in Python code.
- Schedule and orchestrate jobs with sensors and schedules.
- Supports multiple execution environments (local, cloud, Kubernetes).
- Provides a UI for observing runs, assets, and logs.
- Integrates with popular data tools like dbt, Airflow, and Spark.
- Supports software-defined assets for data lineage.
🎯 Use cases
- Orchestrate ETL pipelines that load data into a warehouse.
- Build and monitor machine learning training pipelines.
- Manage data quality checks and alerting.
- Create data lineage and asset catalog for governance.
📦 Installation
🧰 Requirements: Python 3.8 or later, and a Dagster instance (local or cloud) for deployment.
pip install dagster dagster-webserver
For more installation options, see the Dagster installation guide.
🚀 Usage
from dagster import asset, job, op, materialize
@asset
def my_data():
return [1, 2, 3]
# Define a job that materializes the asset
@job
def my_job():
my_data()
# Run the job locally
result = my_job.execute_in_process()
To view the Dagster UI, run dagster dev in your terminal.
⚠️ Good to know
Dagster is actively developed; some features may be experimental and require additional configuration for production use.
❓ FAQ
What is the difference between an asset and a job?
An asset is a data object that is produced or consumed, while a job is a set of operations that materialize assets. Assets can be used to define data lineage.
How do I schedule a job?
You can use the @schedule decorator to schedule a job to run at a specific time, or use sensors to trigger runs based on external events.
Can I run Dagster on Kubernetes?
Yes, Dagster supports running on Kubernetes using the dagster-k8s package, which allows you to run jobs in a Kubernetes cluster.
How do I view the UI?
Run dagster dev in your terminal to start the Dagster webserver and open the UI in your browser.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.