Declarative Automation#

Dagster can automatically execute assets or checks when criteria are met, enabling a declarative approach to automation. Instead of defining explicit workflows and schedules, you describe the conditions under which they should be executed, and the system executes runs in response.

Declarative Automation includes pre-built conditions to handle common use cases, such as executing on a periodic schedule or whenever an upstream dependency updates, but conditions can be customized in a fine-grained manner, allowing precise control over when work gets executed.

Benefits#

Using Declarative Automation helps you:

• Ensure you're working with the most up-to-date data
• Optimize resource usage by only materializing assets or executing checks when needed
• Simplify how your team understands their assets by consolidating all asset logic to a single location
• Avoid thinking about specific workflow boundaries, such as a schedule accounting for timezones or Daylight Savings Time

Prerequisites#

Before continuing, you should be familiar with:

• Asset definitions
• Sensor definitions
• Code locations

How it works#

Declarative Automation is an automation method that executes runs when conditions are met. This method contains two main components:

• An automation condition (AutomationCondition), which represents when an individual asset or check should be executed.
• A sensor (AutomationConditionSensorDefinition), which evaluates each AutomationCondition and launches runs in response to their status.

Automation conditions#

Automation conditions describe the conditions under which work should be executed. Dagster provides three pre-built conditions:

With assets, automation conditions can be set on the @asset decorator or on an AssetSpec:

from dagster import AssetSpec, AutomationCondition, asset

@asset(automation_condition=AutomationCondition.eager())
def my_eager_asset(): ...

AssetSpec("my_cron_asset", automation_condition=AutomationCondition.on_cron("@daily"))

The same is true for asset checks:

from dagster import AssetCheckResult, AssetCheckSpec, AutomationCondition, asset_check


@asset_check(asset=..., automation_condition=AutomationCondition.cron_tick_passed("@daily"))
def expensive_check() -> AssetCheckResult:
    return AssetCheckResult(passed=True)


AssetCheckSpec(
    "expensive_check",
    asset=...,
    automation_condition=AutomationCondition.cron_tick_passed("@daily"),
)

The core AutomationCondition framework is extremely flexible, allowing you to build custom conditions from the ground up. Refer to the Customizing automation conditions guide for more information.

Sensors#

When automation conditions for an asset are met, a sensor will execute a run to materialize the asset. This sensor, named default_automation_condition_sensor, will be available for each code location and monitor all assets within that location. To use multiple sensors or change the properties of the default sensor, refer to the AutomationConditionSensorDefinition documentation.

For an automation condition sensor to run, it must be turned on and an active dagster-daemon process must be running. If you used dagster dev to start the Dagster UI/webserver, the daemon process will be automatically launched alongside the webserver.

After these criteria are met, the sensor's evaluation history will be visible in the UI:

You'll also be able to view a detailed history of each asset's evaluations on the asset's Asset Details page. This allows you to see why an asset was or wasn't materialized at different points in time:

Getting started#

To use Declarative Automation, you'll need to enable the automation condition sensor in the Dagster UI:

1. Navigate to Overview > Sensors
2. Locate the desired code location.
3. In the code location, toggle the default_automation_condition_sensor sensor to on.

From here, you can:

• Define custom automation conditions
• View a history of each evaluation for the sensor
• Navigate to individual assets to see a history of their evaluations

Related#