Overview

Airflow 2.4, released on September 20, 2022, introduces Datasets and data-aware scheduling to trigger DAGs based on data availability.

Main Features

Datasets

Datasets are logical references to data that create dependencies between DAGs based on data availability.

python 
from airflow import Dataset
from airflow.decorators import dag, task
from datetime import datetime

my_dataset = Dataset('s3://bucket/data.parquet')

@dag(schedule_interval='@daily', start_date=datetime(2022, 1, 1))
def producer():
    @task(outlets=[my_dataset])
    def produce():
        return 'data produced'
    produce()

producer()

Data-aware scheduling

Data-aware scheduling automatically triggers a DAG when its dependent datasets are updated, replacing sensors in many cases.

python 
from airflow import Dataset
from airflow.decorators import dag, task
from datetime import datetime

my_dataset = Dataset('s3://bucket/data.parquet')

@dag(schedule=[my_dataset], start_date=datetime(2022, 1, 1))
def consumer():
    @task
    def process():
        return 'data processed'
    process()
# Triggered when my_dataset is updated

Sources