Overview

Airflow 2.5, released on April 8, 2023, improves the datasets system for data-driven orchestration.

Main Features

Improved datasets

Datasets allow triggering DAGs automatically when upstream data is updated, creating data-driven dependencies between pipelines.

python 
from airflow.datasets import Dataset
from airflow.decorators import dag, task
from datetime import datetime

data = Dataset('s3://bucket/data.parquet')

@dag(schedule=[data], start_date=datetime(2023, 1, 1))
def analysis_pipeline():
    @task(outlets=[data])
    def load():
        return 'data loaded'

    load()

analysis_pipeline()

Interface improvements

The Airflow web interface benefits from an improved grid view and better display of dataset dependencies.

python 
from airflow.decorators import dag, task
from datetime import datetime, timedelta

@dag(
    schedule=timedelta(hours=1),
    start_date=datetime(2023, 1, 1),
    catchup=False,
)
def hourly_etl():
    @task
    def extract():
        return {'rows': 1000}

    @task
    def transform(data):
        return {'rows': data['rows'], 'processed': True}

    transform(extract())

hourly_etl()

Sources