Overview

Airflow 2.6, released on July 11, 2023, introduces object storage support and improves the dataset system.

Main Features

Object storage

Airflow 2.6 natively integrates object storage support (S3, GCS, Azure Blob) via a unified ObjectStoragePath API.

python 
from airflow.io.path import ObjectStoragePath
from airflow.decorators import dag, task
from datetime import datetime

@dag(start_date=datetime(2023, 1, 1), schedule=None)
def storage_pipeline():
    @task
    def read_file():
        path = ObjectStoragePath('s3://bucket/data.csv')
        content = path.read_text()
        return len(content.splitlines())

    read_file()

storage_pipeline()

Improved datasets

The dataset system benefits from finer trigger conditions and better update traceability.

python 
from airflow.datasets import Dataset
from airflow.decorators import dag, task
from datetime import datetime

source = Dataset('s3://bucket/source.parquet')
target = Dataset('s3://bucket/result.parquet')

@dag(schedule=[source], start_date=datetime(2023, 1, 1))
def transformation():
    @task(outlets=[target])
    def transform():
        return 'data transformed'

    transform()

transformation()

Sources