Overview

Dask 2023.1, released on January 15, 2023, introduces a query optimizer for DataFrames that significantly improves performance.

Main Features

DataFrame query optimizer

The new optimizer analyzes the DataFrame computation graph and eliminates redundant operations, fuses compatible steps, and reorders filters to minimize processed data.

python 
import dask.dataframe as dd

# Lazy reading of a large CSV file
ddf = dd.read_csv('sales_*.csv')

# The optimizer fuses operations
result = (
    ddf[ddf['amount'] > 100]
    .groupby('category')['amount']
    .sum()
    .compute()
)
print(result)

Scheduler improvements

The distributed scheduler benefits from better memory management and smarter load balancing across workers.

python 
from dask.distributed import Client

# Connect to a Dask cluster
client = Client('scheduler:8786')
print(client.dashboard_link)

# The scheduler intelligently distributes
# tasks based on available memory
import dask.array as da
x = da.random.random((10000, 10000), chunks=(1000, 1000))
result = x.mean().compute()
print(f'Mean: {result:.4f}')

Sources