Overview

PyArrow 3.0, released on January 28, 2021, enriches the Dataset API for reading large files and adds new vectorized compute functions.

Main Features

Dataset API

The pyarrow.dataset API enables lazy reading of partitioned datasets in Parquet, CSV, or other formats. Predicate pushdown and column selection are performed at the reader level for optimal performance.

python 
import pyarrow.dataset as ds

# Lazy reading of a partitioned dataset
dataset = ds.dataset(
    'data/sales/',
    format='parquet',
    partitioning='hive',  # year=2021/month=01/
)

# Predicate pushdown and column selection
table = dataset.to_table(
    filter=(ds.field('year') == 2021),
    columns=['product', 'amount'],
)
print(f'Rows: {table.num_rows}')
print(table.to_pandas().head())

Compute functions

The pyarrow.compute module gains new vectorized compute functions: string operations, arithmetic, aggregations, and sorting. These functions operate directly on Arrow arrays without copying to NumPy.

python 
import pyarrow as pa
import pyarrow.compute as pc

# Vectorized computation on Arrow arrays
prices = pa.array([10.5, 23.0, 7.8, 45.2, 12.0])
quantities = pa.array([3, 1, 5, 2, 4])

# Element-wise multiplication
totals = pc.multiply(prices, quantities)
print(totals)  # [31.5, 23.0, 39.0, 90.4, 48.0]

# Aggregation
print(f'Total: {pc.sum(totals).as_py():.2f}')  # 231.90

# String functions
names = pa.array(['alice', 'BOB', 'Charlie'])
print(pc.utf8_capitalize(names))  # ['Alice', 'Bob', 'Charlie']

Sources