Overview

PyArrow 15.0, released on January 22, 2024, adds the PyCapsule interface for zero-copy exchange between Python libraries.

Main Features

PyCapsule interface

The PyCapsule interface (Arrow C Data Interface) enables zero-copy data exchange between libraries via __arrow_c_array__.

python 
import pyarrow as pa

arr = pa.array([1, 2, 3, 4, 5])

# Export via PyCapsule (Arrow C Data Interface)
schema_capsule, array_capsule = arr.__arrow_c_array__()

# Import from another library
restored = pa.Array._import_from_c_capsule(
    schema_capsule, array_capsule
)
print(restored)  # [1, 2, 3, 4, 5]

Parquet improvements

The Parquet engine gains faster reads through optimized page decoding and better predicate pushdown filtering.

python 
import pyarrow.parquet as pq
import pyarrow as pa

table = pa.table({'id': range(1000), 'val': range(1000)})
pq.write_table(table, '/tmp/data.parquet')

# Read with pushdown filtering
result = pq.read_table(
    '/tmp/data.parquet',
    filters=[('id', '>', 500)],
)
print(result.num_rows)  # 499

Sources