Overview

PyArrow 9.0, released on August 3, 2022, improves Parquet file reading and writing with new compression options.

Main Features

Parquet improvements

Parquet reading is accelerated with better predicate filtering and support for new compression options like ZSTD.

python 
import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [30, 25, 35],
})

# Write with ZSTD compression
pq.write_table(table, 'data.parquet', compression='zstd')

# Read with filtering
result = pq.read_table('data.parquet', filters=[('age', '>', 28)])
print(result.to_pandas())

Enriched compute

The compute module adds new string manipulation and aggregation functions for Arrow arrays.

python 
import pyarrow as pa
import pyarrow.compute as pc

names = pa.array(['alice smith', 'bob jones', 'charlie brown'])

# String manipulation
upper = pc.utf8_upper(names)
print(upper)  # ['ALICE SMITH', ...]

# Pattern extraction
first_names = pc.utf8_split_whitespace(names)
print(first_names)

Sources