Overview

PyArrow 6.0, released on October 27, 2021, adds Substrait standard support and new data types for interoperability.

Main Features

Substrait support

Substrait is an open standard for describing query plans portably across compute engines. PyArrow 6.0 begins supporting Substrait plan serialization and execution.

python 
import pyarrow as pa
import pyarrow.compute as pc

# Using Arrow compute functions
table = pa.table({
    'name': ['Alice', 'Bob', 'Charlie'],
    'score': [85, 92, 78],
})

# Filtering and projection (Substrait basis)
mask = pc.greater(table['score'], 80)
result = table.filter(mask)
print(result.to_pandas())

New data types

PyArrow 6.0 adds month_day_nano types for intervals and improves decimal and temporal type support.

python 
import pyarrow as pa

# New month_day_nano_interval type
interval_type = pa.month_day_nano_interval()
print(f'Type: {interval_type}')

# Improved decimal types
prices = pa.array(['19.99', '42.50', '7.25']).cast(pa.decimal128(10, 2))
print(prices)

# Temporal types
timestamps = pa.array([1635340800, 1635427200], type=pa.timestamp('s'))
print(timestamps)

Sources