Overview

PyArrow 16.0, released on April 24, 2024, adds run-end encoding support for repetitive data.

Main Features

Run-end encoding

Run-end encoding compresses columns with consecutive identical values, reducing memory and speeding up scans.

python 
import pyarrow as pa

# Repetitive data
arr = pa.array(['A'] * 1000 + ['B'] * 2000 + ['A'] * 500)
print(f'Elements: {len(arr)}')  # 3500

# Run-end encoding compresses efficiently
ree = arr.dictionary_encode()
print(f'Reduced memory size: {ree.nbytes}')

Sources