Overview

PySpark 3.3, released on June 17, 2022, introduces Arrow-optimized UDFs for better data transfer performance.

Main Features

Arrow-optimized UDFs

Arrow UDFs use Apache Arrow for data transfer between the JVM and Python, offering up to 100x better performance.

python 
from pyspark.sql import SparkSession
from pyspark.sql.functions import pandas_udf
import pandas as pd

spark = SparkSession.builder.getOrCreate()

@pandas_udf('double')
def normalize(s: pd.Series) -> pd.Series:
    return (s - s.mean()) / s.std()

df = spark.range(1000).selectExpr('id as value')
df.select(normalize('value')).show(5)

Spark SQL improvements

Spark SQL benefits from new query optimizations and better support for complex types like arrays and structs.

python 
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame([
    ('Alice', [90, 85, 92]),
    ('Bob', [78, 88, 95]),
], ['name', 'scores'])

# Array operations
df.select(
    'name',
    F.aggregate('scores', F.lit(0), lambda acc, x: acc + x).alias('total'),
).show()

Sources