Overview

PySpark 3.5, released on September 16, 2023, enables Arrow UDFs by default and adds Spark Connect support.

Main Features

Arrow UDFs by default

Pandas UDFs now use Apache Arrow by default for data transfer, offering much better performance.

python 
from pyspark.sql import SparkSession
from pyspark.sql.functions import pandas_udf
import pandas as pd

spark = SparkSession.builder.getOrCreate()

@pandas_udf('double')
def double_it(s: pd.Series) -> pd.Series:
    return s * 2  # Arrow transfer by default

Spark Connect

Spark Connect lets you connect to a remote Spark cluster via a lightweight API without starting a local JVM.

python 
from pyspark.sql import SparkSession

# Remote connection via Spark Connect
spark = (
    SparkSession.builder
    .remote('sc://cluster:15002')
    .getOrCreate()
)
df = spark.sql('SELECT 1 AS id')
df.show()

Sources