Overview

PySpark 3.4, released on April 14, 2023, introduces Spark Connect, a lightweight Python client for remote connection to a Spark cluster.

Main Features

Spark Connect

Spark Connect enables connecting to a Spark cluster via gRPC without embedding a local JVM. The Python client is lightweight and installs simply via pip.

python 
from pyspark.sql import SparkSession

# Connect via Spark Connect
spark = SparkSession.builder.remote(
    'sc://spark-cluster:15002'
).getOrCreate()

df = spark.read.parquet('data.parquet')
result = df.groupBy('category').count()
result.show()

Lightweight Python client

The Spark Connect Python client does not require a local Java installation. It communicates with the Spark server via an efficient binary protocol.

python 
# Lightweight installation
# pip install pyspark[connect]

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, avg

spark = SparkSession.builder.remote(
    'sc://localhost:15002'
).getOrCreate()

# Same DataFrame API
df = spark.createDataFrame([
    ('Alice', 85), ('Bob', 92), ('Charlie', 78)
], ['name', 'score'])

print(df.select(avg(col('score'))).collect())

Sources