Overview
PySpark 3.2, released on October 14, 2021, integrates Koalas as pyspark.pandas for a Pandas API on Spark.
Main Features
pyspark.pandas (Koalas)
The Koalas API is integrated as pyspark.pandas, enabling familiar Pandas syntax on distributed Spark data.
python
import pyspark.pandas as ps
# Pandas API on Spark
df = ps.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'age': [30, 25, 35],
})
# Familiar Pandas operations
print(df.describe())
print(df.groupby('name').mean())Pandas/Spark conversion
Conversion between Pandas and Spark DataFrames is simplified, allowing switching between APIs as needed.
python
import pyspark.pandas as ps
import pandas as pd
# Pandas -> pyspark.pandas
pdf = pd.DataFrame({'a': [1, 2, 3]})
psdf = ps.from_pandas(pdf)
# pyspark.pandas -> Pandas
back_to_pandas = psdf.to_pandas()
print(type(back_to_pandas)) # pandas.DataFrame