Overview

Pandas 2.0, released on April 4, 2023, is a major release introducing the PyArrow backend, Copy-on-Write mode, and nullable dtypes by default. These changes drastically improve performance and memory management.

Main Features

PyArrow backend

The new pyarrow backend stores data in Arrow columns instead of NumPy, offering superior performance for string, date, and categorical operations.

python 
import pandas as pd

# Read with PyArrow backend
df = pd.read_csv(
    'data.csv',
    dtype_backend='pyarrow',
    engine='pyarrow',
)
print(df.dtypes)
# name      string[pyarrow]
# age       int64[pyarrow]
# score     double[pyarrow]

# String operations much faster
df['name_upper'] = df['name'].str.upper()

Copy-on-Write

Copy-on-Write (CoW) mode avoids unnecessary data copies. Modifications only trigger a copy when needed, significantly reducing memory usage.

python 
import pandas as pd

pd.set_option('mode.copy_on_write', True)

df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})

# No copy here, just a view
df2 = df[['a']]

# Copy only triggered when modifying df2
df2.iloc[0] = 100  # copy triggered here
print(df['a'][0])   # 1 (unchanged)
print(df2['a'][0])  # 100

Nullable dtypes by default

Nullable types (Int64, Float64, boolean) use pd.NA instead of NaN, unifying missing value handling for all data types.

python 
import pandas as pd

# Explicit nullable types
s = pd.Series([1, None, 3], dtype='Int64')
print(s)
# 0       1
# 1    <NA>
# 2       3
# dtype: Int64

# Operations propagate NA correctly
print(s.sum())   # 4
print(s.mean())  # 2.0
print(s[1] is pd.NA)  # True

Sources