Overview

Transformers 4.38, released on March 15, 2024, adds Gemma support and quantized KV cache.

Main Features

Gemma support

Google's Gemma model is natively integrated into the library.

python 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('google/gemma-2b')
tokenizer = AutoTokenizer.from_pretrained('google/gemma-2b')

inputs = tokenizer('Hello, world!', return_tensors='pt')
outputs = model.generate(**inputs, max_new_tokens=50)

Quantized KV cache

The key-value cache can be quantized to int8/int4 to reduce memory usage during inference.

python 
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    'google/gemma-2b',
    device_map='auto',
)
# Quantized KV cache reduces memory footprint
# when generating long sequences

Sources