Overview

Transformers 4.33, released on September 20, 2023, adds Llama 2 support and improves model quantization.

Main Features

Llama 2 support

Meta's Llama 2 models are natively integrated, enabling direct inference and fine-tuning.

python 
from transformers import pipeline

gen = pipeline(
    'text-generation',
    model='meta-llama/Llama-2-7b-chat-hf',
    device_map='auto',
)
result = gen('Hello, how', max_new_tokens=50)
print(result[0]['generated_text'])

Improved quantization

GPTQ and bitsandbytes support lets you load models in 4 bits, reducing required GPU memory.

python 
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype='float16',
)
model = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Llama-2-7b-hf',
    quantization_config=bnb_config,
    device_map='auto',
)

Sources