Overview

Transformers 4.44, released on September 15, 2024, adds GGUF format support to load quantized models efficiently.

Main Features

GGUF support

The GGUF format allows loading quantized models (Q4, Q8) directly in Transformers, reducing memory footprint for local inference.

python 
from transformers import AutoModelForCausalLM, AutoTokenizer

# Loading a quantized GGUF model
model = AutoModelForCausalLM.from_pretrained(
    'TheBloke/Llama-2-7B-GGUF',
    gguf_file='llama-2-7b.Q4_K_M.gguf',
)
tokenizer = AutoTokenizer.from_pretrained(
    'TheBloke/Llama-2-7B-GGUF',
    gguf_file='llama-2-7b.Q4_K_M.gguf',
)
print(f'Model loaded in GGUF Q4')

Sources