Transformers 4.36: Mixtral and KV-cache

Overview

Hugging Face Transformers 4.36, released on December 15, 2023, adds Mixtral support and optimizes KV-cache for inference.

Main Features

Mixtral (Mixture of Experts) is natively supported. The KV-cache improves inference performance for text generation.

from transformers import pipeline

gen = pipeline('text-generation', model='gpt2')
result = gen('Python is', max_new_tokens=20)
print(result[0]['generated_text'])

Sources

• Transformers Releases