Overview

Hugging Face Transformers 4.15, released on November 15, 2021, adds BigScience and FLAN model support for large-scale NLP research.

Main Features

BigScience support

The BigScience project is integrated with tools to train and deploy massive language models in a collaborative and open framework.

python 
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load a text generation model
tokenizer = AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForCausalLM.from_pretrained('gpt2')

# Text generation
inputs = tokenizer('Python is a language', return_tensors='pt')
outputs = model.generate(
    **inputs,
    max_new_tokens=20,
    do_sample=True,
    temperature=0.7,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

FLAN support

FLAN (Fine-tuned Language Net) models are supported, offering instruction-tuned pre-trained models for better zero-shot performance.

python 
from transformers import pipeline

# Zero-shot classification pipeline
classifier = pipeline(
    'zero-shot-classification',
    model='facebook/bart-large-mnli',
)

result = classifier(
    'Python 3.10 introduces structural pattern matching.',
    candidate_labels=['technology', 'sports', 'cooking'],
)
print(f'Label: {result["labels"][0]}')
print(f'Score: {result["scores"][0]:.3f}')

Sources