Tokenizer Redesign

Transformers 5.0 removes the Fast/Slow tokenizer distinction. All tokenizers now use the Rust tokenizers backend by default, providing uniform performance. The old pure-Python classes are removed.

Migration

Code using AutoTokenizer works without changes. Explicit references to *TokenizerFast should be replaced with the base class.

python 
from transformers import AutoTokenizer

# Works without changes
tok = AutoTokenizer.from_pretrained('bert-base-uncased')
print(type(tok))  # BertTokenizer (Rust backend)

# Before: two classes
# from transformers import BertTokenizer      # slow (Python)
# from transformers import BertTokenizerFast  # fast (Rust)

# After: one class, always fast
from transformers import BertTokenizer
tokens = tok('Hello world', return_tensors='pt')
print(tokens.input_ids.shape)  # torch.Size([1, 4])

Sources