Modular Architecture
Transformers 5.0 restructures modeling files into standardized modules. Each model is defined by reusable components (attention layer, MLP, normalization) rather than a monolithic file. This simplifies creating new models and maintenance.
Structuring a Model
Components are defined in separate files and assembled via a configuration file. To extend an existing model, simply override the relevant component without touching the rest.
python
from transformers import AutoModel, AutoConfig
# Load a modular model
config = AutoConfig.from_pretrained('meta-llama/Llama-3-8B')
print(config.model_type) # 'llama'
# Internal structure is now:
# modeling_llama/
# attention.py -> LlamaAttention
# mlp.py -> LlamaMLP
# layer.py -> LlamaDecoderLayer
# model.py -> LlamaModel
# Extend a model: override a component
model = AutoModel.from_pretrained('meta-llama/Llama-3-8B')
print(f'Layers: {len(model.layers)}')