Overview

Metaflow 2.3, released on June 16, 2021, improves AWS Batch integration for more reliable and performant ML pipeline execution.

Main Features

Improved AWS Batch

AWS Batch integration is strengthened with better resource control, priority queue support, and improved failure retry management.

python 
from metaflow import FlowSpec, step, batch

class TrainingFlow(FlowSpec):
    @step
    def start(self):
        self.data = load_data()
        self.next(self.train)

    @batch(cpu=4, memory=16000, gpu=1)
    @step
    def train(self):
        """Executed on AWS Batch with GPU."""
        self.model = train_model(self.data)
        self.next(self.end)

    @step
    def end(self):
        print(f'Trained model: {self.model}')

Artifact management

Artifact management is improved with more efficient storage and the ability to load results from previous runs for reproducibility.

python 
from metaflow import Flow

# Access results from a previous run
run = Flow('TrainingFlow').latest_successful_run
print(f'Run: {run.id}')
print(f'Model: {run.data.model}')

# List all runs
for run in Flow('TrainingFlow').runs():
    print(f'{run.id} - {run.finished_at}')

Sources