Overview

Metaflow 2.7, released on June 15, 2022, adds Argo Workflows and Kubernetes support for ML pipeline orchestration.

Main Features

Argo Workflows support

Metaflow flows can be deployed on Argo Workflows for native Kubernetes orchestration, with built-in scheduling and monitoring.

python 
from metaflow import FlowSpec, step

class TrainingFlow(FlowSpec):
    @step
    def start(self):
        self.data = [1, 2, 3, 4, 5]
        self.next(self.train)

    @step
    def train(self):
        self.model = sum(self.data) / len(self.data)
        self.next(self.end)

    @step
    def end(self):
        print(f'Model: {self.model}')

# python flow.py argo-workflows create

Native Kubernetes

The @kubernetes decorator runs steps on Kubernetes pods with configurable resources.

python 
from metaflow import FlowSpec, step, kubernetes

class GPUFlow(FlowSpec):
    @kubernetes(cpu=4, memory=16000, gpu=1)
    @step
    def train_gpu(self):
        # Runs on a GPU pod
        print('Training on GPU')
        self.next(self.end)

    @step
    def end(self):
        print('Done')

Sources