Overview

Ray 1.6, released on August 2, 2021, introduces runtime environments and a workflow engine for orchestrating distributed tasks.

Main Features

Runtime environments

RuntimeEnv lets you specify Python dependencies, environment variables, and required files for each Ray task or actor, without prior cluster configuration.

python 
import ray

# Runtime environment with dependencies
runtime_env = {
    'pip': ['scikit-learn==1.0', 'pandas==1.3'],
    'env_vars': {'MODE': 'production'},
}

ray.init(runtime_env=runtime_env)

@ray.remote
def train_model(data):
    from sklearn.ensemble import RandomForestClassifier
    clf = RandomForestClassifier(n_estimators=100)
    # clf.fit(data)
    return 'Model trained'

result = ray.get(train_model.remote([1, 2, 3]))
print(result)

Workflow engine

Ray Workflows lets you define durable task pipelines with automatic retry on failure, state persistence, and conditional execution.

python 
from ray import workflow

@ray.remote
def extract(source):
    return {'data': [1, 2, 3], 'source': source}

@ray.remote
def transform(raw_data):
    return [x * 2 for x in raw_data['data']]

@ray.remote
def load(data):
    return f'{len(data)} items loaded'

# Durable ETL pipeline
# etl = load.bind(transform.bind(extract.bind('api')))
# result = workflow.run(etl, workflow_id='etl_001')

Sources