Preporato
Training Pipeline Orchestration: Ordering, Retries, Failure Isolation and Resume
Hands-on lab · IDE in your browser

Training Pipeline Orchestration: Ordering, Retries, Failure Isolation and Resume

Build the orchestrator behind a nightly training pipeline. Put its tasks in dependency order and reject cycles, run each task with exponential-backoff retries, run the whole pipeline so one failure skips only its own branch, resume a half-finished run without repeating the work that already succeeded, and report whether the run passed and what it cost.

Time
45 min
Checked steps
5
Level
Intermediate
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

The job

A team runs a nightly pipeline that ingests data, validates it, builds features, trains a model, evaluates it and registers the result. Right now it is a shell script that starts over whenever anything hiccups. You build the orchestrator it should have had: dependency ordering, retries for flaky steps, failure isolation, and resume from where it stopped.

5 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Order the tasks

    A training pipeline is a set of tasks with dependencies: validate needs ingest, train needs features, and so on.

  2. 2

    Run a task with retries

    Real pipeline tasks fail for boring reasons: a shared node drops the job, a download times out.

  3. 3

    Run the pipeline

    Now run the whole pipeline.

  4. 4

    Resume a half-finished run

    A pipeline that fails halfway should not start over from the top.

  5. 5

    Report the run

    The last thing a pipeline owes you is a straight answer: did it work, what failed, and how much did it cost in retries.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Order the tasks

Order the tasks

A training pipeline is a set of tasks with dependencies: validate needs ingest, train needs features, and so on. Before running anything you have to put them in an order where every task comes after the things it depends on, and reject a pipeline that can never run.

pipeline.py gives you the pipeline. A task is a dict {"name", "deps": [names], "run": fn(ctx), "retries"}. pipeline.build_pipeline() returns the task list and a fresh ctx, and the factories step, flaky and failing build tasks you can use in your own tests.

Write topo_order(tasks): return the task names in an order where each task follows its dependencies. Raise ValueError if a dependency names a task that is not in the set, or if the dependencies form a cycle (which could never be ordered).

orchestrator.py, the file you edit47 lines
"""Your pipeline orchestrator. A pipeline is a list of tasks with dependencies. You order them, run them so
one failure does not corrupt the rest, retry the flaky ones, resume a half-finished run without repeating
work, and report what happened."""
import time


# ---------- Step 1: order the tasks ----------

def topo_order(tasks):
    """The task names in an order where every task comes after its dependencies. Raise ValueError if a
    dependency is unknown or the graph has a cycle."""
    # TODO (Step 1): visit each task depth-first through its deps; a node seen while
    # still visiting is a cycle, a dep not in the task set is unknown; append on the way out.
    raise NotImplementedError("Step 1: write topo_order()")


# ---------- Step 2: run one task, with retries ----------

def attempt(task, ctx, sleep=time.sleep, base=0.1):
    """Run a task's function, retrying up to task['retries'] extra times on any exception, waiting
    base * 2**i before the (i+1)-th retry. Return {"status": "ok"/"failed", "attempts", "error"}."""
    raise NotImplementedError("attempt() arrives in Step 2")


# ---------- Step 3: run the pipeline ----------

def run_pipeline(tasks, ctx, sleep=time.sleep, done=None):
    """Run tasks in dependency order. A task whose dependencies are not all ok is skipped, so one
    failure isolates its branch instead of corrupting the rest. Tasks named in `done` are treated as
    already complete (cached) and not run again. Return {name: result}."""
    raise NotImplementedError("run_pipeline() arrives in Step 3")


# ---------- Step 4: resume a half-finished run ----------

def resume(tasks, ctx, prior, sleep=time.sleep):
    """Re-run only what is not already done: tasks that finished ok in `prior` are kept and skipped, the
    rest (failed, skipped or never run) run again, along with anything downstream of them."""
    raise NotImplementedError("resume() arrives in Step 4")


# ---------- Step 5: report ----------

def summary(results):
    """A run report: counts per status, the names that failed, the total attempts across all tasks, and
    whether the whole run succeeded (every task ok)."""
    raise NotImplementedError("summary() arrives in Step 5")
Provided for you:pipeline.pytry_it.py

Frequently asked questions

What is a DAG in a data or ML pipeline?

A directed acyclic graph: tasks are nodes and a dependency is an edge from a task to the one that needs it. Acyclic means there is a valid order to run them in. Orchestrators like Airflow and Prefect execute pipelines as DAGs, which is what this lab builds in miniature.

Why retry pipeline tasks with exponential backoff?

Many failures are transient: a node is preempted, a network call times out. Retrying a few times clears them, and waiting a growing interval between attempts avoids hammering a service that is already struggling. A task that still fails after its retries is a real failure worth stopping for.

What does it mean to resume a pipeline?

To restart a failed run without repeating the steps that already succeeded. The orchestrator keeps which tasks finished and their artifacts, then runs only the unfinished work and whatever depends on it, instead of starting from the top.

Orchestrating a training pipeline

A training pipeline is a graph of tasks with dependencies, and running it reliably means more than calling them in a row. The orchestrator has to order the tasks, retry the ones that fail transiently, keep one failure from running steps on missing inputs, and resume a run without repeating the hours of work that already finished. In this lab you build that orchestrator from scratch: a topological sort with cycle detection, an attempt function with exponential-backoff retries, a runner that isolates failures to their own branch, a resume that reuses completed work, and a run report.