Preporato

Track · LLMOps and MLOps

MLOps and LLMOps projects: serve, test and monitor models

Put models behind an API and keep them healthy: vLLM and batching, an LLM gateway, load tests against SLOs, evaluation and prompt regression tests in CI, tracing and drift monitoring.

22
Labs
19 h
In total
Intermediate to advanced
Level
Open lab 1Deploy & Serve LLMs in Production (Jupyter) · 45 min

What you will build

  • An LLM served with vLLM, and a model served on CPU within a p95 latency budget
  • A production LLM backend with API keys, rate limits, quotas and streaming
  • An LLM gateway between your apps and the model providers
  • A prompt regression gate in CI and experiment tracking with MLflow
  • Load tests against SLOs, traces through a RAG assistant, and drift monitoring

Before you start

  • Python and comfort with an HTTP API
  • Having called a language model before (the Start with AI track covers it)

Tools you will use

vLLMFastAPIONNXTritonMLflowOpenTelemetryPSI driftServer-sent events

Labs in this track

In order, from the first lab to the hardest. Every lab stands on its own, so start wherever you like.

1

Serve models

From a hand-built batcher to vLLM, a production FastAPI backend, an LLM gateway, streaming and prompt caching.

  1. Lab 1Deploy & Serve LLMs in Production (Jupyter)Go from slow single-request inference to production-ready LLM serving with vLLM. Benchmark throughput, tune settings, and learn when to use vLLM vs Triton vs TGI.45 minIntermediateGPUPro
  2. Lab 2Inference Serving Patterns: Dynamic Batching, Throughput, and the Triton Mental ModelBuild a mini-Triton inference server in ~30 lines of Python: a dynamic batcher with max_batch_size and max_queue_delay knobs, load-tested against a naive baseline, swept for the throughput-latency tradeoff, and bridged to a real Triton config.pbtxt.40 minIntermediateGPUPro
  3. Lab 3vLLM Production Serving: PagedAttention, Continuous Batching, Prefix CachingStand up vLLM and measure the three features that make it the de-facto inference server: PagedAttention's KV-cache capacity, continuous batching throughput, and prefix caching speedups. Then write the production spec: server args, Kubernetes deployment, monitoring, autoscaling.55 minAdvancedGPUPro
  4. Lab 4Batch Size & Precision Sweep: Finding Your Sweet SpotSweep batch sizes and numerical precisions (fp32, fp16, bf16) on a real model to find the throughput/VRAM knee, then ship a production recommendation with SKU-aware precision picks and an accuracy gate.40 minIntermediateGPUPro
  5. Lab 5Serve an ML Model on CPU: ONNX Export, FastAPI, Dynamic Batching and a p95 Latency BudgetTake a trained PyTorch image classifier to production on one CPU core. Export it to ONNX with a dynamic batch dimension and verify it, run it with ONNX Runtime at half PyTorch's latency, serve it with a validating FastAPI endpoint, add a micro-batcher that merges concurrent requests, and load-test with Poisson traffic to choose batching settings that keep p95 under 100 ms.60 minIntermediateHostedPro
  6. Lab 6Ship an LLM Backend: FastAPI with API Keys, Rate Limits, Quotas, Streaming and Safe FailuresBuild the API in front of a hosted model the way production services do: validated requests, hashed API keys checked in constant time, per-key rate limits with Retry-After, daily token quotas, a streaming endpoint that still bills usage, and upstream failures turned into safe 502 and 504 responses with request ids.55 minIntermediateHostedPro
  7. Lab 7Build an LLM Gateway: One Door Between Your Apps and the ModelPut a gateway you wrote in front of the model and give it the four floor capabilities every LLM feature needs: API keys with a request log, a price on every call, per-app rate limits and daily budgets, an exact-match response cache, retries with backoff and idempotency keys, model routing with a fallback, and a report over a replayed morning of traffic that shows who spent what.90 minIntermediateHostedPro
  8. Lab 8Streaming and Prompt Caching: Time to First Token, SSE and Cheaper PromptsStream LLM responses and measure time to first token, parse Server-Sent Events by hand, detect a buffering gateway, reorder a prompt so the provider's prefix cache serves 97 % of it, cancel generation when the user stops, and relay a stream through your own FastAPI backend without buffering.50 minIntermediateHostedPro
  9. Lab 9Context Engineering: Token Budgets, Compaction and Running Notes for Long ChatsKeep a long conversation inside a token budget without losing what matters: count tokens, keep a recent window, trim bulky tool output, compact old turns into running notes that track changed decisions, and assemble each request by priority. Measured on probe questions against the full history.50 minIntermediateHostedPro
2

Test and track

Benchmarks, a golden set that blocks bad prompt changes in CI, experiment tracking and runs you can reproduce.

  1. Lab 10Evaluation & Benchmarking LLMsFour evaluation lenses in one lab: compute real perplexity, expose BLEU's blindness to paraphrase, run side-by-side model comparisons, and build an LLM-as-judge harness with position-bias detection.45 minIntermediateGPUPro
  2. Lab 11Prompt Regression Testing in CI: A Golden Set That Blocks Bad Prompt ChangesBuild the test suite and CI gate for a production prompt. Write deterministic asserts for a golden set, add an LLM judge for rules that string checks cannot express and measure it against hand labels, run each case several times to separate regressions from noise, and gate pull requests on critical cases and pass-rate drops with a GitHub Actions workflow.50 minIntermediateHostedPro
  3. Lab 12CI for Models: Data Contracts, Behavioral Tests, Golden Sets and a Merge GateBuild the CI suite that guards a fraud model. Enforce the data contract on every batch, write behavioral invariants that check the model obeys rules accuracy cannot see, pin the cases it must never get wrong with a golden set, gate on accuracy regression against the committed baseline, and aggregate all four into one merge decision that blocks a candidate whose sign-flip bug slips past the metrics.50 minIntermediateHostedPro
  4. Lab 13Training Pipeline Orchestration: Ordering, Retries, Failure Isolation and ResumeBuild the orchestrator behind a nightly training pipeline. Put its tasks in dependency order and reject cycles, run each task with exponential-backoff retries, run the whole pipeline so one failure skips only its own branch, resume a half-finished run without repeating the work that already succeeded, and report whether the run passed and what it cost.45 minIntermediateHostedPro
  5. Lab 14Data and Model Versioning: Content Addressing, Lineage and Garbage CollectionBuild a content-addressed store for datasets and models, the idea behind tools like DVC. Address blobs by their hash so identical data is stored once, snapshot a dataset as a tree and diff two versions, record the exact data and code behind every model, fingerprint a run to tell real reproductions from changed inputs and catch a tampered blob, then garbage collect what no tagged release needs while keeping every tagged model rollback-ready.45 minIntermediateHostedPro
  6. Lab 15Feature Store Basics: Point-in-Time Correctness and Training-Serving SkewBuild the core of a feature store on a payments fraud model and learn why the same feature computed two ways silently breaks production: point-in-time feature computation that never peeks at the future, a training set built as of each event, an online store materialized for low-latency lookups, a skew report that compares the training and serving code paths, and a measurement of what training-serving skew costs when it ships.60 minIntermediateHostedPro
  7. Lab 16Publish a Model: Card, Evaluation and a Versioned ReleaseShip a churn model responsibly. Evaluate the candidate overall and per customer segment, write a model card whose claims come from that evaluation, cut a content-hashed semantic-version release, gate promotion on a per-segment recall floor that catches a candidate which is better on average but worse for rural customers, then roll back a release that slipped through.45 minIntermediateHostedPro
  8. Lab 17MLflow Experiment Tracking: From Single Run to Team WorkflowWire the four load-bearing pieces of MLflow into a real training loop: tracked runs with params and metrics, a registered model with stage transitions, a multi-run sweep + search, and a production spec (server, k8s Job, tags, autolog).35 minIntermediateGPUPro
  9. Lab 18Reproducible Training: The Flags, The Cost, The ArtifactsMeasure the non-determinism noise floor in default PyTorch, flip every determinism flag until same-seed runs match bit-for-bit, quantify the perf cost, and capture a content-addressable training config that makes a run reproducible forever.40 minIntermediateGPUPro
3

Load test and monitor

TTFT and TPOT against SLOs, traces through a RAG assistant, drift alerts with delayed labels, and an incident drill on a failing model service.

  1. Lab 19LLM Load Testing and SLOs: TTFT, TPOT, Open-Loop Traffic, Goodput and CapacityLoad-test a streaming LLM server the way production traffic will hit it. Measure time to first token and time per output token, build an open-loop generator that shows the queue a closed loop hides, replay a production request mix with Poisson arrivals, report p95 latencies, errors and goodput against an SLO, then bisect for a replica's capacity and size a launch.50 minIntermediateHostedPro
  2. Lab 20LLM Observability: Trace, Cost and Debug a RAG AssistantInstrument a real RAG assistant with traces from scratch: nested spans around retrieval and generation, token usage and cost attributed to every call, a latency and cost report, a triage tool that finds retrieval misses from trace evidence, PII redaction before anything is written, sampling that keeps every error and slow request, a cost budget alarm, and a CI gate that blocks a prompt change that makes the assistant slower or dearer.75 minIntermediateHostedPro
  3. Lab 21ML Model Monitoring: Data Drift with PSI, Delayed Labels, Alert Rules and Root CauseMonitor a late-delivery model through twelve weeks of production. Build a training baseline, track weekly population stability index for every input and the model's score, measure calibration once outcomes arrive two weeks late, write alert rules that page only on proven quality loss, then find the segment behind it and retrain without leaking future labels.50 minIntermediateHostedPro
  4. Lab 22Incident Drill: Debug a Failing ML Service from Logs and MetricsPerch's listing-guard started holding almost every new listing at 14:00 and the review queue is drowning. Read the request log to time the incident, reproduce it deterministically against the live service, localize the model feature that broke, find and fix the root cause in the serving code, then verify recovery and add the release check that would have paged first.45 minIntermediateHostedPro

Graded project

Ship a production LLM API feature

Build an LLM-backed feature against a hosted API the way you would in production, then submit it for a score on every rubric criterion.

Open the project

Guides for this track

GuideMLOps course guide: what you should ship by the endComparisonMLOps vs DevOps: what changes when the artifact is a modelComparisonAI infrastructure engineer vs MLOps engineer

Related collections:LLM inference optimization course

Questions about this track

Other tracks

GPU and Kubernetes

GPU scheduling on your own Kubernetes cluster, the GPU Operator, CUDA kernels and profiling.

20 labs

RAG and search

Chunking, hybrid search and reranking, permission-aware retrieval, GraphRAG and RAG evaluation.

14 labs

LLM training and fine-tuning

LoRA and QLoRA fine-tuning, DPO alignment, and a tokenizer and tiny GPT built from scratch.

14 labs

Every lab with Pro

This track and every other one, plus every practice test. $29.99 a month, cancel any time.

See Pro