Track · LLMOps and MLOps
MLOps and LLMOps projects: serve, test and monitor models
Put models behind an API and keep them healthy: vLLM and batching, an LLM gateway, load tests against SLOs, evaluation and prompt regression tests in CI, tracing and drift monitoring.
- 17
- Labs
- 15 h
- In total
- Intermediate to advanced
- Level
Open lab 1Deploy & Serve LLMs in Production (Jupyter) · 45 min
What you will build
- An LLM served with vLLM, and a model served on CPU within a p95 latency budget
- A production LLM backend with API keys, rate limits, quotas and streaming
- An LLM gateway between your apps and the model providers
- A prompt regression gate in CI and experiment tracking with MLflow
- Load tests against SLOs, traces through a RAG assistant, and drift monitoring
Before you start
- Python and comfort with an HTTP API
- Having called a language model before (the Start with AI track covers it)
Tools you will use
vLLMFastAPIONNXTritonMLflowOpenTelemetryPSI driftServer-sent events
Labs in this track
In order, from the first lab to the hardest. Every lab stands on its own, so start wherever you like.
1
Serve models
From a hand-built batcher to vLLM, a production FastAPI backend, an LLM gateway, streaming and prompt caching.
- Lab 1Deploy & Serve LLMs in Production (Jupyter)Go from slow single-request inference to production-ready LLM serving with vLLM. Benchmark throughput, tune settings, and learn when to use vLLM vs Triton vs TGI.45 minIntermediateGPUPro# vLLM · 10 concurrent$ asyncio.gather(…)wall time ....... 3.8sthroughput ...... 1,842 tok/s
- Lab 2Inference Serving Patterns: Dynamic Batching, Throughput, and the Triton Mental ModelBuild a mini-Triton inference server in ~30 lines of Python: a dynamic batcher with max_batch_size and max_queue_delay knobs, load-tested against a naive baseline, swept for the throughput-latency tradeoff, and bridged to a real Triton config.pbtxt.40 minIntermediateGPUPro
- Lab 3vLLM Production Serving: PagedAttention, Continuous Batching, Prefix CachingStand up vLLM and measure the three features that make it the de-facto inference server: PagedAttention's KV-cache capacity, continuous batching throughput, and prefix caching speedups. Then write the production spec: server args, Kubernetes deployment, monitoring, autoscaling.55 minAdvancedGPUPro
- Lab 4Batch Size & Precision Sweep: Finding Your Sweet SpotSweep batch sizes and numerical precisions (fp32, fp16, bf16) on a real model to find the throughput/VRAM knee, then ship a production recommendation with SKU-aware precision picks and an accuracy gate.40 minIntermediateGPUPro
- Lab 5Serve an ML Model on CPU: ONNX Export, FastAPI, Dynamic Batching and a p95 Latency BudgetTake a trained PyTorch image classifier to production on one CPU core. Export it to ONNX with a dynamic batch dimension and verify it, run it with ONNX Runtime at half PyTorch's latency, serve it with a validating FastAPI endpoint, add a micro-batcher that merges concurrent requests, and load-test with Poisson traffic to choose batching settings that keep p95 under 100 ms.60 minIntermediateHostedPro# serve-on-cpu · step 1$ lab.check(1)Step 1 Completegrade ........... pass
- Lab 6Ship an LLM Backend: FastAPI with API Keys, Rate Limits, Quotas, Streaming and Safe FailuresBuild the API in front of a hosted model the way production services do: validated requests, hashed API keys checked in constant time, per-key rate limits with Retry-After, daily token quotas, a streaming endpoint that still bills usage, and upstream failures turned into safe 502 and 504 responses with request ids.55 minIntermediateHostedPro# ship-llm-backend · step 1$ lab.check(1)Step 1 Completegrade ........... pass
- Lab 7Build an LLM Gateway: One Door Between Your Apps and the ModelPut a gateway you wrote in front of the model and give it the four floor capabilities every LLM feature needs: API keys with a request log, a price on every call, per-app rate limits and daily budgets, an exact-match response cache, retries with backoff and idempotency keys, model routing with a fallback, and a report over a replayed morning of traffic that shows who spent what.90 minIntermediateHostedPro# llm-gateway-from-scratch · step 1$ lab.check(1)Step 1 Completegrade ........... pass
- Lab 8Streaming and Prompt Caching: Time to First Token, SSE and Cheaper PromptsStream LLM responses and measure time to first token, parse Server-Sent Events by hand, detect a buffering gateway, reorder a prompt so the provider's prefix cache serves 97 % of it, cancel generation when the user stops, and relay a stream through your own FastAPI backend without buffering.50 minIntermediateHostedPro# streaming-prompt-caching · step 1$ lab.check(1)Step 1 Completegrade ........... pass
- Lab 9Context Engineering: Token Budgets, Compaction and Running Notes for Long ChatsKeep a long conversation inside a token budget without losing what matters: count tokens, keep a recent window, trim bulky tool output, compact old turns into running notes that track changed decisions, and assemble each request by priority. Measured on probe questions against the full history.50 minIntermediateHostedPro# context-engineering · step 1$ lab.check(1)Step 1 Completegrade ........... pass
2
Test and track
Benchmarks, a golden set that blocks bad prompt changes in CI, experiment tracking and runs you can reproduce.
- Lab 10Evaluation & Benchmarking LLMsFour evaluation lenses in one lab: compute real perplexity, expose BLEU's blindness to paraphrase, run side-by-side model comparisons, and build an LLM-as-judge harness with position-bias detection.45 minIntermediateGPUPro
- Lab 11Prompt Regression Testing in CI: A Golden Set That Blocks Bad Prompt ChangesBuild the test suite and CI gate for a production prompt. Write deterministic asserts for a golden set, add an LLM judge for rules that string checks cannot express and measure it against hand labels, run each case several times to separate regressions from noise, and gate pull requests on critical cases and pass-rate drops with a GitHub Actions workflow.50 minIntermediateHostedPro# prompt-regression-ci · step 1$ lab.check(1)Step 1 Completegrade ........... pass
- Lab 12MLflow Experiment Tracking: From Single Run to Team WorkflowWire the four load-bearing pieces of MLflow into a real training loop: tracked runs with params and metrics, a registered model with stage transitions, a multi-run sweep + search, and a production spec (server, k8s Job, tags, autolog).35 minIntermediateGPUPro
- Lab 13Reproducible Training: The Flags, The Cost, The ArtifactsMeasure the non-determinism noise floor in default PyTorch, flip every determinism flag until same-seed runs match bit-for-bit, quantify the perf cost, and capture a content-addressable training config that makes a run reproducible forever.40 minIntermediateGPUPro# determinism flags$ torch.use_deterministic_…seed ............ 42cudnn.benchmark . Falsereproducible .... TRUE
3
Load test and monitor
TTFT and TPOT against SLOs, traces through a RAG assistant, drift alerts with delayed labels, and an incident drill on a failing model service.
- Lab 14LLM Load Testing and SLOs: TTFT, TPOT, Open-Loop Traffic, Goodput and CapacityLoad-test a streaming LLM server the way production traffic will hit it. Measure time to first token and time per output token, build an open-loop generator that shows the queue a closed loop hides, replay a production request mix with Poisson arrivals, report p95 latencies, errors and goodput against an SLO, then bisect for a replica's capacity and size a launch.50 minIntermediateHostedPro# llm-load-testing · step 1$ lab.check(1)Step 1 Completegrade ........... pass
- Lab 15LLM Observability: Trace, Cost and Debug a RAG AssistantInstrument a real RAG assistant with traces from scratch: nested spans around retrieval and generation, token usage and cost attributed to every call, a latency and cost report, a triage tool that finds retrieval misses from trace evidence, PII redaction before anything is written, sampling that keeps every error and slow request, a cost budget alarm, and a CI gate that blocks a prompt change that makes the assistant slower or dearer.75 minIntermediateHostedPro
- Lab 16ML Model Monitoring: Data Drift with PSI, Delayed Labels, Alert Rules and Root CauseMonitor a late-delivery model through twelve weeks of production. Build a training baseline, track weekly population stability index for every input and the model's score, measure calibration once outcomes arrive two weeks late, write alert rules that page only on proven quality loss, then find the segment behind it and retrain without leaking future labels.50 minIntermediateHostedPro# monitoring-and-drift · step 1$ lab.check(1)Step 1 Completegrade ........... pass
- Lab 17Incident Drill: Debug a Failing ML Service from Logs and MetricsPerch's listing-guard started holding almost every new listing at 14:00 and the review queue is drowning. Read the request log to time the incident, reproduce it deterministically against the live service, localize the model feature that broke, find and fix the root cause in the serving code, then verify recovery and add the release check that would have paged first.45 minIntermediateHostedPro# incident-drill · step 1$ lab.check(1)Step 1 Completegrade ........... pass
Graded project
Ship a production LLM API feature
Build an LLM-backed feature against a hosted API the way you would in production, then submit it for a score on every rubric criterion.
Guides for this track
GuideMLOps course guide: what you should ship by the endComparisonMLOps vs DevOps: what changes when the artifact is a modelComparisonAI infrastructure engineer vs MLOps engineer
Related collections:LLM inference optimization course
Questions about this track
No. The serving labs start an NVIDIA GPU for you in the browser, and the rest run on hosted machines. There is nothing to install.
Both. The serving, testing and monitoring ideas are the same, so the track covers classic models and language models side by side.
Most take 40 to 60 minutes and save your progress between steps.
The labs in this track are part of Pro, which opens every lab and practice test.
Other tracks
GPU and Kubernetes
GPU scheduling on your own Kubernetes cluster, the GPU Operator, CUDA kernels and profiling.
RAG and search
Chunking, hybrid search and reranking, permission-aware retrieval, GraphRAG and RAG evaluation.
LLM training and fine-tuning
LoRA and QLoRA fine-tuning, DPO alignment, and a tokenizer and tiny GPT built from scratch.
Every lab with Pro
This track and every other one, plus every practice test. $29.99 a month, cancel any time.