Hands-on labs for LLMs, RAG & agents. Real GPUs.
Fine-tune LLMs with LoRA, ship RAG pipelines on NVIDIA NIM, build agentic systems, and profile CUDA — on real GPU sandboxes and hosted environments. No setup, no simulators.
Start here
Build a ReAct Agent with NVIDIA NIM
Build a working AI research librarian — an agent that can search a corpus of ML papers, read abstracts, compare methods, and reason over them to answer multi-step questions. Uses LangChain, LangGraph, and NVIDIA NeMo Agent Toolkit on real NIM endpoints.
Build an MCP Tool Server & Connect a LangChain Agent
Build a Model Context Protocol server that exposes your company's tools and data — then connect a LangChain agent to it. Learn how MCP decouples tools from agents, when to use MCP vs Anthropic Skills vs native @tool, and why MCP is the emerging standard for AI tool interop.
Build a RAG Pipeline with NVIDIA NIM
Build a complete Retrieval Augmented Generation pipeline — from document chunking to vector search to an agent that answers questions from your knowledge base.
Agentic AI
Build Two Agents That Talk via the A2A Protocol
Build two independent agents that talk to each other via the A2A protocol — each owned by a different team, running in its own process, discovered through a standardized AgentCard. Learn how A2A differs from multi-agent orchestration and when each architecture fits.
Add Long-Term Memory to an AI Agent: LangGraph + Milvus
Build a sales intelligence assistant that remembers — short-term conversation state with LangGraph checkpointer, long-term facts in Milvus, and reflection loops that auto-extract knowledge. Learn the memory architecture every production agent needs.
Build an AI Agent 3 Ways: ReAct vs Tool Calling vs Plan-and-Execute
Build the same SaaS customer support agent three different ways — ReAct, direct tool calling, and plan-and-execute — then compare them on speed, reasoning quality, and reliability to learn when to use each pattern in production.
Build an MCP Tool Server & Connect a LangChain Agent
Build a Model Context Protocol server that exposes your company's tools and data — then connect a LangChain agent to it. Learn how MCP decouples tools from agents, when to use MCP vs Anthropic Skills vs native @tool, and why MCP is the emerging standard for AI tool interop.
Build a Multi-Agent Supervisor with LangGraph
Build a supervisor agent that routes queries to specialist agents — a core architecture pattern tested on the NCP-AAI exam.
Build a RAG Pipeline with NVIDIA NIM
Build a complete Retrieval Augmented Generation pipeline — from document chunking to vector search to an agent that answers questions from your knowledge base.
Build a ReAct Agent with NVIDIA NIM
Build a working AI research librarian — an agent that can search a corpus of ML papers, read abstracts, compare methods, and reason over them to answer multi-step questions. Uses LangChain, LangGraph, and NVIDIA NeMo Agent Toolkit on real NIM endpoints.
Build NeMo Guardrails for an AI Agent: Jailbreak & Topical Rails
Build a guarded IT support agent that blocks jailbreaks, refuses off-topic questions, and safely handles IT queries — using keyword checks, LLM-based validation, and NeMo Guardrails.
Evaluate an Agent with LLM-as-Judge
Build an eval harness that scores agent responses automatically — correctness via a reference-based judge, plus an accuracy metric and A/B comparison. Same pattern used by NeMo Evaluator for production agent evaluation.
Model Routing & Cost Cascade with NIM
Save 60–80% on inference by cascading queries through cheap → mid → expensive NIM models. Measure real costs via NIM's usage.cost field and compare against an always-large baseline.
Structured Output & Function Calling with NIM
Get reliable machine-parseable data out of an LLM. Compare prompt-only JSON extraction against the function-calling API, chain two tools, and measure the reliability gap on a real extraction task.
Visual Q&A with NVIDIA VLMs
Send images to a Vision-Language Model via NIM, answer questions about them, extract structured fields from a receipt-style image, and compare two VLMs on the same task — all through the OpenAI-compatible chat endpoint.
Multimodal RAG with NeMo Retriever
Build an image-query RAG system: embed a catalog with NeMo Retriever, translate an uploaded image into a retrieval query via a VLM, and ground the VLM's final answer in the retrieved passages.
Insecure Output Handling: SSRF, SQLi, and Command Execution Through an Agent's Tools
Red-team OpsBot, a ReAct tool-using support agent, by shaping the arguments it passes to its own tools. File a poisoned support ticket and a benign on-call query turns into a server-side request forgery against a metadata endpoint, a SQL injection that drops a canary and reads another tenant's rows, and code execution through a transform helper. Measure attack-success-rate, then close every sink: an allow-listed fetch, parameterized queries, and a removed code tool.
Inter-Agent Injection: Propagate a Morris II Worm Across a Two-Agent Graph
Attack a real two-agent support graph where one agent's output is the next agent's input with no authentication and no validation. Plant a self-replicating payload in the only untrusted input, an inbound customer email the Intake agent ingests. Intake forwards it, and the Resolver agent, trusting the inter-agent notes as instructions, performs the attacker-directed action AND re-emits the payload verbatim: a second-hop cascade and the Morris II replication primitive, bounded to two hops. The harness attributes every side effect to the agent that caused it, so you prove the second agent executed. Measure propagation reliability, then harden the channel, a schema-constrained, validated, replication-aware handoff, and prove the cascade is contained while benign tickets still resolve.
MCP Tool Poisoning: Hijack an Agent Through a Tool Description (and a Rug Pull)
Attack a real MCP-style tool registry end to end. OrbitDesk's support agent loads its tools from a runtime registry and reads each tool description as trusted instruction text. Register a poisoned tool whose description hides a routine-looking audit directive, make the agent read an account record and forward its reference to your in-pod collector (data exfiltration through tool metadata), then pull a rug: register the tool benign, pass review, and silently mutate its description after approval. Measure attack-success-rate, then harden the registry, scan descriptions for hidden instructions and pin approved tool objects, and prove a fresh poison and a fresh rug pull are both dead while benign tickets still resolve.
Memory Poisoning: Plant a Note That Re-Fires in a Fresh Session (Persistence)
Attack the long-term memory of a real tool-using ReAct agent. As the low-privilege ticket-ingest account, plant a single benign-looking routing note into DV-ToolAgent's shared, un-namespaced memory store through an ingested ticket. In a brand-new session for a different legitimate user, the agent recalls your note and redirects a GLOBEX invoice to your payee, persistence across the session boundary that single-turn filters never see. Use MINJA-style progressive shortening so the stored record reads as a mundane preference, measure attack-success-rate against benign controls, then harden memory with per-user namespacing and a data-only quarantine and prove the poison dead while legitimate recall still works.
Excessive Agency: Turn a Support Ticket into a Privileged Action (Confused Deputy)
Attack a real tool-using ReAct agent end to end. As the low-privilege ticket-ingest account, plant an authorized-looking record correction in a support ticket and make DV-ToolAgent run a privileged billing-payee redirect under its own shared credential (the confused deputy), then reach an internal-only endpoint and exfiltrate its value through the fetch tool (SSRF via tool args). Measure attack-success-rate, then harden the tool boundary, scope the DB tool read-only, carry per-user authorization, and allow-list the fetch tool, and prove your own exploit is dead while normal lookups still work.
Tool Shadowing: Hijack an Agent's Tool Selection With a Name Collision
Attack a real MCP-style tool registry by shadowing a trusted tool. OrbitDesk's support agent loads its tools from a runtime registry with a flat namespace and last-write-wins resolution, and it picks which tool to call from attacker-controllable descriptions. Register a malicious twin with the same name as the trusted record reader and a more compelling, compliance-approved description, so the agent calls your twin instead. The twin silently reads a cross-tenant record and exfiltrates its reference, and the same shadow hands another tenant's data straight back to the caller. Measure how reliably the shadow wins selection, then harden the registry, namespace tools and reject collisions, and prove a fresh shadow is dead while benign lookups still work.
Defend a RAG Assistant: Build a Guardrail Layer and an Attack-Success-Rate CI Gate
You inherit DV-RAG-Support with a working EchoLeak-style exploit, and you defend it in small sequential steps. Stand the assistant up and trace one benign request, then reproduce the leak: an indirect prompt injection makes the model echo a customer's account record into a markdown image, and the renderer fires it as an outbound request that exfiltrates the data. Watch a naive host deny-list get bypassed by a renamed host, then build the durable guardrail one mechanism per step: an egress allow-list on the render sink so only approved hosts load, then output redaction of the sensitive record so even an approved-host request carries nothing. Verify the sink is closed with benign answers intact, then stand up an attack-success-rate (ASR) gate that runs a probe battery on every change: green while guardrails hold, non-zero the moment a regression re-opens the sink, green again when you back it out. Finish by proving fresh, renamed, and paraphrased payloads are all blocked.
Defend Excessive Agency: Re-scope a Tool Agent to Least Privilege (AuthZ + Human Approval Gate)
Harden DV-ToolAgent, a real tool-using ReAct agent, against the confused-deputy and scope-escalation attacks from the offensive lab, in small sequential steps. Stand up the agent and trace one benign ticket, then reproduce the handed-to-you exploit one surface at a time: an ingested ticket that makes the agent redirect a billing payee under its own shared credential and read across tenants, then an SSRF reach into an internal-only endpoint and a poisoned-memory replant. Watch a naive SQL denylist get bypassed by a case-folded variant, then build the durable control one mechanism per step: least-privilege tool scope (the ingest role holds no write scope), per-argument authorization decided on the session identity (a caller-supplied identity claim is ignored), and a human-in-the-loop approval gate that holds high-impact writes pending an explicit token. Verify the exploit is dead with authorized in-scope work intact, then prove obfuscated, renamed, and spoofed variants are all blocked.
Defend the Agent Supply Chain: Verify, Pin, and Capability-Gate Your Tool Registry
Harden a real MCP-style tool registry until a poisoned, rug-pulled, or shadowing tool manifest cannot reach the agent, in small single-concept steps. OrbitDesk's support agent loads its tools from a runtime registry and reads each tool description as trusted instruction text. You stand the registry up and trace one benign ticket, then reproduce three techniques one at a time: a poisoned tool description that hijacks the agent into leaking an account reference, a post-approval rug-pull mutation the registry serves with no pin, and a shadowing twin the flat namespace selects under a trusted tool name. You apply the obvious fix, a description blocklist, and watch a clean-description variant defeat it through its delegate. Then you build the durable control one mechanism per step: manifest signature verification, then hash pinning (rug-pull / change detection), then a per-tool capability allow-list with namespacing. You finish by proving that freshly planted unsigned, forged, mutated, and shadowing manifests are all refused while a legitimate signed and pinned tool stays admitted and usable, and an inter-agent worm's second hop is contained.
LLM serving & inference
Deploy & Serve LLMs in Production (Jupyter)
Go from slow single-request inference to production-ready LLM serving with vLLM. Benchmark throughput, tune settings, and learn when to use vLLM vs Triton vs TGI.
Inference Serving Patterns: Dynamic Batching, Throughput, and the Triton Mental Model
Build a mini-Triton inference server in ~30 lines of Python: a dynamic batcher with max_batch_size and max_queue_delay knobs, load-tested against a naive baseline, swept for the throughput-latency tradeoff, and bridged to a real Triton config.pbtxt.
Batch Size & Precision Sweep: Finding Your Sweet Spot
Sweep batch sizes and numerical precisions (fp32, fp16, bf16) on a real model to find the throughput/VRAM knee, then ship a production recommendation with SKU-aware precision picks and an accuracy gate.
vLLM Production Serving: PagedAttention, Continuous Batching, Prefix Caching
Stand up vLLM and measure the three features that make it the de-facto inference server: PagedAttention's KV-cache capacity, continuous batching throughput, and prefix caching speedups. Then write the production spec — server args, Kubernetes deployment, monitoring, autoscaling.
Fine-tuning & alignment
GPU ProFine-Tune an LLM with LoRA and QLoRA (Jupyter)
Fine-tune Meta Llama 3 8B on a custom instruction dataset using LoRA and QLoRA. Learn parameter-efficient fine-tuning from data preparation through evaluation — the #1 most demanded AI skill.
GPU ProQuantize & Optimize LLMs with bitsandbytes
Load a model in fp16, INT8, and NF4, then benchmark the three precisions on VRAM, latency, and output quality. See where quantization wins and where it costs you.
GPU ProRLHF & DPO Alignment
Run real Direct Preference Optimization on a small language model with TRL's DPOTrainer. Capture a baseline, build a preference dataset, train, and measurably shift the model's behavior in four steps.
GPU ProFine-Tune Stable Diffusion with LoRA: Custom Text-to-Image
Load Stable Diffusion, attach LoRA adapters to the U-Net's attention layers, run a tiny overfit training loop, and generate with the adapted weights to prove that a few million trainable parameters actually move pixels.
RAG & retrieval
GPU ProAdvanced RAG: Hybrid Search + Cross-Encoder Reranking
Build a production-shape retrieval stack — dense bi-encoder plus from-scratch BM25, fused with Reciprocal Rank Fusion, then re-ordered by a BAAI cross-encoder. The exact architecture behind modern enterprise RAG.
GPU ProRetrieval-Augmented Generation (RAG) Pipeline with Local Models
Build an end-to-end RAG pipeline on a single GPU: BGE embeddings, L2-normalized vector retrieval by dot product, and a local generator that answers with and without retrieved context so you can see exactly what retrieval changes.
HOSTED ProPersistent Storage for AI Workloads — PVCs, StorageClass & the Checkpoint Pattern
Stop losing your training checkpoints when pods restart. Learn the PersistentVolumeClaim model end-to-end — StorageClass selection, accessModes (and the RWO-is-per-node trap), the bind/mount/persist lifecycle, and three triage scenarios where storage chains break.
HOSTEDFreeIndirect Prompt Injection: Exfiltrate Data from a RAG Assistant
Attack a real Retrieval-Augmented Generation assistant end to end: a Milvus vector store, NVIDIA embeddings, and a multi-tenant knowledge base. Win semantic retrieval with a poisoned document, exfiltrate a customer's confidential account record through the EchoLeak markdown-image channel, measure attack-success-rate, bypass a naive defense, then ship the real fix.
HOSTED ProCross-Tenant Leakage: Break RAG Metadata Isolation and Exfiltrate Another Tenant's Contract
Attack the multi-tenant isolation of a real Retrieval-Augmented Generation assistant. Two stacked bugs in one retriever, a caller-controlled tenant scope and a string-concatenated metadata filter, let a Globex-scoped caller read Initech's confidential contract from a Milvus + NVIDIA embeddings store. Chain the cross-tenant read into the EchoLeak markdown-image sink to exfiltrate the data to a listener, then harden the pipeline so isolation and the sink both hold.
HOSTED ProRetrieval Poisoning: Win Top-k Across a Whole Query Class and Steer the Answer
Attack a real Retrieval-Augmented Generation assistant where it is most exposed: retrieval. Plant one document in a Milvus + NVIDIA embeddings knowledge base, craft it to win cosine top-k for one account question, then for the whole account-query class, then steer the generated answer through a directive framed as routine policy. Measure broad-class coverage and steering attack-success-rate, then harden in two distinct moves: treat retrieved context as data behind a non-spoofable boundary, and cap how many top-k slots any single source may take. Re-run the same battery and watch attack-success-rate collapse.
HOSTED ProRecon and Harness: Map a RAG Attack Surface and Measure Attack-Success-Rate
Open your AI Red Team engagement against a real Retrieval-Augmented Generation assistant and build the methodology the whole path reuses. Stand up the service and trace one request, enumerate its attack surface into a structured, machine-checkable map, encode a single probe, then build a deterministic side-channel oracle that counts real effects instead of the model's talk. Triage true positives from verbal-only false positives, scale to an Attack-Success-Rate harness with a per-class breakdown, wire it into a CI gate that goes red on the vulnerable build, then ship the render-path allow-list and watch the same gate go green while benign questions still answer.
HOSTED ProSystem-Prompt Extraction: Recover a RAG Assistant's Hidden Instructions
Red-team Aria, a real Retrieval-Augmented Generation support assistant: a Milvus vector store, NVIDIA embeddings, and an LLM that answers from one shared context window. Confirm a hidden system prompt with an embedded secret exists, recover it through ordinary chat by direct echo, climb the extraction ladder when a stronger refusal posture resists, defeat a naive output filter with encoding-egress, measure the extraction Attack-Success-Rate across the techniques, then ship the durable fix (minimize the secret out of the prompt) and verify extraction yields nothing.
HOSTED ProInsecure Output Handling: Zero-Click Exfiltration Through Rendered Model Output (EchoLeak)
Treat the model as an untrusted source whose output flows into a sink: the chat client's markdown renderer. Prove the renderer auto-fetches, plant a document so a benign account question makes the assistant echo a customer's own account reference into a markdown image URL, and watch the renderer auto-fetch it (zero-click exfil, the EchoLeak channel). Measure its attack-success-rate, defeat a CSP-style allow-list through a first-party open proxy, measure that bypass too, then harden in two moves: close the render sink so untrusted output never fires an outbound request, add an audited host allow-list the open proxy cannot defeat, and re-run both attacks to watch ASR fall to zero.
HOSTED ProSensitive Data Disclosure: Leak Confidential Records from a RAG Assistant
Attack a real Retrieval-Augmented Generation assistant where the system prompt only asks for privacy: a Milvus vector store, NVIDIA embeddings, and a multi-tenant knowledge base. Force disclosure of your own gated billing secret, pull another customer's record across a disabled tenant filter, harvest an accidentally-indexed service key, then ship the real fix with pre-retrieval authorization, corpus hygiene, and output redaction.
HOSTED ProDefense in Depth: Wire Four Control Points Around a RAG Assistant
Harden DV-RAG-Support, a real Retrieval-Augmented Generation assistant, by building a guard harness with four independent control points one mechanism per step: input mediation, retrieval and context control, output mediation, and action authorization. You are handed a working four-attack battery (direct injection, cross-tenant retrieval, sensitive-field exfiltration, and an unauthorized image fetch). Stand the pipeline up, reproduce all four attacks one per stage, watch a single naive filter get bypassed, then build each control point in its own step so every attack class is stopped at its matching layer while a benign customer request still passes clean through all four. Verify the coverage matrix reads four-for-four, then prove reworded and renamed bypass variants are all resisted.
HOSTED ProDefend a RAG Assistant: Block Indirect-Injection Exfil (EchoLeak)
Harden the same deliberately-vulnerable RAG assistant the offensive lab broke, in small sequential steps. Stand up the pipeline and trace one benign request, reproduce the EchoLeak markdown-image exfil, then watch a naive deny-list get bypassed by a renamed host. Build the durable fix one mechanism per step: an egress allow-list on the render sink that pins the parsed host (defeating userinfo, IP-encoded, and IPv6 spellings) and covers reference-style images, then provenance isolation so retrieved documents cannot emit instructions. Verify both controls together, resist a userinfo / IP-encoding / paraphrase bypass battery, and pass a final ship gate where attack success rate is 0 and benign quality holds.
HOSTED ProBuild a RAG Firewall: Reject Poisoned Ingestion and Enforce Tenant Isolation
Defend the same multi-tenant RAG assistant the offensive labs attack, in small sequential steps. Stand up the pipeline and trace one benign request, then reproduce two handed-to-you exploits one at a time: a poisoned document that wins retrieval and steers the answer, and a caller-controlled tenant scope that reads another tenant's confidential contract. Watch a naive deny-list get bypassed by a fresh payload, then build the durable control one mechanism per step: a server-side tenant predicate the caller cannot widen, then an ingestion screen that rejects directive-shaped documents before indexing. Verify both exploits are blocked with benign traffic intact, then prove fresh, paraphrased, and renamed variants are all blocked on a real Milvus + NVIDIA embeddings stack.
HOSTED ProDefend: Secret Isolation for a RAG Assistant
Harden the same RAG support assistant that the extraction lab broke, in small sequential steps. A live signing key, an internal build id, and a canary token are baked into the system prompt, so the secret is exposed by construction: it shares one context window with the customer's message. Stand the service up, reproduce the exposure (the secret is present in the model's context), and watch a naive cleartext output filter fall to encoding-egress. Then build the durable control one mechanism per step: a vault boundary that holds the secret out of the model's context, a seeded canary tripwire, and a fail-closed decoding leak detector that matches the secret and its Base64/ROT13/hex forms. Verify the secret is unrecoverable and benign answers are intact, then resist a re-planted, re-encoded exfil battery.
Training & pretraining
GPU ProContinued Pre-Training: Adapt a Pretrained LM to a New Domain
Take GPT-2 and domain-adapt it to Python code in 150 steps, measuring both the gain on code and the cost in catastrophic forgetting on English. The exact recipe behind Code Llama, BloombergGPT, and every domain-specialized LLM of the last three years.
GPU ProTrain a Small Language Model from Scratch
Train a real GPT-style language model from zero on TinyStories: tokenize, wire up the optimizer and LR schedule, run the training loop with validation perplexity, and generate coherent text from your own weights. End-to-end pretraining in minutes on one GPU.
GPU ProBuild a Transformer from Scratch: Attention, Masking & LayerNorm
Build every piece of a decoder-only transformer by hand — scaled dot-product attention, multi-head attention, the full block with residuals and LayerNorm, then assemble a tiny GPT and train it. No shortcuts, no pre-built attention modules.
CUDA & kernel optimization
GPU ProCUDA Programming Fundamentals
Write four real CUDA C++ kernels and run them from PyTorch: vector add, 2D matrix add, tiled matmul with shared memory, and a custom autograd op.
GPU ProGPU Sharing: Streams, MPS, MIG, and the Real Cost of Contention
Measure four ways to share a single GPU — CUDA streams, multi-process time-slicing, MPS, and MIG — and write the production artifacts (start scripts, k8s device-plugin ConfigMaps, MIG geometries) that turn 15%-utilized fleets into 80%-utilized ones.
GPU ProNsight Systems Profiling: Finding the Bottleneck That Costs You 40% of Your GPU
Run the full profile-then-fix loop with NVIDIA Nsight Systems — instrument a training loop with NVTX ranges, capture a .nsys-rep, parse the NVTX summary to pinpoint the bottleneck, then apply a targeted fix and measure the speedup.
Profiling & performance
GPU ProProfile PyTorch Training with the Built-in Profiler
Instrument a training loop with torch.profiler, read the op-level table, inspect the Chrome/Perfetto timeline, and decide when to reach for Nsight Systems instead.
GPU ProGPU Cost & Efficiency Audit
Build a four-stage cost-audit pipeline — measure, classify, price, recommend — that turns raw NVML samples into dollar-denominated waste and specific remediation actions. The skeleton behind every enterprise GPU cost product.
Multimodal
Data & pipelines
GPU ProNVIDIA DALI: GPU-Accelerated Data Pipelines
Move image decoding, resizing, and augmentation from CPU to GPU with NVIDIA DALI, and benchmark it against a standard PyTorch DataLoader. The input-pipeline fix that unlocks real multi-GPU throughput.
GPU ProData Preparation for LLM Training
Build a real pretraining/instruction data pipeline: load a raw corpus, apply quality filters, deduplicate, train a BPE tokenizer, and batch-validate on GPU. This is the unglamorous work that actually decides how good your model will be.
GPU ProSynthetic Data Generation for Model Training
Build a Self-Instruct style synthetic dataset end-to-end: seed instructions, LLM-driven generation, robust parsing, quality filtering, and dedup + diversity scoring. The same pipeline that produced Alpaca, WizardLM, and most modern instruction-tuning corpora.
GPU infrastructure
HOSTED ProWelcome to NCA-AIIO Labs — Schedule Your First GPU Pod
Smoke-test your NCA-AIIO lab environment: inspect your isolated Kubernetes cluster, schedule a Pod that requests an NVIDIA GPU, and observe realistic nvidia-smi output. The first 5-minute lab to verify everything works end-to-end.
HOSTED ProKubernetes Resource Requests & Limits — Who Gets What, and Who Survives
Master the most consequential six lines in any Kubernetes manifest: requests, limits, and how they decide scheduling, throttling, eviction, and survival under pressure. Includes the CFS throttling controversy and what 2026 production teams actually do with CPU limits.
HOSTED ProInside the NVIDIA GPU Operator — From Helm to Workload-Ready
Walk the chain that turns a Helm install into the `nvidia.com/gpu` your workload requests. Inspect the cluster the way platform engineers do — node labels, capacity, RuntimeClass — and learn to attribute every piece of evidence to the GPU Operator component that produced it. Finishes with a Triage Day where three broken GPU pods each break a different chain link.
GPU ProGPU Container Lifecycle: Build, Test, Ship, Rollback
Walk through the full lifecycle of a production GPU container — multi-stage Dockerfile, self-hosted GPU CI, a fail-fast smoke test, and a Kubernetes Deployment with readiness probes gated on real GPU compute. The pipeline that stops bad images before users see a 500.
GPU ProGPU Health Checks + Auto-Remediation
Build a production-grade GPU watchdog: multi-dimensional NVML health probe, rogue-process detection, auto-remediation that kills the offender and verifies recovery, then wire it up with Prometheus alerts and Kubernetes liveness probes.
GPU ProMLflow Experiment Tracking: From Single Run to Team Workflow
Wire the four load-bearing pieces of MLflow into a real training loop — tracked runs with params and metrics, a registered model with stage transitions, a multi-run sweep + search, and a production spec (server, k8s Job, tags, autolog).
GPU ProNVIDIA GPU Operator on k3s: Single-Node Kubernetes for GPU Workloads
Bring up a lightweight single-node Kubernetes cluster with the NVIDIA GPU Operator — k3s install, containerd wiring, Helm values, workload manifests with RBAC and ResourceQuota, plus a full runbook (validation plan, troubleshooting matrix, day-2 ops).
HOSTED ProPriorityClass & Preemption — Who Survives the GPU Squeeze
When GPU capacity is full and a critical training job lands, who wins? This lab builds the mental model behind PriorityClass and preemption — the only mechanism Kubernetes gives you for resolving GPU contention with intent rather than first-come-first-served. Includes the `preemptionPolicy: Never` escape hatch most teams misuse.
HOSTED ProWorkload Controllers — Deployment, StatefulSet, DaemonSet for AI
Three controller types, three workload shapes, three different production failure modes. Learn when to use a Deployment for inference, a StatefulSet for distributed training, and a DaemonSet for per-node GPU infrastructure — and how to spot when someone picked the wrong one.
HOSTED ProRolling Updates, Rollback & Blue-Green for AI Inference
Ship a new model version without dropping requests. Master Kubernetes' three deployment strategies — rolling update with readiness probes, rollback after a bad release, and blue-green via Service-selector swap — all on a stand-in inference Deployment.
HOSTED ProStuck-Pending Triage Day — Diagnose Any GPU Pod That Won't Run
The capstone NCA-AIIO operations lab. Walk the five-stage pod lifecycle (admission → scheduling → image pull → runtime → readiness), learn which `kubectl describe` field signals each stuck point, and finish by fixing four broken GPU pods, each broken at a different stage.
HOSTED ProMulti-GPU-Type Targeting — nodeSelector, nodeAffinity & Tolerations
Your fleet has A100s for training, L40S for inference, and Tesla-K80s for dev. Workloads need to land on the right hardware. Master the four primitives Kubernetes gives you — nodeSelector, nodeAffinity (required vs preferred), taints + tolerations — across a real multi-pool cluster.
Monitoring & ops
Accelerated data science
More labs
GPU ProEvaluation & Benchmarking LLMs
Four evaluation lenses in one lab: compute real perplexity, expose BLEU's blindness to paraphrase, run side-by-side model comparisons, and build an LLM-as-judge harness with position-bias detection.
GPU ProReproducible Training: The Flags, The Cost, The Artifacts
Measure the non-determinism noise floor in default PyTorch, flip every determinism flag until same-seed runs match bit-for-bit, quantify the perf cost, and capture a content-addressable training config that makes a run reproducible forever.
GPU ProGPU Environment Smoke Test
Validate the GPU lab environment: terminal, file operations, PyTorch, CUDA, and model loading.
HOSTED ProFuzz an LLM App with garak: Run, Read, and Triage True vs False Positives
Run NVIDIA garak as an automated fuzzer against a real vulnerable RAG support assistant, read the JSONL run log and the per-probe DEFCON report, then do the skill that separates a scanner operator from a red teamer: triage the hits. Dismiss a detector false positive with evidence, confirm a genuine indirect prompt injection against an in-pod exfil listener, and watch the finding regress to zero after the fix ships.
HOSTED ProDefend: A Contextual Output Mediator for XSS, SSRF, SQLi, and RCE
Defend DV-ToolAgent, a tool-using support agent whose model output flows raw into four interpreters, in small sequential steps. Stand up the agent and trace one benign request, then reproduce each sink one at a time: a behavioral SSRF (the agent fetches an internal node-health endpoint), a behavioral cross-tenant SQL read (an entitlement comparison pulls another tenant's record), then OS command execution, a SQL write, and stored XSS demonstrated structurally at the sink. Watch a naive blocklist get bypassed by a variant, then build one contextual output mediator one sink-class at a time: an SSRF host allow-list with an IP-literal guard and parameterized tenant-scoped SQL, then refuse-arbitrary-code for the runtime and contextual HTML encoding for the browser. Verify every freshly planted payload and its bypass variant is blocked at the sink while benign output still renders, queries, fetches, and computes.
Every lab runs on real AI infrastructure.
No video simulators, no canned outputs. Spin up a real GPU, or hook into our hosted stack — either way, you're graded on the metrics you actually produce.