NVIDIA AI Enterprise for Agents: Platform Integration Guide

Enterprise Support Tiers:

• Standard Support: Business-hours support, 1-business-day response for critical issues
• Premium Support: 24/7 support, 4-hour response for critical severity issues, dedicated technical account manager
• Both tiers include: Access to all certified containers, security patches, NIM microservices, and NIM Operator

Exam Tip: The NCP-AAI exam may test whether you understand that licensing is per-GPU (not per-node, per-model, or per-user). A server with 8 GPUs requires 8 licenses.

Core Components for Agentic AI

1. NVIDIA NIM (Inference Microservices)

Purpose: Deploy LLMs as scalable, containerized microservices with optimized inference.

NIM containers are pre-packaged with the model, inference engine (TensorRT-LLM), and OpenAI-compatible APIs. This eliminates weeks of manual model conversion, optimization, and API development.

What each NIM container includes:

• Optimized AI model -- Pre-configured with TensorRT-LLM optimizations
• Inference engine -- TensorRT-LLM or Triton Inference Server
• Industry-standard APIs -- OpenAI-compatible REST and gRPC endpoints
• Runtime dependencies -- CUDA, cuDNN, and all required libraries pre-installed
• Health checks -- Built-in readiness and liveness probes for Kubernetes

Performance characteristics:

• 2-4x faster inference vs. standard deployment without TensorRT-LLM
• Auto-scaling: Kubernetes-native via NIM Operator and HPA
• Multi-model hosting: Run multiple models on a single GPU with resource isolation
• Optimizations: TensorRT-LLM, INT8/FP16 quantization, KV-cache optimization

Exam Scenario: "An agent needs sub-500ms latency for tool-calling decisions. Which NVIDIA tool optimizes inference?" Answer: NVIDIA NIM with TensorRT-LLM -- the pre-packaged container eliminates optimization overhead while TensorRT-LLM provides maximum inference speed.

NIM Deployment Patterns for Agentic AI

The NCP-AAI exam tests several NIM deployment patterns. Understanding when to use each pattern is critical.

Pattern 1: Single-Agent with Dedicated NIM

The simplest deployment -- one agent backed by one NIM instance. Suitable for focused use cases like a customer service chatbot.

Agent Application
    ↓ OpenAI-compatible API
NIM Container (Llama-3-70B)
    ↓
Single GPU (H100/A100)

When to use: Low-to-medium traffic, single-purpose agent, predictable load.

Pattern 2: Multi-Agent RAG Pipeline

Multiple specialized NIMs serve different roles in a RAG-augmented agent pipeline. This is the most commonly tested pattern on the NCP-AAI exam.

Orchestrator Agent
    ├─→ LLM NIM (Llama-3-70B) ── Reasoning & Planning
    ├─→ Embedding NIM (NV-Embed-v2) ── Document Encoding
    ├─→ Reranker NIM (NV-RerankQA) ── Result Refinement
    └─→ Guardrails NIM (NeMo Guardrails) ── Safety Validation
         ↓
    Vector Database (Milvus)

When to use: Knowledge-intensive agents, document Q&A, enterprise search, compliance-critical deployments.

Pattern 3: Multi-Agent Swarm with Shared NIM Pool

Multiple agents share a pool of NIM instances, with load balancing distributing inference requests. This is the most resource-efficient pattern for large-scale deployments.

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│ Research     │  │ Analysis    │  │ Report      │
│ Agent        │  │ Agent       │  │ Agent       │
└──────┬──────┘  └──────┬──────┘  └──────┬──────┘
       └────────────────┼────────────────┘
                        ↓
              Load Balancer (K8s Service)
                        ↓
        ┌───────────────┼───────────────┐
        ↓               ↓               ↓
   NIM Replica 1   NIM Replica 2   NIM Replica 3
   (Llama-3-70B)   (Llama-3-70B)   (Llama-3-70B)

When to use: High-throughput multi-agent systems, bursty workloads, cost-sensitive deployments needing GPU sharing.

NIM Auto-Scaling with Kubernetes HPA

The NIM Operator enables auto-scaling based on GPU-specific metrics, not just CPU/memory. This is a key differentiator tested on the exam.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nim-llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nim-llm-deployment
  minReplicas: 1
  maxReplicas: 8
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_cache_usage_perc  # NIM-specific KV cache metric
      target:
        type: AverageValue
        averageValue: "75"          # Scale up when KV cache hits 75%

Exam Key Point: NIM exposes a Prometheus endpoint with metrics like gpu_cache_usage_perc (KV-cache utilization). Auto-scaling based on KV-cache usage is more effective than CPU-based scaling for LLM workloads because GPU memory pressure is the actual bottleneck, not CPU.

Scaling metrics available:

• gpu_cache_usage_perc -- KV-cache utilization (most recommended for LLMs)
• request_queue_length -- Pending inference requests
• inference_latency_p99 -- 99th percentile latency
• Custom metrics via Prometheus adapter

2. NeMo Agent Toolkit (Formerly AgentIQ)

Purpose: Build, connect, evaluate, and optimize teams of AI agents across any framework.

The NVIDIA NeMo Agent Toolkit (previously known as AgentIQ or AIQ Toolkit) is an open-source library that adds enterprise-grade instrumentation, observability, and continuous learning to AI agents. It is framework-agnostic, working alongside LangChain, LlamaIndex, CrewAI, Microsoft Semantic Kernel, Google ADK, and custom Python agents.

Exam Tip: The NCP-AAI exam may use the older name "AgentIQ" or "AIQ Toolkit" interchangeably with "NeMo Agent Toolkit." They refer to the same product.

Core Architecture Principles

NeMo Agent Toolkit treats agents, tools, and workflows as simple function calls, enabling true composability. This "build once, reuse anywhere" philosophy is central to its design.

Key capabilities:

• Framework-agnostic integration -- Wraps existing agents from any framework without rewriting
• Composability -- Agents and tools are interchangeable function-call primitives
• Observability -- Built-in tracing, metrics, and profiling for every agent interaction
• Continuous learning -- Automatic reinforcement learning to improve agent quality over time
• MCP and A2A support -- Publish workflows as MCP servers, coordinate distributed agents via A2A protocol

Project Scaffolding and Configuration

Setting up a new agent project uses the workflow create command:

# Create a new agent project scaffold
aiq workflow create --name my-agent-project

This generates a standard project structure:

my-agent-project/
├── pyproject.toml          # Plugin definitions and dependencies
├── config.yaml             # Workflow component configuration
├── src/
│   ├── agents/             # Agent definitions
│   ├── tools/              # Tool implementations
│   ├── workflows/          # Workflow orchestration
│   └── evaluations/        # Evaluation configs
└── tests/                  # Test suites

The config.yaml file defines the workflow components:

# config.yaml - NeMo Agent Toolkit workflow configuration
workflow:
  name: enterprise-rag-agent
  description: "RAG agent with tool calling and guardrails"

  llm:
    provider: nim
    model: meta/llama-3.1-70b-instruct
    endpoint: http://nim-llm:8000/v1

  tools:
    - name: vector_search
      type: retriever
      config:
        collection: enterprise_docs
        top_k: 5
    - name: calculator
      type: function
      module: src.tools.calculator

  memory:
    backend: redis
    ttl: 3600

  guardrails:
    config_path: ./guardrails/config.yml

Tool Registration

Tools in NeMo Agent Toolkit are registered as typed function calls with metadata:

from nemo_agent_toolkit import tool, ToolConfig

@tool(
    name="search_knowledge_base",
    description="Search internal knowledge base for relevant documents",
    config=ToolConfig(
        timeout=10.0,
        retries=3,
        cache_ttl=300
    )
)
def search_knowledge_base(query: str, top_k: int = 5) -> list[dict]:
    """Retrieve relevant documents from the vector store."""
    results = vector_store.similarity_search(query, k=top_k)
    return [{"content": r.page_content, "score": r.score} for r in results]

Exam Key Point: Tool registration in NeMo Agent Toolkit uses decorators with typed parameters. The framework automatically generates tool descriptions for the LLM from the function signature and docstring.

Memory Backends

NeMo Agent Toolkit supports multiple memory backends for agent state persistence:

• Redis -- Fast, in-memory store for conversation buffers and short-term memory
• PostgreSQL -- Durable storage for long-term agent memory and audit trails
• Vector databases (Milvus, ChromaDB, Pinecone) -- Semantic memory for RAG-based recall
• In-memory -- Development/testing only, no persistence across restarts

Exam Scenario: "A production agent needs conversation history that survives pod restarts in Kubernetes. Which memory backend is appropriate?" Answer: Redis or PostgreSQL -- in-memory backends lose state on restart. Redis provides the fastest access for conversation buffers, while PostgreSQL ensures durability for audit-critical deployments.

Evaluation Framework

NeMo Agent Toolkit includes a built-in evaluation system that functions as a verifier for reinforcement learning. This is a significant exam topic.

from nemo_agent_toolkit.evaluation import EvaluationSuite, metrics

suite = EvaluationSuite(
    name="rag-agent-evaluation",
    metrics=[
        metrics.answer_relevancy,      # Is the answer relevant to the query?
        metrics.faithfulness,           # Is the answer grounded in retrieved docs?
        metrics.tool_selection_accuracy,# Did the agent pick the right tool?
        metrics.latency_p95,           # Performance within SLA?
        metrics.cost_per_query,        # Token efficiency
    ],
    dataset="eval_dataset.jsonl"
)

results = suite.run(agent=my_agent)
print(results.summary())

Advanced feature -- Automatic Reinforcement Learning: NeMo Agent Toolkit can use evaluation results to fine-tune open LLMs via GRPO (with OpenPipe ART) or DPO (with NeMo Customizer), creating a continuous improvement loop where agent performance improves automatically based on evaluation signals.

LangGraph Automatic Wrapper

For teams with existing LangGraph agents, NeMo Agent Toolkit provides an automatic wrapper that adds observability and evaluation without rewriting agent code:

from nemo_agent_toolkit.wrappers import wrap_langgraph_agent

# Existing LangGraph agent -- no modification needed
wrapped_agent = wrap_langgraph_agent(
    existing_langgraph_agent,
    tracing=True,
    evaluation=True
)

Exam Key Point: NeMo Agent Toolkit does not replace existing frameworks. It wraps them to add enterprise capabilities. This is a common exam distinction -- the toolkit is complementary, not competitive, with LangChain, LlamaIndex, and CrewAI.

3. NeMo Guardrails

Purpose: Add programmable safety, compliance, and topical constraints to LLM-based agentic systems.

NeMo Guardrails is an open-source toolkit that uses Colang, an event-driven interaction modeling language, to define rules (rails) that govern agent behavior. The NCP-AAI exam tests both conceptual understanding and configuration-level knowledge of guardrails.

Colang 2.0 Syntax and Concepts

Colang 2.0 (introduced in NeMo Guardrails v0.8+) replaces the older Colang 1.0 with an event-driven architecture. The two core concepts are messages and flows.

Messages represent user and bot utterances:

define user ask about competitors
  "What do you think about [competitor]?"
  "How does [competitor] compare?"
  "Is [competitor] better?"

define bot refuse competitor discussion
  "I can only discuss our products and services.
   How can I help you with those?"

Flows define interaction patterns:

define flow handle competitor questions
  user ask about competitors
  bot refuse competitor discussion

Configuration file (config.yml):

# Enable Colang 2.0
colang_version: "2.x"

models:
  - type: main
    engine: nim
    model: meta/llama-3.1-70b-instruct

rails:
  input:
    flows:
      - check jailbreak
      - check toxicity
      - check pii
  output:
    flows:
      - check hallucination
      - check sensitive topics
      - enforce response format

Rail Types and Chains

The NCP-AAI exam distinguishes between several types of rails. Understanding the execution order is critical.

Input Rails -- Validate and filter user messages before they reach the LLM:

define flow check pii
  """Block messages containing personal identifiable information."""
  user said $message
  if contains_pii($message)
    bot say "I cannot process messages containing personal information.
             Please remove any SSNs, credit card numbers, or addresses."
    stop

Output Rails -- Validate and filter LLM responses before they reach the user:

define flow check hallucination
  """Verify that responses are grounded in retrieved context."""
  bot said $response
  $grounded = check_grounding($response, $retrieved_context)
  if not $grounded
    bot say "I don't have enough information to answer that accurately.
             Let me search for more details."
    stop

Topical Rails -- Keep the agent focused on its designated domain:

define flow enforce topic boundaries
  """Prevent the agent from discussing off-topic subjects."""
  user said $message
  $is_on_topic = check_topic($message, allowed_topics=["product support",
    "billing", "technical documentation"])
  if not $is_on_topic
    bot say "I'm specialized in product support, billing, and technical
             documentation. How can I help with those topics?"
    stop

Retrieval-Augmented Rails -- Use a knowledge base to validate responses:

define flow retrieval_augmented_check
  """Cross-reference responses against approved knowledge base."""
  bot said $response
  $facts = retrieve_from_kb($response, knowledge_base="approved_facts")
  $consistency = check_consistency($response, $facts)
  if $consistency < 0.85
    bot say "Let me verify that information..."
    $corrected = generate_from_facts($facts)
    bot say $corrected

Parallel Rails Execution

Recent NeMo Guardrails releases support parallel execution of rails, reducing latency when multiple rails are configured. Instead of running input rails sequentially (check jailbreak, then check toxicity, then check PII), they execute concurrently:

• Sequential execution: Total latency = sum of all rail latencies
• Parallel execution: Total latency = maximum of any single rail latency

Exam Key Point: Parallel rails execution is a performance optimization, not a safety compromise. All rails must still pass before the message proceeds.

4. NVIDIA AI Workbench

Purpose: Developer toolkit for creating, customizing, and running AI projects across local and cloud environments.

AI Workbench became generally available with NVIDIA AI Enterprise 5.0 and is the recommended development environment for building agentic AI applications.

Project Structure

An AI Workbench project is a structured Git repository with a .project/spec.yaml file that declares the containerized development environment:

agent-project/
├── .project/
│   └── spec.yaml           # Environment specification
├── code/                    # Application source code
│   ├── agents/
│   ├── tools/
│   └── workflows/
├── models/                  # Local model files or NIM configs
├── data/                    # Training data, eval datasets
├── scratch/                 # Temporary/experimental work
└── README.md

The spec.yaml file controls the build environment:

# .project/spec.yaml
specVersion: v2
meta:
  name: agentic-ai-project
  description: "NCP-AAI agent development project"
environment:
  base: nvcr.io/nvidia/ai-workbench/pytorch:latest
  variables:
    NIM_ENDPOINT: http://localhost:8000
  packages:
    pip:
      - nemo-agent-toolkit>=1.5
      - nemo-guardrails>=0.10
      - langchain>=0.3

Hybrid Workflow

The "hybrid" in AI Workbench refers to seamless transitions between compute environments:

1. Local development -- Prototype on a laptop or RTX workstation with smaller models
2. Cloud scale-up -- Push the same project to a cloud instance with H100 GPUs for full-scale testing
3. Production deployment -- Deploy to Kubernetes with NIM Operator from the same codebase

Key features:

• Automatic GPU configuration -- Cloned projects auto-detect and configure available GPUs
• Docker Compose support -- Multi-container environments for complex agent pipelines
• Application sharing -- Share running applications via single-user URLs for team review
• Git-native -- Branching, merging, and diffs integrated into the workflow
• Environment reproducibility -- Containerized environments ensure consistency across machines

Exam Scenario: "A developer prototypes an agent on a local RTX 4090 and needs to test with a 70B model on cloud GPUs. Which tool provides this workflow?" Answer: NVIDIA AI Workbench -- the hybrid workflow allows developing locally and scaling to cloud instances without changing the project structure.

Enterprise Deployment Architecture

Production Stack:

User Requests
    ↓
Load Balancer (NGINX / K8s Ingress)
    ↓
NeMo Guardrails (Input Rails - Safety Check)
    ↓
NVIDIA NIM (LLM Inference)
    ↓
NeMo Agent Toolkit (Orchestration)
    ↓
Tool Execution Layer
    ├─ Internal APIs
    ├─ Vector Databases (Milvus)
    └─ External Services
    ↓
NeMo Guardrails (Output Rails - Safety Check)
    ↓
Response to User

Exam Focus: Understand the three-tier architecture: inference (NIM), orchestration (NeMo Agent Toolkit), and safety (NeMo Guardrails at both input and output).

Enterprise Monitoring with NVIDIA DCGM

What is DCGM?

NVIDIA Data Center GPU Manager (DCGM) is a suite of tools for managing and monitoring NVIDIA datacenter GPUs in cluster environments. For agentic AI deployments, DCGM provides the telemetry needed to maintain SLAs, optimize costs, and diagnose performance issues.

Key Metric Categories

Utilization Metrics:

• DCGM_FI_DEV_GPU_UTIL -- GPU compute utilization percentage (0-100%)
• DCGM_FI_DEV_MEM_COPY_UTIL -- Memory bandwidth utilization percentage
• DCGM_FI_PROF_SM_ACTIVE -- Ratio of cycles an SM has at least 1 warp assigned
• DCGM_FI_PROF_PIPE_TENSOR_ACTIVE -- Tensor Core (HMMA pipe) utilization ratio

Memory Metrics:

• DCGM_FI_DEV_FB_FREE -- Free framebuffer memory (MiB)
• DCGM_FI_DEV_FB_USED -- Used framebuffer memory (MiB)

Thermal and Power:

• DCGM_FI_DEV_GPU_TEMP -- GPU temperature (Celsius)
• DCGM_FI_DEV_MEM_TEMP -- Memory temperature (Celsius)
• DCGM_FI_DEV_POWER_USAGE -- Current power draw (Watts)
• DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION -- Total energy since boot (mJ)

Clock Frequencies:

• DCGM_FI_DEV_SM_CLOCK -- SM clock frequency (MHz)
• DCGM_FI_DEV_MEM_CLOCK -- Memory clock frequency (MHz)

DCGM Integration with Prometheus and Grafana

DCGM includes the dcgm-exporter that exposes GPU metrics as Prometheus endpoints. The default sampling rate is 1 Hz (every 1000ms), configurable down to a minimum of 100ms.

# dcgm-exporter Kubernetes DaemonSet (simplified)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
spec:
  template:
    spec:
      containers:
      - name: dcgm-exporter
        image: nvcr.io/nvidia/k8s/dcgm-exporter:latest
        ports:
        - containerPort: 9400
          name: metrics

Inference Throughput Dashboard Metrics:

• Tokens per second -- Throughput of the NIM inference endpoint
• KV-cache utilization -- Percentage of GPU memory used for key-value cache
• Request queue depth -- Number of pending inference requests
• Time-to-first-token (TTFT) -- Latency from request to first generated token
• Inter-token latency (ITL) -- Time between consecutive generated tokens

Exam Key Point: DCGM monitors the GPU hardware. NIM exposes inference-level metrics. Production dashboards combine both layers for comprehensive visibility. The NCP-AAI exam may ask which metric indicates GPU memory pressure (answer: DCGM_FI_DEV_FB_USED approaching total framebuffer capacity) vs. inference bottleneck (answer: gpu_cache_usage_perc from NIM).