Preporato
Multimodal RAG with NeMo Retriever
Hands-on lab · Runs in your browser

Multimodal RAG with NeMo Retriever

Build an image-query RAG system: embed a catalog with NeMo Retriever, translate an uploaded image into a retrieval query via a VLM, and ground the VLM's final answer in the retrieved passages.

Time
35 min
Checked steps
4
Level
Intermediate
Setup
None
ncp-aainca-genm
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

Lab cockpit35 min · 4 stepsSession running
2 / 4 steps passingDescribe the image, then retrieve · step 3 of 4
main.py▶ Run✓ Check
make_chair_image() def data_url(p):    with open(p,'rb') as f: return 'data:image/jpeg;base64,'+base64.b64encode(f.read()).decode() def image_to_query(url: str) -> str:
TerminalOutput

4 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Embed a product corpus

    NVIDIA's llama-nemotron-embed-1b-v2 is a 2048-dim text embedding model optimized for retrieval.

  2. 2

    Cosine-similarity retriever

    For a catalog of <1M rows, cosine similarity over an in-memory matrix is fast enough and lets you stay focused on the retrieval *pattern* rather than a vector DB.

  3. 3

    Describe the image, then retrieve

    The VLM acts as an image-to-text translator: take a user-uploaded image, describe it in words, feed those words into the retriever.

  4. 4

    End-to-end grounded answer

    Now close the loop.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Embed a product corpus

NeMo Retriever embeddings

NVIDIA's llama-nemotron-embed-1b-v2 is a 2048-dim text embedding model optimized for retrieval. The NIM API follows the OpenAI /v1/embeddings spec, but adds an input_type parameter you should always set:

  • input_type: "passage" when indexing documents
  • input_type: "query" when embedding a user query

This asymmetry matters — using the wrong one degrades recall by ~10–20%.

Your task

Add to main.py:

  1. Build CORPUS: at least 5 short product descriptions (1–2 sentences each). Mix categories: furniture, electronics, kitchenware.
  2. Call the embedding endpoint with input_type="passage" and store the result as a numpy array passage_embeddings of shape (N, 2048).
  3. Print the shape.
main.py, the file you edit16 lines
from openai import OpenAI
import numpy as np

client = OpenAI(base_url="http://nim-proxy.labs.svc:8080/v1", api_key="nvapi-inject")
EMBED_MODEL = "nvidia/llama-nemotron-embed-vl-1b-v2"

# TODO: define CORPUS with >=5 product descriptions
CORPUS = [
    # "Product N: description ...",
]

# TODO: embed with input_type="passage" and store as numpy array
passage_embeddings = ______

print('corpus size:', len(CORPUS))
print('embedding shape:', passage_embeddings.shape)

Exam domains covered

Retrieval-Augmented GenerationMultimodal AINVIDIA Platform Implementation

Frequently asked questions

What does image-as-query actually mean in a RAG pipeline?

Traditional RAG takes a text question, embeds it, retrieves text chunks, and generates a text answer. Image-as-query flips the input: the user uploads a photo instead of typing. The pipeline converts the image into something the retriever understands — in this lab, a natural-language description produced by a VLM — and the rest of retrieval looks identical. The VLM is an image-to-text translator; the retriever is still text-over-text. This is the simplest and most portable multimodal RAG pattern.

Why use a VLM to describe the image instead of an image embedding model directly?

Image embedders like CLIP produce a vector that lives in a joint image-text space, which works if your corpus is also embedded in that space. This lab's corpus is text embedded with llama-nemotron-embed-1b-v2 (a 2048-dim text-only model), so a CLIP vector wouldn't match. Using a VLM to generate a text description and then embedding that description lets you reuse your existing text retriever unchanged — same vector DB, same index, same similarity metric — and usually produces better recall on descriptive queries than a raw image vector would.

Why does input_type matter so much for NeMo Retriever embeddings?

llama-nemotron-embed-1b-v2 is asymmetric — it prepends different instruction prefixes at encoding time for passages versus queries, and it's trained so that a passage vector and a query vector of semantically related text are close in cosine space. Pass input_type="query" when indexing and you get back vectors that don't align with the passage distribution; recall drops ~10–20% on retrieval benchmarks. The lab enforces the distinction by having you implement separate embed_corpus and embed_query calls.

Do I need a vector database for this lab?

No. The corpus is small — a handful of product descriptions — so cosine-over-a-numpy-matrix is fast enough and keeps the lab focused on the retrieval pattern rather than database operations. Because the embeddings come back L2-normalized from NeMo Retriever, cosine similarity collapses to a single dot product (CORPUS_VECTORS @ query_vec), which is one numpy call. The rag-pipeline-nim lab covers the Milvus side for when your corpus outgrows memory.

How does Step 4's end-to-end answer differ from just returning the top retrieval?

The top retrieval gives you the single best-matching product description. Step 4's answer_with_rag goes further: it builds a chat completion with the original image in the content parts and the top-k retrieved passages as context, then asks the VLM a grounded question like "which of these products is the closest match, and why?" The VLM now answers with both pixels and text in view — it can reason about color, shape, or framing from the image and reference specific product attributes from the retrieved text. This is the full multimodal RAG loop, not just retrieval.

What's the difference between this lab and rag-pipeline-nim?

rag-pipeline-nim is text-in/text-out RAG built on Milvus Lite and an agentic ReAct loop — the user asks a typed question, the agent decides whether to retrieve, and the answer is grounded in text chunks. This lab keeps the same retrieval concept but makes the query image-based and uses NeMo Retriever's asymmetric text embeddings plus a VLM for image-to-query translation. They're meant to be taken in order: the text pipeline teaches retrieval fundamentals, this one layers multimodality on top.

What you'll build in this multimodal RAG lab

Multimodal RAG — where the user's query is an image rather than text — is what makes e-commerce visual search, ticket triage from screenshots, and document Q&A from photographed forms actually work. This lab builds an image-as-query retrieval pipeline against NeMo Retriever and a Nemotron VLM, both served via NVIDIA NIM endpoints we provision. You walk away with a working answer_with_rag(image, question) function, the mental model for when to use a VLM as an image-to-text translator versus a joint image-text embedding model, and a pattern that ports directly onto production multimodal search.

The technical substance is the three-layer pipeline that makes this reliable. Retrieval uses llama-nemotron-embed-1b-v2 with the asymmetric input_type flag — "passage" at indexing, "query" at lookup — to produce 2048-dim vectors, and you do cosine similarity as a single dot product over the L2-normalised corpus matrix. The VLM — google/gemma-4-31b-it — acts as an image-to-text translator: image_to_query(data_url) turns a photo of a dark-brown wooden chair into a natural-language description that matches a corpus entry about the MidnightOak chair. The final step composes both — a chat completion with the original image part and the top-k retrieved passages both in scope — so the VLM grounds its answer in pixels and text simultaneously, dodging the classic RAG hallucination mode.

Prerequisites: the rag-pipeline-nim lab (or equivalent text RAG exposure), the vlm-visual-qa lab for VLM basics, and numpy comfort with dot products and L2 normalisation. The hosted environment ships with the OpenAI Python SDK and numpy preinstalled, pointing at our managed NIM proxy where the embedder, VLM, and generator all share the same endpoint — no keys, no GPU pod. About 35 minutes of focused work. You leave with a 2048-dim corpus matrix, an asymmetric-aware retriever, a VLM-powered image-to-query translator, and an end-to-end grounded answer that references the retrieved product by name — the exact shape production visual search shops in.