Send one image and ask a question. Our run: There is a blue square in the center of a yellow background.
Intermediate · in your browser
Machine learning and deep learning
Visual Q&A with NVIDIA VLMs
Send images to a Vision-Language Model via NIM, answer questions about them, extract structured fields from a receipt-style image, and compare two VLMs on the same task, all through the OpenAI-compatible chat endpoint.
No account needed. Pro opens all 4 steps. See Pro plans
The 4 steps
each one checked
A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.
- 1
Send one image and ask a question
Vision-Language Models use the same OpenAI-compatible chat endpoint as text-only models, but the content field inside a user message becomes a list of content parts:
Read this step
Calling a VLM
Vision-Language Models use the same OpenAI-compatible chat endpoint as text-only models, but the
contentfield inside a user message becomes a list of content parts:{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,<...>"}}, ] }You can pass an image as a data URL (base64-inlined) or as an
https://URL if the model can reach it. Data URLs are the reliable choice inside a lab.The lab image
The starter code writes a small test image to
/tmp/test_img.jpg: a blue square on a yellow background. Your job is to describe it correctly.Choosing a VLM
We'll use
google/gemma-4-31b-it, a multimodal model that handles charts, screenshots and documents and, unlike most vision models, also supports function calling. That second property is what step 4 turns into a lesson. Give it ~800 tokens of headroom.NVIDIA retired its own hosted Nemotron vision model, and no NVIDIA vision model is currently servable. That is itself the lesson: pin a hosted model in your code and you inherit its retirement date.
Your task
Add to
main.py:- Read
/tmp/test_img.jpg, base64-encode it, prefix withdata:image/jpeg;base64,. - Build a chat request with one
textpart and oneimage_urlpart. - Call
google/gemma-4-31b-itwithmax_tokens=800. - Print the answer.
main.py, the file you edit28 linesfrom openai import OpenAI from PIL import Image, ImageDraw import base64 # Create a test image: yellow background with a blue square in the middle def make_image(path="/tmp/test_img.jpg"): img = Image.new('RGB', (300, 200), color='yellow') ImageDraw.Draw(img).rectangle([100, 60, 200, 140], fill='blue', outline='black', width=3) img.save(path, 'JPEG', quality=85) return path def image_to_data_url(path: str) -> str: with open(path, 'rb') as f: b64 = base64.b64encode(f.read()).decode() return f"data:image/jpeg;base64,{b64}" make_image() data_url = image_to_data_url("/tmp/test_img.jpg") client = OpenAI(base_url="http://nim-proxy.labs.svc:8080/v1", api_key="nvapi-inject") VLM_MODEL = "google/gemma-4-31b-it" # TODO: describe_image(prompt, data_url) -> str def describe_image(prompt: str, data_url: str) -> str: ______ answer = describe_image("In one sentence: what shape is in the center, what is its color, and what is the background color?", data_url) print(answer) - Read
- 2
Reason across two images
A VLM can compare images in a single turn, you simply pass more than one image_url part in the content list.
- 3
Structured extraction from an image
VLMs can produce structured output via the same tools / tool_choice primitives you used in structured-output-tools, the schema is enforced at the API boundary, so you skip the regex-and-pray parsing dance.
- 4
Compare two VLMs with different extraction strategies
Not every VLM supports the tools / tool_choice API.
Before you start
3 things
- Completed `react-agent-nim` or comparable NIM exposure
- Comfortable reading/writing base64 image payloads
- Familiarity with JSON
What you get
- A hosted environment in your browser
- A check at the end of every step
- Hints and the full solution
- Nothing to install or configure
Exam domains covered
NCP-AAI, NCA-GENM, NCA-AMAD
The Machine learning and deep learning track
lab 16 of 17
scikit-learn and PyTorch on real datasets, with the metrics and debugging that decide what ships.
- Beginner · 50 min · In browser · ProYour First Machine Learning Model: Predict Churn with scikit-learn
- Beginner · 45 min · In browser · ProMetrics That Match the Business: Choose a Threshold from a Cost Table
- Beginner · 50 min · In browser · ProData Cleaning and Feature Engineering: Predict Rents From a Messy Export
- Intermediate · 45 min · In browser · ProOverfitting and Cross-Validation: Find What Really Makes a Better Cup
- Intermediate · 50 min · In browser · ProGradient Boosting with XGBoost: Predict Late Deliveries and Explain Every Prediction
- Intermediate · 50 min · In browser · ProCustomer Segmentation with k-means and PCA: Scaling, Choosing k, Stability and Deployment
- Intermediate · 50 min · In browser · ProTime-Series Forecasting: Baselines, Backtesting, Calendar Features, Prediction Intervals and the Newsvendor Rule
- Intermediate · 55 min · In browser · ProImbalanced Data and Anomaly Detection: Average Precision, Rebalancing, Alert Budgets and Unseen Failures
- Intermediate · 50 min · In browser · ProExplain a Model: Permutation Importance, Target Leakage, Partial Dependence and Exact Shapley Values
- Intermediate · 60 min · In browser · ProBuild a Recommender: Collaborative Filtering, Matrix Factorization and the Metrics That Disagree
- Beginner · 45 min · In browser · ProPyTorch in 45 Minutes: Tensors, Autograd and Your First Training Loop
- Beginner · 45 min · In browser · ProA PyTorch Training Loop From Scratch: Datasets, Epochs, Evaluation and Early Stopping
- Intermediate · 50 min · In browser · ProDebug a PyTorch Training Run: NaN Loss, Broadcasting Bugs, Learning Rate and Data Leaks
- Intermediate · 50 min · In browser · ProImage Classification with Transfer Learning in PyTorch: From a Small CNN to a Fine-Tuned ResNet
- Intermediate · 35 min · NVIDIA GPU · ProVision-Language Models: Captioning and Visual QA
- this labVisual Q&A with NVIDIA VLMs
- Intermediate · 55 min · In browser · ProReinforcement Learning Basics: Q-Learning and Reward Shaping on a Gridworld
More lab paths
1
Questions about this lab
6 answers
How do I pass an image to a VLM through the chat completions API?
Swap the content string on the user message for a list of content parts. Each part has a type field: type: "text" for the prompt, type: "image_url" for an image. The image_url can be an https:// URL the model can fetch, or a data URL like data:image/jpeg;base64,<b64>, the base64-inlined form is the reliable choice inside a lab because there's no network fetch to depend on. Nothing else about the request changes, model, messages, temperature, and the response format are the same chat completion you already know.
Can a VLM reason about multiple images in a single turn?
Yes. Put more than one image_url part in the content list of a single user message and the VLM attends to all of them inside one context. Step 2 of this lab puts a 3-circle image next to a 5-circle image and asks which has more; the model sees both and answers from the joint context. This is the primitive behind comparison tasks, A/B screenshots, receipt vs. ledger entry, before/after photos, and it scales up to any number of images the model's context window can hold.
Why does function calling work with one vision model but not the other?
Function calling is a per-model capability, not a blanket feature of multimodal endpoints. google/gemma-4-31b-it ships with tools support, so Step 3 gets back a clean tool_calls with validated arguments. google/gemma-3-12b-it does not: it answers in prose and ignores the schema, so a parser expecting JSON gets free text. Step 4 uses this split deliberately: the agent wrapper you build has to detect which mode is available and fall back to prompt-only JSON when tools aren't supported.
Is prompt-only JSON extraction reliable enough on a VLM?
Less reliable than function calling, but workable for a single well-scoped schema. On the fake receipt in Step 4, google/gemma-3-12b-it asked for JSON in prose produces mostly-correct output most of the time, but you need defensive parsing (strip markdown fences, tolerate trailing commas, null-check every field) and you should treat extraction failures as expected. For complex schemas or production-scale extraction you'd want a VLM that supports tools, or you'd use a VLM to describe the image and a tool-capable text model to do the actual structured extraction.
How is the Step 3 structured extraction different from a text-only tool call?
Mechanically it's identical, same tools parameter, same tool_choice, same function.arguments string on the return, but the model is looking at pixels to fill the schema instead of reading text. That means extraction accuracy depends on image quality and rendering style, not just language. The lab deliberately draws a clean, high-contrast receipt so the vision stage isn't the bottleneck and you can focus on the schema-enforcement pattern. Noisy receipts would add an OCR-style error mode on top.
What's the difference between this VLM lab and the multimodal RAG one?
This lab focuses on VLM core capabilities: captioning, multi-image comparison, and schema-enforced extraction from a single image. The multimodal-rag lab layers a VLM into a retrieval pipeline, the image becomes a query that gets translated into text, retrieved against a product corpus, and grounded in the top-k passages. Here the VLM answers from pixels alone; there the VLM is one component in a larger retrieval-augmented system. The lab order matters: you need the primitives from this one to understand how they plug into the RAG one.
What you'll build in this VLM visual Q&A lab
about
Vision-Language Models are the fastest-growing surface in production LLM apps, receipt parsing, screenshot triage, document extraction, multimodal agent input, and the tool that separates teams who can ship these features from teams still trying to bolt OCR together. This lab goes from sending a single image into a VLM all the way to a head-to-head comparison between two VLMs on a structured-extraction task, all against NVIDIA NIM endpoints we provision. You finish with working code for single-image Q&A, multi-image reasoning, schema-enforced extraction from a receipt, and a tool-tolerant helper that handles the reality that not every VLM supports function calling.
The technical core is the OpenAI-compatible multimodal content format, text and image_url parts coexist inside a single user message, the VLM fuses them into one context, and the response surface is the exact same chat completion shape text-only calls return. You'll work through base64 data URL payloads (the reliable image-payload choice when you don't want a network fetch in the loop), multi-image reasoning that's just more parts in the list rather than a separate API, and schema-enforced extraction using tools plus tool_choice for a save_receipt function with fields vendor, order_id, line items, subtotal, tax, total. The final step runs the same receipt through the tool-capable model with function calling and through google/gemma-3-12b-it with prompt-only JSON, because the second model ignores the tool schema and answers in prose when you pass tools, a real production split your code needs to handle.
Prerequisites: Python, prior NIM exposure (the react-agent-nim lab works), base64 payload comfort, and basic JSON. No VLM-specific library is assumed, everything goes through the OpenAI Python SDK pointed at our managed NIM proxy, where both google/gemma-4-31b-it and google/gemma-3-12b-it are reached via the same OpenAI-compatible endpoint with no GPU provisioning. About 30 minutes of focused work. You leave with a single-image Q&A call, a multi-image comparison, schema-validated extraction via function calling, and a dual-path extractor that falls back to prompt-only JSON when tools aren't supported, the same defensive shape production VLM code needs.