Step 1: Load Qwen2-VL and a test image
Vision-Language Models (VLMs) take an image plus text and output text. Captioning ("describe this image"), Visual Question Answering ("what color is the cat?"), OCR, document understanding, chart reading, GUI agents — all the same architectural pattern:
image → [vision encoder] → visual tokens ┐
├─→ [LLM] → text answer
text → [tokenizer] → text tokens ┘
The central insight is a modality projector: a tiny MLP (sometimes just one linear layer) that maps visual features into the LLM's token embedding space. To the LLM, the image becomes a short sequence of "visual tokens" it treats exactly like text tokens. Everything else is standard attention.
The key papers that built this stack
- CLIP (Radford et al., 2021) — OpenAI's contrastive image-text pretraining. Produces the vision encoder that most modern VLMs still reuse (or its SigLIP descendant).
- BLIP-2 (Li et al., 2023) — Salesforce, 2023. Introduced the Q-Former: a small querying transformer bridging frozen CLIP to a frozen LLM with just ~100M trainable params.
- LLaVA (Liu et al., 2023) — the open-source alternative: CLIP + MLP projector + Llama. Trained with synthetic instruction data generated by GPT-4. Arguably the most influential open VLM paper.
- Qwen-VL (Bai et al., 2023) / Qwen2-VL (Wang et al., 2024) — Alibaba's VLM family. As of 2024 the best open-weight VLM under 10B params.
- SigLIP (Zhai et al., 2023) — CLIP with sigmoid loss; the default vision encoder in most 2024+ VLMs.
- InternVL (Chen et al., 2024) / LLaVA-OneVision (Li et al., 2024) — current open SOTA, video + image + multi-image in one model.
The model we'll use
Qwen2-VL-2B-Instruct — Alibaba's 2B-parameter open VLM with:
- Custom ViT-based vision encoder (~600M params)
- Qwen2 1.5B language model
- A simple MLP merging visual tokens into the LM's embedding space
In fp16 it takes ~4 GB — fits comfortably in 8 GB. It supports the chat-template interface, so we can talk to it like a normal instruct model.
What you'll do
- Load Qwen2-VL-2B-Instruct, load a test image
- Caption the image
- Visual QA — ask 3+ questions, get answers
- Inspect the architecture — count params in vision encoder, projector (merger), and LM
import torch
import io
from PIL import Image
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
device = torch.device('cuda')
MODEL_ID = 'Qwen/Qwen2-VL-2B-Instruct'
print('Loading Qwen2-VL-2B-Instruct (first run downloads ~4 GB)...')
vlm = Qwen2VLForConditionalGeneration.from_pretrained(
MODEL_ID, torch_dtype=torch.float16,
).to(device)
vlm.eval()
vlm_tokenizer = AutoProcessor.from_pretrained(MODEL_ID)
print(f'Qwen2-VL-2B: {sum(p.numel() for p in vlm.parameters())/1e9:.2f}B params on {device}')
# Famous CLIP-test image: two cats on a couch with remote controls (COCO val).
# Pre-downloaded to the network volume at setup time because lab pods have no internet.
test_image = Image.open('/workspace/assets/coco-two-cats.jpg').convert('RGB')
print(f'Loaded test image: {test_image.size[0]}x{test_image.size[1]} {test_image.mode}')
test_image
