Deploy & Serve LLMs in Production (Jupyter)

The Problem

Loading a model with transformers and calling model.generate() works for experimentation, but it's terrible for production:

• Sequential — one request at a time, everyone else waits
• No batching — GPU sits idle between tokens
• Slow — no kernel optimizations, no continuous batching

Load Llama 3 8B Instruct with plain transformers and time 5 sequential requests. Note the average — we'll compare to vLLM next.

import torch
import time
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "/models/meta-llama--Meta-Llama-3-8B-Instruct"

print("Loading Llama 3 8B Instruct with transformers...")
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.float16,
    device_map="auto",
)
print(f"Model loaded: {sum(p.numel() for p in model.parameters()) / 1e6:.0f}M params")
print(f"VRAM: {torch.cuda.memory_allocated() / 1e9:.1f} GB")

test_prompts = [
    "What are the symptoms of diabetes?",
    "Explain how a transformer model works in 3 sentences.",
    "Write a Python function to calculate fibonacci numbers.",
    "What is the difference between TCP and UDP?",
    "Summarize the key ideas of reinforcement learning.",
]

print("Running 5 sequential requests...\n")
naive_times = []
for i, prompt in enumerate(test_prompts):
    messages = [{"role": "user", "content": prompt}]
    input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

    start = time.time()
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=200,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    elapsed = time.time() - start

    response = tokenizer.decode(output[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
    tokens = len(tokenizer.encode(response))
    naive_times.append(elapsed)
    print(f"  Request {i+1}: {elapsed:.1f}s ({tokens} tokens, {tokens/elapsed:.0f} tok/s)")

naive_avg = sum(naive_times) / len(naive_times)
print(f"\nAverage: {naive_avg:.1f}s per request")
print(f"Total for 5 requests: {sum(naive_times):.1f}s (sequential)")

Free the GPU for vLLM

Naive loading took all the VRAM. Before launching vLLM, free it up.

Heads up: In Jupyter, del model alone doesn't free GPU memory because IPython's output history (the Out[] dict and _, __, ___) silently holds references to anything a cell returned. We must clear those too.

import gc
from IPython import get_ipython

# Drop all direct references (tokenizer, model, tensors from the benchmark loop)
for _name in ['model', 'tokenizer', 'inputs', 'output', 'outputs', 'response']:
    if _name in globals():
        del globals()[_name]

# Critical: clear IPython's output history — it holds refs to previous cell outputs
ip = get_ipython()
if ip is not None:
    ip.run_line_magic('reset', '-f out')  # clears Out[] and _, __, ___

gc.collect()
torch.cuda.empty_cache()
torch.cuda.synchronize()

print(f"VRAM after cleanup: {torch.cuda.memory_allocated() / 1e9:.2f} GB (allocated)")
print(f"VRAM reserved:       {torch.cuda.memory_reserved() / 1e9:.2f} GB")