Fine-Tune an LLM with LoRA and QLoRA (Jupyter)

What You'll Build

In this lab, you'll take Meta Llama 3 8B — a raw base model with no instruction-following ability — and fine-tune it into a medical Q&A specialist. By the end, your model will go from incoherent text completion to structured, accurate clinical answers.

The journey:

1. Explore what the base model can and can't do (spoiler: it can't follow instructions)
2. Prepare a training dataset in the right format
3. Understand LoRA — how and why it works
4. Run QLoRA fine-tuning and watch loss decrease
5. Understand quantization — FP16 vs 4-bit
6. Evaluate: see the dramatic before/after difference
7. Merge and export your fine-tuned model

---

Before fine-tuning, you need to understand what the model can't do. The base Llama 3 8B was trained on raw text — it predicts the next token, nothing more. It has never been taught to follow instructions or answer questions.

Ask it "What is diabetes?" and it'll respond with something like "What is diabetes? It is a disease that..." — just continuing the text pattern, not answering your question.

This is what makes fine-tuning so valuable: you take this raw capability and shape it into something useful.

Load the base model

Let's load Llama 3 8B (the base version — no instruction tuning).

In Hugging Face Transformers, you load models and tokenizers with from_pretrained():

tokenizer = AutoTokenizer.from_pretrained("model-path")
model = AutoModelForCausalLM.from_pretrained("model-path", ...)

Both are already imported above. The model path is stored in model_name. For loading the model, pass these two keyword arguments:

• dtype=torch.float16 — half precision, uses ~16 GB VRAM instead of ~32 GB
• device_map="auto" — automatically places layers on available GPUs

Fill in the two blanks:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "/models/meta-llama--Meta-Llama-3-8B"

# Load the tokenizer
tokenizer = ______
tokenizer.pad_token = tokenizer.eos_token

# Load the model in float16 on GPU
model = ______

print(f"Model loaded: {sum(p.numel() for p in model.parameters()) / 1e6:.0f}M parameters")
print(f"VRAM used: {torch.cuda.memory_allocated() / 1e9:.1f} GB")

Test the base model

Now let's send medical questions to the base model. Pay attention to how it responds — it should not answer the questions properly. Instead, it will just continue the text pattern (like autocomplete).

This is the behavior we'll fix through fine-tuning. Run the cell below (no changes needed).

prompts = [
    "What are the first-line treatments for Type 2 diabetes?",
    "A patient presents with sudden chest pain and shortness of breath. What is the most likely diagnosis?",
    "List the contraindications for metformin.",
]

for prompt in prompts:
    print(f"\n{'='*60}")
    print(f"Q: {prompt}")
    print(f"{'='*60}")

    # Base model has no chat template — just feed raw text
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=150,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )

    response = tokenizer.decode(output[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
    print(f"A: {response}")

print("\n\nNotice: the base model doesn't answer questions — it just completes text.")
print("After fine-tuning, it will actually follow instructions and give structured answers.")

See how the base model just rambles? It might repeat the question, continue with loosely related text, or go off on tangents. It has knowledge about medicine (it was trained on internet text) but it has no ability to follow instructions.

By Step 6, your fine-tuned model will answer these same questions with structured, concise responses.

Click Check Step 1 below when you're ready to continue.