NCP-GENL Fine-Tuning Guide: LoRA, QLoRA & PEFT for Production LLMs

When to Use QLoRA vs LoRA vs Full Fine-Tuning

Other PEFT Methods

The exam tests awareness of PEFT methods beyond LoRA, though LoRA/QLoRA questions dominate.

Prefix Tuning

Prepends learnable "virtual tokens" (prefix vectors) to the input at each transformer layer. The model processes these virtual tokens alongside the real input, learning task-specific conditioning.

Trainable parameters: prefix_length x hidden_dim x n_layers. For prefix length 20, hidden dim 4096, 32 layers: 20 x 4096 x 32 = 2.6M parameters.

When to use: Sequence-to-sequence tasks, translation, summarization. Less effective than LoRA for instruction following and chat.

Adapter Layers

Inserts small bottleneck layers (down-projection, nonlinearity, up-projection) between existing transformer layers. Each adapter typically has a hidden dimension of 64-256.

Trainable parameters: 2 x adapter_dim x hidden_dim x n_adapters x n_layers. For adapter_dim=64, hidden_dim=4096, 2 adapters per layer, 32 layers: 2 x 64 x 4096 x 2 x 32 = 33.6M parameters.

When to use: Tasks requiring additional representational capacity beyond what LoRA provides. Adapters add new parameters rather than modifying existing weight matrices.

IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations)

Learns per-element scaling vectors for keys, values, and feed-forward activations. Even fewer parameters than LoRA — scales rather than adds.

Trainable parameters: (d_k + d_v + d_ff) x n_layers. Dramatically smaller than LoRA but less expressive.

Fine-Tuning with NVIDIA NeMo Framework

NVIDIA NeMo is the primary framework for fine-tuning LLMs on NVIDIA hardware. The exam tests NeMo-specific configuration and workflow.

NeMo Fine-Tuning Pipeline

1. Prepare dataset (JSONL format)
   → {"input": "instruction", "output": "response"}

2. Load base model (.nemo checkpoint)
   → Llama, Mistral, Nemotron, etc.

3. Configure PEFT method
   → LoRA rank, alpha, target modules

4. Configure training
   → Learning rate, batch size, epochs, scheduler

5. Train with NeMo Launcher
   → Single-GPU or distributed (Megatron-LM backend)

6. Export adapter
   → Small .nemo file (50-200MB)

7. Deploy
   → Merge adapter into base or serve separately via Triton/NIM

NeMo LoRA Configuration

from nemo.collections.nlp.models.language_modeling import MegatronGPTSFTModel

# PEFT configuration
peft_cfg = {
    "peft_scheme": "lora",
    "lora_tuning": {
        "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"],
        "adapter_dim": 16,       # rank
        "alpha": 32,             # alpha = 2 * rank
        "adapter_dropout": 0.05,
    }
}

# Training configuration
training_cfg = {
    "trainer": {
        "max_epochs": 3,
        "precision": "bf16-mixed",
        "devices": 1,             # single GPU for LoRA
        "accumulate_grad_batches": 4,
    },
    "model": {
        "learning_rate": 2e-4,    # higher LR safe with LoRA
        "weight_decay": 0.01,
        "warmup_steps": 100,
    }
}

NeMo Data Format

NeMo expects data in JSONL format for supervised fine-tuning (SFT):

{"input": "Summarize the following research paper abstract:", "output": "The paper presents..."}
{"input": "Translate to French: The weather is nice today.", "output": "Le temps est beau aujourd'hui."}

For instruction tuning (chat format):

{"conversations": [{"role": "user", "content": "Explain quantization."}, {"role": "assistant", "content": "Quantization reduces..."}]}

Catastrophic Forgetting Prevention

Catastrophic forgetting occurs when fine-tuning causes the model to lose previously learned capabilities. The exam tests multiple prevention strategies.

Prevention Strategies

Data Mixing Ratios:

The exam often asks about optimal data mixing ratios for preventing catastrophic forgetting:

Fine-Tuning Data Preparation

The exam overlaps between the Fine-Tuning domain (13%) and the Data Preparation domain (9%). Quality of training data directly determines fine-tuning success.

Data Quality Checklist

1. Diversity: Cover the full range of expected inputs and outputs
2. Accuracy: All output labels/responses must be correct
3. Consistency: Same format and style across examples
4. Volume: 1K-10K examples for LoRA, 10K-100K+ for full fine-tuning
5. Deduplication: Remove duplicate or near-duplicate examples
6. Length distribution: Match expected production input/output lengths
7. Edge cases: Include boundary conditions and unusual inputs

Training Data Size Guidelines

RLHF and DPO: Alignment Fine-Tuning

The exam tests awareness of alignment techniques at a conceptual level, not implementation depth.

Reinforcement Learning from Human Feedback (RLHF)

RLHF aligns model outputs with human preferences using a three-stage pipeline:

1. Supervised Fine-Tuning (SFT): Fine-tune the base model on high-quality instruction-response pairs
2. Reward Model Training: Train a separate model to score outputs based on human preference rankings
3. PPO Optimization: Use proximal policy optimization to fine-tune the SFT model, maximizing the reward model's score while staying close to the SFT model (KL divergence penalty)

Direct Preference Optimization (DPO)

DPO simplifies RLHF by eliminating the reward model and PPO stages. It directly optimizes the model using preference pairs (chosen vs rejected responses).

DPO advantages over RLHF:

• No separate reward model needed (reduces memory and complexity)
• More stable training (no PPO hyperparameter tuning)
• Comparable alignment quality for most use cases

Practice Questions

Question 1: You need to fine-tune a 70B model for medical report generation using 8,000 high-quality training examples. You have access to a single A6000 48GB GPU. Which approach is most appropriate?

A) Full fine-tuning with gradient checkpointing B) LoRA with r=16 targeting all attention modules C) QLoRA with r=32, NF4 quantization, targeting attention and MLP modules D) Prefix tuning with prefix length 50

Answer: C. The 70B model in BF16 requires 140GB, far exceeding the 48GB GPU even with gradient checkpointing (A is impossible). Standard LoRA (B) still needs the base model in BF16 (140GB) — does not fit. QLoRA (C) stores the base model in NF4 (35GB), fitting on the 48GB GPU with room for LoRA adapters and activations. r=32 with attention+MLP targeting provides sufficient capacity for medical domain adaptation. Prefix tuning (D) could fit but provides significantly lower quality for this type of task.

Question 2: After fine-tuning a 13B model with LoRA (r=8) on 3,000 customer support examples, the model performs well on support queries but has significantly degraded general reasoning capabilities. What is the most likely cause and fix?

A) LoRA rank is too high — reduce to r=4 B) The base model was corrupted during LoRA training — retrain from checkpoint C) The training data caused catastrophic forgetting — add general-purpose data mixing at 30-40% ratio D) LoRA is inherently unable to preserve base capabilities — switch to full fine-tuning

Answer: A is wrong because r=8 is already low and reducing rank would hurt domain performance. B is wrong because LoRA freezes base weights, so they cannot be corrupted. D is wrong because LoRA preserves base weights by design. The most likely issue is that the 3,000 examples are narrowly focused, and even though LoRA freezes base weights, the adapter outputs can dominate the model's behavior for general queries. The fix (C) is to include general-purpose examples in the training mix to ensure the adapter does not over-specialize. Note: this is a nuanced scenario — LoRA typically prevents catastrophic forgetting, but extreme domain-specific training data can cause the adapter to steer outputs away from general capabilities.

Question 3: You are serving 5 different LoRA adapters (medical, legal, financial, code, support) from a single 70B base model. What is the total GPU memory requirement compared to serving 5 separate full fine-tuned models?

A) 5x base model + 5x adapter overhead B) 1x base model + 5x adapter overhead C) 1x base model + 1x adapter overhead (swap at request time) D) 5x base model (adapters are merged)

Answer: B. The base model is loaded once (140GB in FP16 for 70B). Each LoRA adapter is small (50-200MB) and can be loaded alongside the base model. All 5 adapters can be in memory simultaneously (~1GB total), with routing logic selecting the appropriate adapter per request. The alternative — 5 separate full fine-tuned models — would require 5 x 140GB = 700GB. LoRA serving reduces this to ~141GB, a 5x memory savings.

For comprehensive practice across all 10 NCP-GENL domains, try our NCP-GENL practice exams.

Summary: Fine-Tuning Key Takeaways

For the full preparation strategy, see our How to Pass NCP-GENL guide and NCP-GENL Cheat Sheet for quick reference.