Experimentation is 22% of NCA-GENL, and it is the domain that separates people who have improved an LLM system from people who have only used one. Its questions describe teams making changes (a new prompt, a fine-tune, a model swap) and ask what makes the resulting claim trustworthy, and the answers are the scientific method wearing LLM clothes: baselines before improvements, held-out data, one variable at a time, metrics that match the task. None of it is hard; all of it is skippable, which is why it reliably harvests points from candidates who studied everything else. This guide covers the method, the metric families, and the experiment shapes the exam draws from, with two worked scenarios.
Start Here
The complete guide frames the whole exam; the transformer fundamentals guide covers the domain this one builds on. Drill both at weight in the practice tests. Start with the free NCA-GENL sample questions (no signup) to see the question style and find your weakest domain before you buy.
The method, in four graded habits
Baseline first. Every improvement claim needs a before: the current prompt's score, the base model's accuracy, the simple approach's cost. Stems where a team ships a change because "it looks better" are asking you to demand the measured baseline.
Hold out the test data. Evaluation runs on data the system was not tuned on; scoring a fine-tune on its own training examples is the domain's cardinal sin, and it appears in stems dressed as impressive results that vanish in production. The graded diagnosis is contamination between tuning and testing.
One variable at a time. A team that changes the prompt, the model, and the temperature together learns nothing from the improvement; stems reward isolating the change, and the distractor bundles them for speed.
Match the metric to the task. Exact-match accuracy suits classification and extraction; overlap metrics suit summarization loosely; human or model-graded rubrics carry open-ended quality; latency and cost ride along in production decisions. The recurring trap is a precise-sounding metric applied to a task it cannot measure.
Try 15 NCA-GENL practice questions free
They come from the NCA-GENL practice tests, and each option shows its explanation once you answer.
- 6 timed practice exams
- Every answer explained
- A score for each domain
- $19.99 one-time
Evaluating generation, honestly
The domain expects a working map of how LLM output gets scored. Reference-based metrics compare against a gold answer and fit tasks with one; rubric-based approaches (human raters, or a model grading against criteria) fit open-ended generation, with the caveat the exam likes: model-graded evaluation is a scalable signal that still needs human spot-checking, never an unsupervised authority. Task success metrics (did the extraction validate, did the user's issue resolve) outrank text-similarity metrics whenever the stem offers both, because production optimizes outcomes rather than resemblance.
Task to metric family
| Task in the stem | Graded metric family | Trap |
|---|---|---|
| Classification or extraction | Exact-match accuracy, precision and recall | Fluency judgments on a correctness task |
| Summarization or translation | Overlap metrics plus human review | Treating overlap scores as full quality |
| Open-ended assistance | Rubric-graded (human or model, spot-checked) | Model self-scores as final authority |
| Production rollout decision | Task success plus latency and cost | Quality metrics alone, ignoring economics |
Prompt experiments and fine-tuning runs
The domain's applied questions come in two shapes. Prompt experiments are the cheap loop: version the prompts, run each against the same evaluation set, compare like with like; the graded answer keeps a prompt library with scores rather than one mutating prompt in a text file. Fine-tuning experiments raise the stakes: a training set and a held-out evaluation set, a base-model baseline, and a check that the tune helped the target task without degrading general behavior. When a stem shows a fine-tune scoring near-perfectly, the reflex is contamination suspicion before celebration; when it shows marginal gains over a good prompt, the graded conclusion is that the cheaper rung already sufficed, which connects this domain to the adaptation ladder in the fundamentals guide.
Don't just read about it — run it
The evaluation lab builds the baseline-and-holdout habit on an actual model, and the LoRA lab gives you a fine-tuning run with a before and after you measured yourself.
Three questions from the NCA-GENL practice tests
In language model decoding, what effect does the temperature parameter have on the softmax function during token sampling?
[object Object]
Correct. Higher temperature divides logits by a larger value, flattening the softmax distribution and increasing output diversity.
A. Temperature doesn't affect sequence length. Maximum generation length is controlled by separate parameters like max_length or max_new_tokens in generation configuration, not by the sampling temperature.
B. Temperature is an inference-time parameter used for sampling, not a training hyperparameter. Learning rate scheduling is handled separately during the optimization process and is unrelated to temperature-based sampling.
D. This is backwards. Lower temperature values actually decrease randomness by making the probability distribution more peaked, causing the model to favor high-probability tokens more strongly. This results in more deterministic, focused outputs.
When building a domain-specific language model, what is the primary consideration when choosing the vocabulary size? (Select TWO) (Select TWO)
[object Object]
Correct! This is the other side of the vocabulary size trade-off. A smaller vocabulary forces the tokenizer to break words into more subword pieces, increasing sequence lengths and thus increasing the computational cost of attention (which is quadratic in sequence length). Conversely, a larger vocabulary reduces sequence lengths but requires a larger embedding matrix (vocab_size × embedding_dim), increasing model size and softmax computation cost. Finding the right balance between these factors is essential when designing a domain-specific tokenizer.
Correct. Vocabulary size balances domain-term coverage against embedding parameter count and sequence-length efficiency, with 30K-50K typical for general English.
A. An excessively large vocabulary has significant drawbacks: increased memory requirements for the embedding matrix, slower training and inference due to the large softmax layer, and many rare tokens with poor representations. The vocabulary size should be optimized based on the domain and available data.
C. Vocabulary size should be significantly smaller than unique words in training data. Modern tokenizers use subword algorithms (BPE, WordPiece) to handle unseen words by breaking them into known subwords, allowing efficient representation of rare and out-of-vocabulary terms without requiring every unique word.
E. Vocabulary size is a configurable hyperparameter, not an architectural constraint. You can choose vocabulary size when training a tokenizer and corresponding model. However, changing vocabulary size after training requires retraining the embedding and output layers.
In transformer models, how does the embedding dimension (d_model) affect model capacity and performance?
[object Object]
Correct. d_model determines how much information each token representation can encode and is the primary lever for per-token model capacity.
B. Embedding dimension and vocabulary size are independent hyperparameters serving different purposes. Vocabulary size determines how many unique tokens the model can represent, while embedding dimension determines the richness of each token's representation. They typically have very different values.
C. The embedding dimension affects the entire transformer architecture. All layers process representations of this dimension: attention mechanisms compute queries, keys, and values based on d_model; feedforward layers expand from and project back to d_model; and layer normalizations operate on d_model-sized vectors. It's a fundamental architectural hyperparameter.
D. While larger embedding dimensions increase model capacity, they come with significant trade-offs including increased memory usage, slower training and inference, higher risk of overfitting on small datasets, and diminishing returns beyond a certain point. The optimal dimension depends on task complexity and data availability.
Full NCA-GENL set: 6 timed exams, every answer explained, $19.99 one-time.
See the practice testsWorked scenario: the miracle fine-tune
A team fine-tunes a support classifier and reports 99% accuracy, up from 78% with prompting. Deployed, it performs barely better than the prompt did. What happened?
The evaluation almost certainly leaked: the reported score came from data overlapping the training set, so the model was graded on memorized examples, and production supplied the first honest test. The graded fix rebuilds the split (train on one set, evaluate on untouched data) and re-measures before any rollout claim. Distractors blame the deployment environment or demand more training data, both of which treat a measurement failure as a modeling failure.
Worked scenario: the improved prompt, maybe
An engineer rewrites a summarization prompt, runs it on five documents, prefers the outputs, and wants to ship. What does the exam want first?
A real comparison: both prompt versions over the same adequate evaluation set, scored the same way (rubric or reference), with the difference large enough to trust. Five hand-picked documents and a preference is anecdote; the graded habits are same-data comparison, sufficient sample, and a defined metric. The stem's tell is "prefers": preference without measurement is exactly what the domain exists to upgrade.
Key Takeaways
0/6 completedNext steps
Make the habits reflexes before the 60-minute clock tests them: the practice suite asks this domain at full weight, the mistakes article catalogs how its points leak, and Preporato Pro covers the labs where you run these experiments on real models.
Sources:
![NCA-GENL Experimentation: Evaluation, Baselines & Fine-Tuning [2026]](/blog/nca-genl-experimentation-evaluation-guide.webp)