Take-home: a security-controls assistant on the NIST SP 800-53 catalog

The brief, as sent to the candidate:

Customer. Larkspur Health (fictional) sells appointment-scheduling software to clinics. Its hospital customers send security questionnaires mapped to NIST SP 800-53. Priya, the compliance lead, has a team of four. Engineers send them about 150 questions a week, and the team wants an assistant that engineers can ask first.

What engineers ask. Three kinds of request:

• A. Exact lookups: "What does AC-2(4) require?", "show me SC-7", "ac-2.4". The answer must be exactly that control. A neighboring control counts as wrong.
• B. Open questions: "How should we handle shared admin accounts?", "What do we need for audit log retention?" The answer names the relevant controls, explains them in plain words and cites control ids.
• C. Scoped tasks: "List the moderate-baseline controls in the AU family, and for each one, one piece of evidence we would collect for our system." The answer must be complete for that scope, with nothing extra.

Data.

• The NIST SP 800-53 Rev. 5 catalog in OSCAL JSON, published by NIST in its OSCAL content repository on GitHub (usnistgov/oscal-content). The file is about 10 MB, and it also contains the SP 800-53A assessment procedures.
• In release 5.2.0 (the version in the repository on 2026-11-03), the catalog has 20 families and 1,014 controls and control enhancements that are not withdrawn: 300 base controls and 714 enhancements. 182 more are marked withdrawn. Most of them link to where their content went.
• The moderate baseline profile (SP 800-53B) in the same repository lists 287 controls and enhancements.
• The file labels controls with zero padding ("AC-02(04)"), while engineers type "AC-2(4)".
• Larkspur's system, as Priya describes it: "A multi-tenant web app on a public cloud. Postgres for data. An admin console used by 12 engineers. Single sign-on through an identity provider. Logs shipped to a hosted log service. A weekly on-call rotation."

Rules.

• About 5 hours of work, inside a window of 5 calendar days.
• Use any LLM provider and any embedding model. Keep your total API spend under $25.
• Log the cost and latency of every query.
• AI coding help is allowed in this practice version. Say where you used it. Real companies set their own rules, and some forbid AI help on take-homes, so practise both ways.
• For your cost estimates, assume: large model $3 per million input tokens and $15 per million output tokens, with a 200,000-token context window; small model $1 and $5; cached input 10% of the input price, and writing a cache entry 125% of it; embeddings $0.10 per million tokens.

Deliverables.

1. A repository with a README: one command to set up, one to run.
2. A runnable app (a command-line tool or a small web page) that handles all three request types and cites control ids.
3. An eval report of one or two pages:
   • At least 45 labeled queries that you wrote, 15 per request type, each with its expected control ids.
   • Metrics per request type.
   • Median and p95 cost and latency per query.
   • Three failures you found, and what you would do about each.
4. A one-page architecture document: the components, why this design over two alternatives you considered, and what changes with 100 times the data.
5. A recorded walkthrough of 5 to 8 minutes, as if you were showing it to Priya.

The defense. 60 minutes with a panel: a 10-minute demo, then questions about your choices, your measurements and what you would do next.

Before you build anything, write a one-page plan: how each of the three request types will be answered, how you will measure each one, and what you will leave out of 5 hours.