The Control Evidence Agent — Grounded GenAI for NIST 800-171 / CMMC

This is the summary. The full case study — with the interactive per-family chart — lives here: View the full visual case study →. There is also an AI-engineering deep-dive for the systems detail.

A retrieval-augmented agent that drafts NIST SP 800-171 assessment findings — a status (met / not_met / customer / inherited) plus a cited justification for each of the standard’s 320 assessment objectives — and self-checks every sentence against the source it cites before the finding is allowed to stand. Built entirely as internal R&D on my own CMMC Level 2 / NIST 800-171 AWS landing zone, with a stub → local-model → Amazon Bedrock provider ladder behind a single interface.

The project was never “an LLM writes compliance text.” It was to build the machinery that makes an LLM’s compliance text trustworthy and measurable — and, ultimately, an eval-validated map of exactly where a language model can be trusted on this task and where it can’t. That map is the deliverable.

The Challenge

CMMC Level 2 requires a contractor to show, for each of 110 requirements (320 assessment objectives), that a control is implemented — and to keep the SSP narrative honest about what is actually evidenced versus merely claimed. In practice the implementation matrix over-claims: a row says “implemented” while the defining artifact is a TODO, or the mapped Config rule checks something adjacent to what the objective actually asks.

Reconciling claim against evidence, at objective granularity, across a whole standard, is exactly the tedious, high-stakes work a grounded LLM should help with — if it can be trusted not to invent the evidence, and if you can measure where its judgment is sound.

What I Built

Four layers, deterministic-first, each testable at $0 with no model, no network, and no AWS:

• A deterministic control graph (SQLite) — the standard’s requirements and objectives (vendored, pinned, hashed OSCAL), the implementation matrix (the claims), typed evidence (Config rules plus IaC / SCP / AWS-inherited artifacts with file-and-line provenance), and a hand-labeled answer key. Every importer is a gate that refuses a graph it can’t vouch for.
• A grounded generator — retrieve a typed context pack per control, draft a per-objective status plus one cited sentence, then pass it through a two-layer grounding gate: a deterministic citation check, then an LLM entailment judge. An agentic revise loop re-drafts ungrounded claims — with the status frozen, so it can only fix wording and citations, never change what it asserts.
• An MCP server — the reconciliation tools exposed over JSON-RPC (standard library only) so any MCP client can drive the harness.
• An evaluation harness — scores every objective against the answer key on two orthogonal axes (grounding vs. correctness), across a ladder of models, with every accuracy change attributable to a specific diagnosed cause.

Python standard library only (boto3 the single optional dependency, lazily imported for the Bedrock path). temperature=0 and a fixed seed, so runs reproduce exactly. 175 tests, none touching a model, a network, or AWS.

The Headline Result

I brought all 14 families / 320 objectives to hand-verified ground truth, then ran the identical harness with three generators: a $0 lexical stub, a local qwen2.5:14b (Ollama, on a 16 GB laptop), and Claude Haiku 4.5 on Bedrock. The stub is the floor a real model must beat — “has evidence → met” — and it structurally cannot emit customer or inherited.

Sorting the 14 families by the 14B’s status-correctness margin over the stub produces a clean, monotone curve in how much each family’s ground truth requires human judgment:

Where a family’s ground truth is dominated by human judgment the heuristic can’t express — Awareness & Training is customer 9/9, so the stub scores 0 and the reasoning model scores 100 — the model wins by the maximum possible margin. Where the ground truth aligns with an evidence-presence heuristic (Configuration Management is met-heavy because the IaC codebase is the config baseline), the model loses by refusing to rubber-stamp the borderline met calls the stub gets for free.

The takeaway is methodological and load-bearing: aggregate correctness rank is a property of the standard’s shape, not of the models. A single accuracy number, averaged across families, would rank a reasoning model below a one-line heuristic — because half the standard is exactly where the heuristic looks good. That is why the harness scores the per-objective diff, and the curve above is the proof it has to.

Self-Hosted vs. Bedrock, for CUI

For CUI you’d love to keep data on infrastructure you control — so could a self-hosted model do the generator or judge job, or is a managed frontier model required? I ran the identical harness, differing only in the transport (a standard-library HTTP call to Ollama vs. the AWS SDK), so a difference in the numbers is a difference in the model, not the scaffolding. On family 3.3, temperature 0:

The 8B is a usable judge but an unusable generator — below the trivial stub, failing to emit parseable output for a third of objectives (a structured-output ceiling that recurred, near-identically, on all 14 families). A bigger local model crossed to usable (38% → 69%, zero unassessed), within 7 points of the frontier — so the bottleneck was reasoning, not formatting. The recommendation, with a number behind it: Bedrock in GovCloud (FedRAMP-High) is the default for generation, a 14B-class model is a defensible self-hosted alternative, and an 8B is a $0 judge pre-filter — never the sole gate.

What the Eval Caught (That I Didn’t)

An eval earns its keep by catching your own errors. Across the build it surfaced:

• A systematic 23-objective defect in the vendored ground-truth catalog — every single-determination objective mis-split upstream. Unrepaired, the importer would have silently scored those requirements against blanks. Fixed at import, verified against the authoritative NIST PDF, locked by tests.
• An inconsistency in my own answer key — two near-identical objectives labeled differently, which the key itself had flagged. Provably not score-gaming: the fix didn’t move the deterministic stub.
• False matrix claims, family after family — an MFA-requiring SCP, an “unused-credentials” Config rule, pipeline approvals, GuardDuty Malware Protection, a wired incident-notification path, an “immutable” Object-Lock archive, an SSM patch module aimed at hosts that don’t exist. Each claimed in docs and absent in code — exactly the “claimed but not implemented” gap a C3PAO exists to find.

Honest Limitations

• One run per family at temperature 0 — reproducible, but these numbers carry real run-to-run variance; the direction and the per-objective diff are the measurement, not the third significant figure.
• One local hardware tier — a 16 GB laptop caps at ~14B; whether a 32B/70B closes the last 7 points to the frontier is inferred, not measured.
• The residual generator errors are a genuine judgment boundary, not fabrications — chiefly the customer vs. not_met line (is a missing process control the contractor’s duty, or a gap the platform should close?). That is precisely where a human assessor still owns the decision, and the tool’s job is to surface the evidence, not pretend to settle it.

Results

An eval that catches your own ground-truth errors is doing its job.

The project produced an eval-validated, per-objective map of where a language model can be trusted on NIST 800-171 assessment — the delta between models, measured on one harness, rather than asserted. Deterministic first, model last. For the full write-up with the interactive per-family chart, read the complete visual case study →.