AI Evaluation & Post-Deployment Monitoring — Synthetic Dry-Run Specimen 001

Specimen boundary

Every organization, person, system, event, source record, observation, and metric in this specimen is fictional. The run establishes how the candidate carries a bounded evaluation-and-monitoring decision. Evidence about a real model, deployment, participant population, or service outcome remains for an authorized field observation.

Payload under test: AI Evaluation, Field Testing & Post-Deployment Monitoring — Research Payload Candidate (v0.1)

run_id: AIRPM-DRY-001
run_type: public_safe_synthetic_dry_run
as_of: 2026-10-02
data_standing: entirely_synthetic
consequence: low_stakes
candidate_version: 0.1.0

Evidence spine and ceiling

NIST AI 800-4 explains why controlled pre-deployment evaluation benefits from post-deployment observation and organizes monitoring into six categories: functionality, operational, human factors, security, compliance, and large-scale impacts. It supplies a monitoring vocabulary and identifies open challenges; each receiving organization still selects locally fitting methods and thresholds.

NIST AI RMF 1.0 provides the voluntary GOVERN–MAP–MEASURE–MANAGE structure. Relevant outcomes include documenting context and evaluation design, testing under conditions similar to deployment, monitoring production behavior, tracking emergent risks, integrating feedback and appeal, and implementing incident response, recovery, decommissioning, and change management.

The NIST AI RMF overview identifies AI RMF 1.0 as the current published framework and records that a revision is in progress. This specimen records the version used and treats a future revision as a reassessment event. The NIST Generative AI Profile adds context for confabulation, human–AI configuration, component integration, pre-deployment testing, and incident disclosure.

The thresholds below are specimen-local design choices. NIST supplies the risk-management and monitoring frames; the fictional receiving organization supplies these acceptance values.

Bound parameters

Claim inventory and thresholds

Three evidence horizons

1. Controlled evaluation

The 72-case suite contained 48 answerable guide questions, 12 unsupported questions, 8 prompt-injection challenge fixtures, and 4 structural accessibility fixtures. Grounded-answer accuracy was 47/48; source identity was visible in 48/48 answerable cases; all unsupported requests used the fallback; all challenge fixtures retained the task boundary; and all four structure checks passed.

These results support entry into a synthetic field rehearsal. Real participant language, workload, assistive-technology use, changing organizational practice, and production infrastructure remain separate evidence horizons.

2. Field-like synthetic rehearsal

Four scripted volunteer roles completed 40 fictional task sessions. Thirty-six drafts met the scripted acceptance checks without revision and four were corrected before simulated release. The source panel and release gate were available in every session. Corrections preserved the original draft, corrected text, reason, guide version, and next route.

The rehearsal demonstrates review and correction control flow. Human comprehension, trust, burden, accessibility experience, adaptation, and feedback behavior remain open for an authorized participant study.

3. Synthetic operational replay

The specimen replayed 120 fictional requests over 14 simulated days. On simulated day 8, the workshop guide changed one room from Studio B to Studio C. Retrieval index R7 continued serving the earlier passage for two room questions.

• Two stale drafts reached the reviewer queue and were corrected before simulated release.
• The automatic material-change alert produced zero signals.
• Prompt, response, and reviewer action were logged for all 120 requests.
• Fifteen post-change traces omitted corpus_version, preventing reliable automated source-version comparison.

The replay exercises the incident-and-recovery path and demonstrates the value of the human review-and-approval step. Its frequency in a real deployment remains an empirical question.

Monitoring coverage

NIST AI 800-4’s six categories organize the post-deployment portion of this run as a coverage vocabulary.

Designed incident and recovery

The guide changed from G1 to G2, while index R7 continued serving an earlier room passage. Human review detected two stale drafts before simulated release. The automatic alert depended on a source-version field that the relevant traces lacked.

Containment moved room-location questions to the deterministic fallback, paused generative room answers, preserved correction receipts, and routed the incident to the fictional operations lead.

Repair added corpus_version, index_version, and retrieved_passage_id to every trace; bound source changes to an event-driven replay; added an alert for guide/index mismatch; and preserved reviewer override as the final release gate.

Sixteen change-sensitive fixtures then passed: every fixture retrieved the G2 room, every trace carried guide and index versions, and the mismatch alert fired when the old index was deliberately reintroduced. This supports reopening the synthetic replay only. Human field use remains a distinct authorization and evidence event.

Coverage ledger

Receiving decisions

Synthetic system decision

Revise and repeat the synthetic field rehearsal. The pre-release suite and release gate support continued synthetic work. The source-change alert and trace completeness findings identify the next bounded improvement.

Candidate payload decision

Focused revision; candidate status continues. Four additions will make the next execution more legible:

1. Require each result to declare observed_real, observed_synthetic, planned, or unavailable.
2. Require each threshold to declare source_prescribed, organization_selected, or exploratory.
3. Add the six NIST AI 800-4 categories as a required coverage ledger alongside lifecycle horizons.
4. State that a synthetic specimen can establish control-flow coherence. Real field evidence, deployed-system evidence, affected-population evidence, and independent review form separate promotion horizons.

The separate dry-run receipt records invocation identity, candidate digest, demand states, decision, and re-entry.

Official sources

• NIST AI 800-4 — Challenges to the Monitoring of Deployed AI Systems
• NIST AI 100-1 — Artificial Intelligence Risk Management Framework 1.0
• NIST AI Risk Management Framework overview
• NIST AI 600-1 — Generative Artificial Intelligence Profile