AI Evaluation, Field Testing & Post-Deployment Monitoring — Research Payload Candidate (v0.1)

Research job

Support a named decision about whether an AI-enabled system should continue, expand, change, narrow, pause, roll back, or retire within a declared context of use.

The candidate joins four evidence horizons that are often separated:

1. pre-release evaluation;
2. field testing under bounded operating conditions;
3. post-deployment monitoring;
4. reassessment after material change, incident, or population shift.

The result should show which claim each observation supports, where detection remains limited, and which accountable receiver can act.

Why this candidate is timely

NIST AI 800-4 — Challenges to the monitoring of deployed AI systems, published 2026-03-06, distinguishes controlled evaluation from the need to observe systems in real deployment contexts. The report describes monitoring categories while finding that methods, validated practice, and shared terminology are still developing.

That evidence supports a configurable local research instrument. The NIST report orients monitoring design; the named receiving context supplies its population, metrics, thresholds, recourse, and operating authority.

Required parameters

A missing value that changes the governing population, metric, authority, or threshold produces a scoping pause.

Evidence classes

• Controlled evaluation: repeatable tasks, fixtures, simulations, or benchmarks under declared conditions.
• Field evidence: bounded use in the receiving environment with named participants, time window, and safeguards.
• Operational evidence: production traces, service measures, incidents, corrections, overrides, and recovery.
• Affected-population evidence: accessibility, burden, access, outcomes, recourse, and subgroup experience.
• Source and policy evidence: current standards, regulation, documentation, contracts, and organizational policy.
• Interpretive judgment: source-linked reasoning whose status remains visible.
• Open local hypothesis: a claim with a defined observation, threshold, and disconfirming result.

Several dashboards repeating one underlying signal remain one evidence lineage.

Evaluation and monitoring map

Research demands

1. Context and consequence

Define intended use, foreseeable adjacent use, affected populations, material consequences, and the role that retains decision authority.

2. Claim-to-evidence map

For every material claim, record the fitting observation, source class, population, environment, time horizon, acceptance observation or decision rule, and coverage boundary. Every result also declares its observation standing as observed_real, observed_synthetic, planned, or unavailable; every threshold declares its provenance as source_prescribed, organization_selected, or exploratory.

3. Change inventory

Identify changes to model, prompt, retrieval corpus, tool, software, policy, data, population, interface, workflow, and dependency. Classify which changes preserve evidence and which reopen evaluation.

4. Detection design

For every monitored signal, record denominator, observation window, collection method, detection limit, alert threshold, false-positive and false-negative consequence, owner, and response. Review coverage across the NIST AI 800-4 categories of functionality, operational conditions, human factors, security, compliance, and large-scale impacts; an open category remains visible with its fitting next observation.

5. Human review and recourse

Observe review workload, override and correction behavior, escalation, explanation, appeal, access, and the path back into source or system change.

6. Counterevidence and benign explanations

Search for alternate causes such as workflow change, data latency, coding change, seasonality, instrumentation failure, selection effects, or reviewer adaptation before fixing a causal interpretation.

7. Continue–change–pause criteria

Define evidence thresholds and the authority for continue, expand, revise, restrict, pause, rollback, retire, and study-next decisions.

8. Longitudinal receipt

Preserve system identity, evaluation version, operating conditions, evidence date, decisions, open gaps, next review, and events that force reassessment.

Minimum output contract

1. context-of-use declaration;
2. system and change inventory;
3. claim-to-evidence matrix;
4. pre-release evaluation plan and results;
5. field-test plan and results where authorized;
6. deployed monitoring specification;
7. affected-population and recourse evidence;
8. incident, correction, and recovery route;
9. thresholds and accountable decisions;
10. coverage ledger;
11. completion judgment;
12. run receipt and next review.

Companion instruments

• Research Payload Family — Shared Core
• Evidence-Bearing Verification Annex
• Decision & Data Shape Parameter Pack
• Goal Fidelity Under Partial Context — Deep Research Result: Evidence-Bearing Collaboration Across Model, Harness, Human & Organization
• Material Verification — Connect Checks to Behavior, Fault & Observable Evidence

Promotion work

Executable status requires:

• one worked specimen that spans controlled evaluation, bounded field evidence, deployed monitoring design, and a change decision;
• one partial or unavailable evidence case;
• one incident or recovery path;
• one run receipt;
• a completed coverage ledger;
• a review by a fitting system, domain, and affected-population vantage;
• a promotion decision recorded in the portfolio.

Until those observations exist, this page remains a public-grade candidate and a usable planning surface.

Focused refinement from dry run

The synthetic dry-run specimen and its separate run receipt exercised the candidate across controlled evaluation, field-like rehearsal, operational replay, material change, containment, repair, and recovery.

The run established control-flow coherence and exposed four focused improvements now carried by v0.1.1:

1. every result declares observed_real, observed_synthetic, planned, or unavailable;
2. every threshold declares source_prescribed, organization_selected, or exploratory;
3. the six NIST AI 800-4 monitoring categories form a required coverage ledger alongside lifecycle horizons;
4. evidence from synthetic execution and evidence for promotion remain distinct, so real field, deployed-system, affected-population, and fitting independent-review observations retain their own standing.

The candidate remains Blooming. The next promotion observation is an authorized real-world specimen with system, domain, and affected-population review.

Candidate machine node