---
name: architect-eval-baseline
description: >
  Scaffolds an eval baseline for any AI-producing component before it ships. Operationalizes
  the BOOTSTRAP-BRIEF policy "no eval baseline equals no ship." Triggered when the operator
  says: "build an eval", "no eval yet", "eval baseline", "scaffold an eval", "what should
  the eval look like", "ship gate", "eval methodology", "set up the harness", "we need to
  measure this", "how do we know if it's working". The skill walks the four eval methodologies
  (llm-judge, human, synthetic, hybrid), demands a concrete dataset source + size, names ship
  thresholds per metric, sets refresh cadence, and pins the reproducibility quadruple. Refuses
  to declare an eval "ready" without dataset, threshold, cadence, and pin all named. Pairs
  with architect-reproducibility-gate (the gate this baseline feeds into) and architect-adr
  (eval methodology decisions are typically ADR-worthy).
---

# architect-eval-baseline

Scaffolds the eval baseline before any new AI-producing component ships. The brief says "no eval baseline equals no ship"; this skill makes the policy executable.

## When to invoke

- Operator says: "build an eval", "no eval yet", "eval baseline", "scaffold an eval", "what should the eval look like", "ship gate", "eval methodology", "set up the harness", "we need to measure this", "how do we know if it's working"
- A new AI-producing component is being scoped and the L3 framework contract's eval methodology is `none-yet` with an open closes-by date
- An existing eval is suspected stale (faithfulness drift, dataset drift, model bump invalidating prior baseline)
- ADR-0003-equivalent decision in a deployed instance proposes shipping without an eval; refuse and invoke this skill

## Why this exists as a refusal-encoded skill

In 2020-2023, building an eval baseline took a sprint of engineering effort and the rational decision was often to skip it on first ship. AI assistance compressed that cost to roughly an hour of scaffolding plus a few dollars in tokens. The cost of skipping has not compressed: a silent regression still takes weeks to debug at the time you can least afford it.

The economics flipped. This skill encodes the new economics as a refusal: refuses to declare an eval "ready" without dataset, threshold, cadence, and reproducibility pin all named. Refusal-encoded because operator-discipline alone has historically failed to keep eval baselines current.

## Operating procedure

### Step 1: Read the L3 framework contract

Read `policies/framework-contract.md` for the component's eval methodology answer (Q5 of the localization set). One of:

- `none-yet` with closes-by date - this skill's job is to close it
- `llm-judge` - dataset + LLM scoring; verify thresholds + cadence
- `human` - human-rated; verify reviewer pool + cadence
- `synthetic` - generated test cases; verify generation method + coverage
- `hybrid` - typically LLM-judge plus some human sampling; verify both legs

If the contract names a methodology already, this skill verifies completeness; if `none-yet`, this skill scaffolds.

### Step 2: Lock the dataset

Demand four facts about the eval dataset:

1. **Source:** where the data came from (production logs, curated corpus, vendor data, synthetic generation)
2. **Size:** number of examples; floor for V1 is typically 100-1000 depending on task
3. **Coverage:** what slices the dataset stratifies (market segment, language, complexity tier, edge cases)
4. **Refresh cadence:** how often the dataset is regenerated (quarterly default; on every model bump for high-stakes evals)

Refusal: a dataset without source, size, coverage, and cadence is not a dataset; it is a pile of examples. Surface as a finding.

### Step 3: Lock the ship thresholds per metric

For each metric the eval scores (faithfulness, fluency, safety, citation-correctness, etc.), name the ship threshold as a number:

- `faithfulness >= 0.92`
- `fluency >= 0.85` (lower bar is acceptable when output is structured)
- `human-pass >= 0.95` (when human review is in scope)
- `safety >= 1.00` (red lines must be 100%)

Refusal: thresholds described qualitatively ("high faithfulness", "good fluency") are not thresholds. Demand numbers.

### Step 4: Lock the reproducibility pins

Per L3 contract's telemetry schema, every eval run pins:

- `model_version` (the model under eval)
- `prompt_version` (the prompt being scored)
- `tools_version` (any tool calls in the chain)
- `data_snapshot_version` (the eval dataset version)
- `eval_harness_version` (the scoring code itself)

The fifth pin (`eval_harness_version`) is specific to evals; bumping the harness invalidates score comparisons across versions even if the other four are stable.

### Step 5: Ship-gate logic

Define what happens on eval pass / fail:

- **Pass (all metrics meet thresholds):** ship is permitted; the build pipeline proceeds
- **Fail (any metric below threshold):** ship is blocked; finding surfaces with which metric and how far below
- **Indeterminate (eval did not run; harness error; missing pin):** ship is blocked; treated more strictly than fail because the silent-regression risk is higher

Encode the ship-gate in CI or in the deploy skill's pre-ship check.

### Step 6: Cadence + drift detection

Schedule:

- Eval runs on every change to the model, prompt, or tool set
- Eval runs on every dataset refresh
- Eval runs on a calendar cadence (typically quarterly) even without changes, to catch silent drift

Drift detection: if eval scores trend down across runs (without any deliberate change), surface as a finding. Drift is the silent failure mode evals exist to catch.

### Step 7: Write or update the eval contract

The eval baseline lands in two places:

1. **`policies/framework-contract.md` Section "Eval methodology":** updates the L3 contract's eval section with dataset, thresholds, cadence, pins
2. **`kit/eval-harness/`:** the executable scoring code (in the L4 platform kit; mirrored or locally owned per posture)
3. **`kit/eval-data/`:** the dataset itself (typically locally owned even if the harness is mirrored)

If the eval methodology decision is ADR-worthy (most are), invoke architect-adr to scaffold ADR-{NNNN}-eval-methodology.

## Refusal conditions

- Refuse to declare an eval "ready" without dataset source, size, coverage, and cadence all named
- Refuse to declare an eval "ready" without numeric ship thresholds per metric
- Refuse to declare an eval "ready" without all five pins (model + prompt + tools + data + eval harness) captured in telemetry
- Refuse to ship a component where the L3 contract's eval methodology is `none-yet` past the closes-by date without a new ADR explaining the slip

## Cross-references inside the kit

- `BOOTSTRAP-BRIEF.md` Section IX (testing posture) and Section XIII (testing posture in detail)
- `templates/L3-framework-contract.md` - the eval methodology section this skill populates
- `examples/martech-mature-instance/policies/framework-contract.md` - reference example with hybrid LLM-judge + 10% human-review eval
- `skills/architect-reproducibility-gate/SKILL.md` - the gate this eval feeds into
- `skills/architect-adr/SKILL.md` - paired when eval methodology is ADR-worthy
- `skills/validate/SKILL.md` - the audit skill checks eval pass status as part of L3 contract conformance
