# RESULT — Campaign100 row 030 confidence-calibration

**Tier: T3-exploratory.** CPU-only re-analysis of frozen artifacts (022 `scores.json` + 021
`pairs.json`/`labels_v1.json`). No new GPU (all confidence fields present in 022's capture).
Ran locally, `uv run --with numpy`, BLAS≤8, synchronous, NO watchers. Disk: out/ 28K.

## Question
Is the SONAR decoder's per-token `p_top1` (its confidence in its own greedy emission) calibrated to
actual reconstruction fidelity — does high confidence predict a token exactly matches the original?

## What ran
- Per-token table over **1500 greedy reconstructions = 30,742 tokens**. Ground truth `correct` =
  decoded token matches the aligned original word (word-level difflib, reusing 022's
  `substituted_token_idx`; `equal` blocks→correct, `replace`/`insert`→incorrect). Global exact-match
  correct-rate **0.9406**. Token classes: content 17,146 / function 10,034 / punct 3,562.
- "Fab tokens" (abstention target) = substituted tokens in non-`faithful-paraphrase` sentences
  (021 labels) = **384 tokens (1.25%)**.
- Reliability diagram (equal-width + equal-count deciles), ECE/MCE, per class; PAV/isotonic
  recalibration; risk–coverage curve.

## Headline: confidence is monotone but systematically **UNDER-confident**, and a weak abstention lever
- **Calibration curve is cleanly increasing** (equal-width deciles, acc monotone **9/9**):
  acc 0.111→0.972 as `p_top1` rises; AUC(conf→correct) = **0.7377**. `p_top1` *does* rank-order
  fidelity. (Equal-count deciles 0.72→0.97 with two ~0.01 ceiling wiggles = noise.)
- **Direction = UNDER-confident, not over.** Mean signed gap (conf−acc) = **−0.0703**; *every*
  populated bin has accuracy ABOVE its stated confidence (0/3 high-conf bins over-confident). E.g.
  conf-bin [0.50,0.60): conf 0.550 / acc 0.736; [0.90,1.00): conf 0.927 / acc 0.972. **ECE 0.0703,
  MCE 0.186.** Interpretation: `p_top1` measures confidence in the *emitted* token, but the decoder's
  argmax matches the *original* word even when its probability mass is spread over synonym/surface
  variants — so at `p_top1≈0.55` the greedy token still reconstructs faithfully ~74% of the time.
  Opposite of the usual LLM over-confidence; specific to the match-original target.
- **As an absolute probability of correctness, raw `p_top1` is near-useless:** raw Brier **0.0553**
  barely beats the base-rate Brier **0.0558** (predicting the constant 0.94). **PAV/isotonic
  recalibration** (89 blocks, monotone ✓) recovers a proper monotone map and cuts Brier to
  **0.0484** (−12.5%). A usable monotone recalibration exists.
- **Per class:** ECE punct 0.039 < function 0.072 < content 0.078. Function tokens best-reconstructed
  (acc 0.964, conf 0.891); content worst (acc 0.929, conf 0.851). Function marginally better
  calibrated than content.

## The abstention connection (feeds 087)
Confidence is a **coarse, non-deployable** abstention signal at the 1.25% fab base rate
(AUC lowconf→fab 0.720):
| abstain (lowest-conf first) | coverage | selective exact-acc | fab removed | correct-token cost |
|---|---|---|---|---|
| 0% | 1.00 | 0.941 | 0% | 0% |
| 5% | 0.95 | 0.957 | 27% | 3.4% |
| 10% | 0.90 | 0.965 | 47% | 7.7% |
| 20% | 0.80 | 0.970 | 61% | 17.5% |
| 50% | 0.50 | 0.972 | 75% | 48% |

To **remove 90% of fab tokens you must abstain on 84% of ALL tokens** (correct-token cost 84%) —
because the systematic under-confidence means a large mass of *correct* tokens also carry low
`p_top1`. Low-abstention operating points are mildly useful (drop lowest 10% → +2.4pt selective
accuracy, ~half the fab tokens gone at 7.7% correct cost) but there is no clean separation. Coheres
with 029 (no post-hoc detector reaches deployable precision) and 022 (entropy AUC 0.83 redundant with
cosine): **087 should treat abstention as a graded coverage/fidelity tradeoff, not a precise gate.**

## Predictions (frozen in PREREG_LITE) → outcomes
- (a) P=0.65 decoder OVER-confident (ECE>0.05 w/ high-conf bins below stated acc) → **FALSE**
  (under-confident; ECE 0.070>0.05 but wrong direction, 0/3 hi-conf bins over).
- (b) P=0.90 `p_top1` monotonically predicts correctness (curve increasing) → **TRUE**
  (equal-width 9/9 monotone, AUC 0.738).
- (c) P=0.30 remove 90% fab costs <20% correct → **FALSE** (costs 84%).
- (d) P=0.55 function better-calibrated than content → **TRUE** (ECE 0.072<0.078, marginal).
- **Brier(preds) = 0.1813.** (Biggest hit: pred a — I under-weighted the reconstruction-specific
  under-confidence and defaulted to the LLM over-confidence prior.)

## Gates — all PASS
- G1 alignment sanity: exact-`greedy021` sentences → **0 substituted tokens in 859/859** (0 length
  mismatches). Global correct-rate 0.941 plausible vs 021 exact-recon.
- G2 `p_top1` ∈ [0,1] for all 30,742 tokens; ECE/MCE finite.
- G3 not anti-correlated: AUC(conf→correct) 0.738 > 0.5 (curve increasing, not decreasing).
- G4 PAV monotone map exists (89 blocks, non-decreasing); PAV Brier ≤ raw Brier.

## Limitations
- "Correct" = exact match to aligned original *word* (difflib on lower/stripped words); faithful
  paraphrase synonyms count as incorrect reconstructions (by design — this is *fidelity* calibration,
  not fabrication-detection calibration). The abstention/fab analysis uses the sparser 021-labelled
  substitution set (384 tokens) for the fabrication-specific referent.
- Word-diff alignment, not SPM-token alignment (avoids re-tokenizing originals; no GPU). Sub-word
  tokens inside a substituted word are all marked incorrect together.
- Single decode strategy (greedy), single corpus (021's C4-family 1500). Binding-battery
  "filler/other" class split is N/A on this free-form corpus (content/function/punct used instead).

## Follow-up worth funding? **Y (narrow).**
Feed the PAV-recalibrated confidence + the graded risk-coverage curve directly into **087
(fail-closed decoder)** as the abstention primitive — the honest lever is coverage tradeoff, not a
precise fab gate. The under-confidence finding suggests **temperature/label-smoothing recalibration of
the decoder itself** could sharpen `p_top1`, but the ceiling is low (raw Brier ≈ base rate).