# RESULT — 022 entropy-signature

**Verdict: POSITIVE (certified) — there IS a decoder-confidence signature of fabrication, but it is
REDUNDANT with cosine, not complementary. T3-exploratory.**

During greedy decoding of the 021 corpus, tokens at **substituted (fabricated) spans are emitted at
sharply higher entropy / lower confidence** than faithful tokens (entropy +1.4 nats, p_top1 0.64 vs
0.86; Cohen's **d≈1.2**). A free sentence-level entropy detector reaches **AUC 0.87** (all pairs) and
**0.83** on the gate-blind subset (cos>0.95). BUT cosine alone is *stronger* (0.92 / 0.89), and
adding entropy to cosine does **not** help (−0.01 AUC) — entropy and cosine are correlated readouts
of the same decode difficulty. So entropy is an attractive *cheap standalone* flag (no re-encode
needed) but not an additive gain over the round-trip cosine gate.

## What ran (GPU0/CVD=0→phys0, 131 s, then CPU analysis)
- **Decode-with-scores (out/scores.json).** Re-decoded the SAME 1500 021 `orig` sentences (read from
  021 `out/pairs.json` → identical corpus). Encoder = 021's `TextToEmbeddingModelPipeline`; decoder =
  manual **greedy** loop (`dm.decode`→`dm.final_proj`, argmax; the 020-validated path, no length/
  repetition penalty). Captured per emitted token: Shannon entropy (nats), p_top1, top1–top2 prob
  margin, chosen-token logprob.
- **Analysis (numpy, no relabel).** Fabrication ground truth = 021 `labels_v1.json` greedy label ≠
  faithful-paraphrase (54/1500 fabs). Span alignment via difflib word-diff of orig vs token-recon →
  substituted recon words → token indices → per-token stats. Detector = numpy logistic, stratified
  60/40 × 3 seeds.

## Gates
- **Determinism: match021 = 0.998** (1497/1500 exact-match to 021's greedy) ≥ 0.99 → **PASS**. All 3
  mismatches are long-bin (26-40 tok) degenerate/max-length edge cases (e.g. id 1373 "James ⁇ ⁇ ⁇…"
  repetition; ids 1085/1292/1373) — trailing-token differences on pathological outputs, not a
  systematic decode change. All captured scores finite.
- **Span hand-verification: 20/20** spot-checked spans correctly locate the substituted token(s)
  (e.g. "military"→"army", "Hasan"→"Hassan", "personnel"→"personal", "Geopoetry"→"Geopoe·ia").
  ~4/20 also over-capture adjacent faithful words on rephrasings/reorderings (id 192, 1031) — a
  word-diff limitation (see Limitations), never a miss.
- Smoke (20 sents incl. 12 known fabs) passed: capture finite + determinism 1.000 + span align OK.
- GPU0-only honored (verified 35 MiB/0% pre-launch; freed post-run); night8 venv; unset CONDA_PREFIX;
  BLAS≤8. No foreign process/tmux touched.

## (1)+(2) Confidence at substituted spans (entity-substitution + number-change)
38 substituted greedy fab sentences, 159 substituted tokens vs within-sentence content controls:

| metric | substituted span | within-sentence ctrl | pos-matched ctrl |
|---|---|---|---|
| entropy (nats) | **2.85** | 1.58 | (per-index faithful mean) |
| p_top1 | **0.64** | 0.86 | — |
| Cohen's d (entropy) | — | **d = 1.20** (sub vs within) | **d = 1.04** (pos-matched) |

Paired per-sentence entropy diff = **+1.40 nats, bootstrap 95% CI [1.00, 1.83]** (excludes 0).
→ Substituted tokens are decoded with markedly more uncertainty. **Prediction (a) TRUE** (large effect).
Mechanistic reading: the substitution occurs exactly where the decoder branches away from the source —
these are *not* confident errors; they carry a live entropy fingerprint. (Caveat: "substituted span"
is definitionally the divergence point, so some elevation is expected; the within-sentence and
position-matched controls make it a meaningful, not circular, contrast.)

## (3) Detector — sentence "contains a fabrication" (test AUC, mean±sd over 3 seeds)
Features: {max_entropy, min_margin, mean_chosen_logprob, #(entropy>1.0 nat)}.

| detector | ALL pairs (n=1500, 54 pos) | gate-blind cos>0.95 (n=1460, 35 pos) |
|---|---|---|
| cosine-only | **0.917 ± 0.022** | **0.888 ± 0.018** |
| entropy-only | 0.871 ± 0.024 | 0.834 ± 0.054 |
| cosine + entropy | 0.904 ± 0.026 | 0.893 ± 0.042 |

- **Entropy-only AUC 0.871 > 0.7 → prediction (b) TRUE.**
- **Entropy-only AUC 0.834 > 0.65 on gate-blind → prediction (c) TRUE.**
- **cos+entropy − cos = −0.013 (NOT >+0.05) → prediction (d) FALSE.** Cosine already dominates and the
  two features are correlated; adding entropy (5-feature logistic on 54 positives) slightly *lowers*
  test AUC. No complementary gain.
- Nuance on "gate-blind": a hard cos>0.95 *threshold* misses ~2/3 of fabs (021), but the *graded*
  cosine still ranks them (fab rate 0.16 at 0.95-0.98 vs 0.011 at 0.98-1.0) → cosine-only AUC stays
  0.89 even here. Entropy provides an independent 0.83-AUC readout in that band.

## Prediction Brier (frozen preds a-d; mean **0.267**)
| pred | p | outcome | brier |
|---|---|---|---|
| (a) sub-span entropy > within, d>0.3 | 0.45 | **1** (d=1.20) | 0.303 |
| (b) detector AUC>0.7 all pairs | 0.55 | **1** (0.871) | 0.203 |
| (c) detector AUC>0.65 gate-blind | 0.40 | **1** (0.834) | 0.360 |
| (d) cos+entropy beats cos by >0.05 | 0.45 | **0** (−0.013) | 0.203 |

Calibration lesson: I was **under-confident on (a)/(b)/(c)** — I over-weighted the "gate-blind fabs are
*confident* errors" hypothesis, which the data refute (they are higher-entropy). I correctly doubted
(d): entropy is real but redundant with cosine.

## Limitations
- **Small positive n**: 54 greedy fabs; span stats rest on 38 sentences / 159 tokens; detector test
  folds have 22 (all) / 14 (gate-blind) positives → AUC CIs wide (esp. gate-blind entropy sd 0.054).
- **Word-diff over-capture**: on reordered/rephrased fabs the substituted span includes some faithful
  tokens (id 192, 1031), diluting the sub-span contrast toward the null → the true d is if anything an
  *under*-estimate. No span was missed (20/20 hand-verified).
- **Greedy only** (per design); nucleus scores not captured. **n_hi_ent τ=1.0 nat fixed** (not tuned).
- **Circularity caveat** on (a) stated above; the detector (3) is the non-circular test (features are
  span-agnostic) and it is what licenses the "free detector" claim.
- Single-judge fabrication labels inherited from 021 (T3).

## Follow-up worth funding? **Y (targeted)**
- The headline for the program: **entropy ≈ cosine, not additive** → a fabrication monitor gains little
  from stacking them; pick entropy when you want a *decode-time, no-re-encode* flag (086 latent-monitor).
- **030 confidence-calibration**: per-token-class PAV on these exact scores (does p_top1 calibrate to
  fidelity?) is the natural next step; 022 hands it out/scores.json (per-token stats + labels).
- **087 fail-closed-decoder**: an entropy/margin abstention threshold at decode time, ROC already
  sketched here (AUC 0.87) — worth a proper abstention-tradeoff curve.
- Nucleus-strategy score capture (026) would test whether sampling widens the entropy gap.

## Artifacts
`out/{scores.json (1500 × per-token stats + token strings + cos), meta.json (determinism 0.998),
results.json (span stats + detector AUCs + Brier), span_spotcheck.json (20 hand-verified spans)}`.
`src/{capture_scores.py (GPU decode-with-scores + determinism gate), analyze_scores.py (span+detector,
numpy-only), run_022.sh, HARVEST_NOTES.md}`. Box out ~small; no push.