# 029 audit-v2 — decode-then-verify (NLI) auditor: end-to-end P/R on labeled corruptions

**Verdict: ★ CLEAN NEGATIVE (certified, all gates pass) — T3-exploratory.**
A decode-then-verify **NLI auditor does NOT detect TAE fabrication usefully, and is actively
counterproductive**: NLI-contradiction ranks fabrications *below chance* (AUC 0.478 on all pairs)
because TAE fabrications are minimal high-fluency content edits (entity/number/role swaps) that NLI
reads as non-contradictions, while it false-alarms on benign synonym paraphrase. The cheap
cos+entropy signals remain the best detector but **cannot reach a deployable operating point**
(precision ≈ 0.17-0.30 @ recall 0.8) at the 2-4% base rate. The 3.6%-clean vs 20-30%-messy gap
**is reconciled mechanically** by input cleanliness (fixed auditor: 0.5-3.9% clean, 24% messy).

## What ran
- **Corpus/truth**: 021's frozen 1500 (orig, greedy-recon) pairs; positive = 021 codex label ≠
  faithful-paraphrase. **N=1500, 54 fab (3.60%)**; gate-blind subset cos_greedy>0.95 **n=1460,
  35 fab (2.40%)**. (Fab mix: entity-sub 36, garbled 5, added 4, number 3, dropped 3, role-swap 2, negation 1.)
- **NLI verifier** (reused the original program's `rp_judge.py` stack): primary
  `cross-encoder/nli-deberta-v3-large` bidirectional (A=orig→recon, B=recon→orig) + secondary
  `facebook/bart-large-mnli` (dir A). Scores: `nli_contra=max(conA,conB)`, `nli_entdrop=1-min(entA,entB)`.
- **Cheap signals**: cosine gate `1-cos_greedy` (021; = deployable self-consistency cos, 025);
  decoder entropy (022 `scores.json`, per-sentence max/mean token entropy).
- **Eval**: each detector ALONE + logistic stacks (60/40 stratified × 3 seeds, fit on full train,
  eval on full test AND gate-blind test), on all pairs and the gate-blind subset. Reconciliation:
  same NLI flag rule across 021-clean, rp_real-autodec (clean roundtrip), rp_real-tae (messy
  inferred-zhat) — the latter two already NLI-scored in the program's `rp_nli_real.json`.
- **NOT run**: 024 prior-implausibility is per-token on only ~371 sentences (not full-corpus) → excluded
  from the stack (noted, not an instrument failure). No re-encode/re-decode of 021 (reused frozen artifacts).

## Numbers

**Single detectors — AUC / precision@recall0.8:**
| detector | all AUC | all P@R0.8 | gate-blind AUC | gb P@R0.8 |
|---|---|---|---|---|
| cosine (1-cos) | **0.920** | 0.171 | **0.892** | 0.090 |
| entropy_max | 0.818 | 0.069 | 0.773 | 0.042 |
| nli_contra (deberta) | **0.478** | 0.036 | 0.453 | 0.025 |
| nli_entdrop (deberta) | 0.600 | 0.038 | 0.577 | 0.026 |
| bart_contra | 0.607 | 0.037 | 0.558 | 0.024 |

*(cosine AUC 0.920 reproduces 022's 0.917 → pipeline sane. Base rate 3.60% / 2.40% → P@R0.8 ceilings are low.)*

**Logistic stacks (CV test) — AUC / P@R0.8:**
| stack | all AUC | all P@R0.8 | gb AUC | gb P@R0.8 |
|---|---|---|---|---|
| cos | 0.931 | 0.265 | 0.915 | 0.142 |
| ent | 0.886 | 0.170 | 0.857 | 0.101 |
| nli | 0.658 | 0.042 | 0.665 | 0.029 |
| **cheap (cos+ent)** | **0.933** | **0.298** | **0.919** | 0.174 |
| full (+nli) | 0.929 | 0.274 | 0.913 | 0.197 |

**cos+entropy is the best detector everywhere; adding NLI does NOT help** (all: −0.004 AUC; gate-blind:
−0.006 AUC, P@R0.8 +0.023 but within seed noise sd≈0.10). NLI alone is near-useless (AUC 0.66 stacked).

**Why NLI fails (mechanism, not a bug):** mean nli_contra separates (fab 0.112 ≫ faithful 0.006) but the
RANK does not — **85% of fabs have contra<0.1 (median 0.001)**; role-swap → **exactly 0.000**, number-change
median 0.002, entity-sub median 0.001 (35/36 missed). Only negation-flip (0.996) and a few gross swaps fire.
Meanwhile faithful *paraphrases* carry a higher median contradiction (0.0022) than the razor-subtle fabs, and
NLI false-alarms on benign synonyms (streetcar→tramway contra 0.987; award→medal 0.940). TAE fabrication =
high-fluency single-content-word edits — exactly NLI's blind spot (it scores sentence-level logical relations,
not entity/slot fidelity). This is the SAME blind spot as the cosine gate (025), not a complementary signal.

**Reconciliation (same NLI-contradiction auditor, flag = max bidir contra > 0.7):**
| corpus | input | flag rate |
|---|---|---|
| 021 greedy | clean (true-z decode) | **0.47%** |
| rp_real autodec | clean roundtrip | **3.85%** |
| rp_real tae | messy (inferred-zhat, off-manifold) | **24.0%** |

Holding the auditor fixed and varying only input cleanliness reproduces the ~20× clean→messy gap → the
3.6%-vs-20-30% discrepancy is **input cleanliness, not method**. Caveat: NLI-contra *under-reports* clean
fabrication (021: 0.47% flagged vs 3.6% codex-true) because it misses subtle fabs; the clean autodec corpus
happens to land at 3.85% ≈ 3.6%. The entailment-drop rule (min-bidir-entail<0.5) is unusable — it flags 92.5%
of messy pairs (fires on length/detail mismatch), confirming contradiction (not entailment-drop) is the right rule.

## Predictions (frozen in PREREG_LITE) → resolution
- (a) P=0.80 NLI AUC>0.75 all → **FALSE** (0.478). Large miss.
- (b) P=0.45 combined P@R0.8>0.5 gate-blind → **FALSE** (0.197).
- (c) P=0.50 NLI adds >0.05 AUC over cos+entropy gate-blind → **FALSE** (−0.006).
- (d) P=0.88 reproduce ~3.6% clean AND >15% messy → **FALSE** on the pre-registered 021-clean referent
  (021 = 0.47%, not 3.6%); messy 24% > 15% ✓. *Lenient reading (autodec-clean 3.85% ≈ 3.6%) → TRUE.*
- **Brier = 0.467** (strict, 021 referent; 0.277 under the lenient (d)). I was systematically over-confident
  that NLI would be a useful semantic detector — the honest, humbling miss is the result. (d)'s peek-informed
  0.88 was on the wrong referent: the peek confirmed autodec/messy, but 021-clean under-flags.

## Gates
- **G1 NLI sanity PASS**: identical sentences entail min 0.980 (>0.8); known contradictions contra min 0.999 (>0.7).
- **G2 label alignment PASS**: recomputed base rate 3.60% == 021; mean contra fab 0.112 ≫ faithful 0.006 (mean
  separates; rank does not — the finding). Gate-blind = 65% of fabs (== 021).
- **G3 not-trivial PASS**: base rates reported (3.60% / 2.40%); best detector (cheap stack) AUC 0.93 ≫ 0.5 majority.

## Limitations
- One corpus (021 SONAR/English), greedy decode only, tiny positive count (54 all / 35 gate-blind → CV P@R0.8
  sd≈0.10, high variance). Two cached NLI models only. Reconciliation clean-side is corpus-dependent (0.47%
  021 vs 3.85% autodec). 024 prior feature not full-corpus (excluded). Correlational, T3.

## Follow-up worth funding? **Y (narrow).**
The NLI/decode-then-verify path is decisively refuted for TAE monitoring — do not pursue sentence-level NLI.
The failures are all **slot-level** (entity/number/role), so a targeted **structured-slot checker** (extract
(agent,verb,patient,numbers,named-entities) from orig and recon, compare slots) is the mechanistically-indicated
next auditor — connects to 086 (latent-monitor) and the binding-program's who-did-what focus. The standing
lesson stands reinforced: **no cheap post-hoc auditor reaches a deployable precision at TAE's 2-4% fabrication
base rate**; abstention/fail-closed decoding (087) is the more promising lever than detection.