# RESULT — 025 fabrication-axis (fidelity-gate reliability, quantitative)

**Tier: T3-exploratory.** Box compute (200 base sentences × 20 nearest-neighbour interpolations ×
6 coefficients = 24 000 chord reconstructions + 200 dose-0 + a 600-record gradient-guided variant,
each greedy-decoded and **re-encoded twice** for the two fidelity cosines) + LOCAL codex 3-way
faithfulness judging (1275 unique reconstructions, **0 parse failures**) + analysis, run per the
frozen `PREREG_LITE.md`. One post-freeze addendum (two-gate reporting) was already logged in the
prereg; no other deviations.

## What was measured

A "fidelity gate" is a proposed monitor for a SONAR text autoencoder: after decoding a (possibly
corrupted) latent to text, re-encode that text and accept the reconstruction only if a cosine
similarity clears a threshold `T`. We quantify **how reliably this gate detects meaning changes** —
i.e. what fraction of reconstructions that are *fluent but no longer mean the same thing as the
original* the gate lets through (its miss rate), and how that varies with `T`.

Two versions of the gate are distinguished because they behave very differently:

- **self** — `cos(z', re-encode(decode(z')))`: the **deployable** auditor's actual check. It has
  only the (possibly-tampered) latent `z'` and asks "is the decode self-consistent with the latent
  I hold?" **This is the primary gate for the reliability verdict** (task framing).
- **orig** — `cos(z_orig, re-encode(decode(z')))`: a **non-deployable reference** that assumes the
  monitor already holds the clean original latent. It is the frozen-prereg referent for predictions
  (a/c/d) and the 023 comparison, reported alongside.

For each of 200 base sentences and each gate/threshold `T ∈ {0.90, 0.95, 0.97, 0.99}`, we filtered
the 20×6 interpolation grid to candidates passing the gate (`gate_cos ≥ T`, not degenerate) and
selected the one whose decode is **most semantically distant** from the original (max chrF-distance)
— i.e. the largest meaning change the gate would still accept for that sentence. Each selected
reconstruction was rated **faithful / meaning-changed-but-fluent ("altered") / garbled**.

## Verdict — the DEPLOYABLE self-consistency gate is essentially UNRELIABLE as a meaning-preservation check

At **every** threshold from 0.90 to 0.99, the self-consistency gate accepts a fluent reconstruction
that means something **entirely different** from the original for **~90% of sentences**, and
tightening the threshold does **not** help:

| gate | T=0.90 | T=0.95 | T=0.97 | T=0.99 |
|------|:------:|:------:|:------:|:------:|
| **self** (deployable) — miss rate for fluent meaning changes | **0.905** | **0.905** | **0.900** | **0.910** |
| self — mean chrF of accepted decode | 10.0 | 10.1 | 10.2 | 10.3 |
| **orig** (non-deployable reference) — miss rate | 0.315 | 0.215 | 0.080 | 0.035 |
| orig — mean chrF of accepted decode | 67.8 | 76.3 | 82.2 | 91.4 |

- The self gate's accepted decodes have **mean chrF ≈ 10** against the original — they are not
  subtle edits, they are **wholesale replacements by different fluent sentences**, and it passes
  them at cosine ≥ 0.99. Its faithful fraction is **0.000** at every threshold: the max-distance
  selection *always* finds a fluent-but-unrelated decode that clears the gate, for essentially all
  200 sentences (0 sentences had no passing candidate at any threshold).
- **Why:** self-consistency only asks whether the decode is a plausible sentence *for the latent the
  monitor holds*. Because that latent was itself moved (on-manifold, toward a real neighbour), a
  fluent decode of it is trivially self-consistent — the check never references the original meaning.
- The **orig** reference gate is far more reliable and genuinely tightens (miss rate 0.315 → 0.035
  as T → 0.99), **but it is not deployable** — it requires the clean original latent the monitor
  does not have. The gap between the two curves *is* the blind spot.

**Quantified blind spot (headline).** Of the 181 self-gate-accepted meaning-changed reconstructions
at T=0.95 (mean self-cosine **0.973**), **181/181 (100%)** have orig-cosine **< 0.95** (mean
**0.262**). Every reconstruction that fools the deployable gate would be caught instantly by a
monitor that had the original — the deployable auditor is blind to exactly the cases the reference
gate flags.

## The gate's blind spot — 5 verbatim largest meaning-changes accepted at self-gate ≥ 0.95

Each passes the deployable self gate (cosS ≥ 0.95) while being a completely different sentence; the
orig reference cosine (cosO) that would have caught it is shown for contrast.

1. **cosS 0.973 · cosO 0.188 · chrF 2.9**
   ORIG: *The Blue Mosque in Istanbul and the Topkapi Palace in Istanbul are renowned examples of Ottoman architecture.*
   RECON: *Sun, Y., & Korashan, M. (2011).*
2. **cosS 0.968 · cosO 0.161 · chrF 4.8**
   ORIG: *He believed that McClellan was not using his army effectively and that the nation was losing valuable time and resources.*
   RECON: *Hardy knew he had work to do.*
3. **cosS 0.978 · cosO 0.188 · chrF 5.6**
   ORIG: *In short, "Hear Me Calling / Natural Born That Way" is a double album that's not to be missed.*
   RECON: *My top pick was "Lady Bird".*
4. **cosS 0.976 · cosO 0.209 · chrF 5.8**
   ORIG: *Although he was never charged, he was known to have connections with several high-profile criminals of the time.*
   RECON: *Hardy knew he had work to do.*
5. **cosS 0.974 · cosO 0.237 · chrF 5.8**
   ORIG: *His journey shows that with determination, hard work, and a clear vision, anything is possible.*
   RECON: *It was a dream come true.*

For completeness, the same selection on the **orig** reference gate (cosO ≥ 0.95) yields only subtle
paraphrase-scale changes, because that gate is binding — e.g. *"We were so carefree, so oblivious to
the horrors…"* → *"We were so uncaring, so unaware of the horrors…"* (cosO 0.963, chrF 60); *"…the
cruelty and desperation of the characters' world"* → *"…the cruelty and hope of the characters'
world"* (cosO 0.953, an antonym/sentiment flip); *"We will not be silenced"* → *"We are not
silenced"* (cosO 0.976). These are genuine meaning changes but small — the reference gate caps the
achievable corruption at synonym/polarity edits, whereas the deployable gate caps nothing.

## Comparison to 023 (non-optimised chord)

On the **orig** reference gate, optimisation (K=20 nearest-neighbour interpolation + threshold
filtering + max-distance selection) buys only a modest sharpening over 023's fixed-dose chord: 025
reaches altered ≈ **0.215 at cosO ≥ 0.95**, comparable to 023's non-adversarial *dose-0.50* rate
(0.22) but achieved at a **higher** cosine (0.966 vs 0.911). So against a monitor that knows the
original, the gate remains genuinely binding and optimisation helps only a little — consistent with
023's core lesson. **The new and important result is entirely on the self gate**, which 023 did not
isolate: the *deployable* form of the same gate is ~90% blind regardless of threshold.

## Gradient-guided variant (secondary family, ran fully)

The gradient variant (teacher-forced NLL-max PGD on the latent, budgets {0.1,0.2,0.35}, 600 records)
**does not beat the chord family and does not change the bound**: at T=0.95 its altered rate is 0.11
(orig) / 0.115 (self) vs chord's 0.215 / **0.905**, and it fails the gate outright for a large and
growing share of sentences as T tightens (self no-passing 6 → 59 → 101 → 173 of 200 across
0.90→0.99). NLL-maximisation pushes the latent off-manifold and mostly produces garble that the gate
rejects; the on-manifold chord (interpolation toward a real neighbour) is what produces the fluent,
self-consistent, meaning-changed decodes. The worst-case reliability bound is set by the chord
family.

## Frozen predictions → outcomes → Brier (scored on the orig referent, per prereg)

| id | statement | p | outcome |
|----|-----------|---|---------|
| (a) | chord altered-but-fluent rate > 0.30 at cosRE(orig) ≥ 0.95 | 0.55 | **FALSE** (0.215) |
| (b) | the 5 worst cases include genuine meaning-inverting flips, not vague drift | 0.75 | **TRUE (adjudicated, borderline)** |
| (c) | miss rate rises monotonically as the threshold loosens 0.99→0.90 | 0.90 | **TRUE** (0.035→0.080→0.215→0.315) |
| (d) | gradient variant beats chord altered rate by > 0.10 at cosRE ≥ 0.95 | 0.30 | **FALSE** (grad 0.11 < chord 0.215) |

**Brier = 0.1163.** (a) missed on the *orig* gate — the frozen prediction was written against the
non-deployable referent, where the gate is binding; the striking failure is on the deployable self
gate, which the predictions did not target. **Prediction (b) is a manager adjudication and is the
weakest of the four**: the orig-gate worst-5 are synonym-drift-dominated with ~2/5 genuine
inversions (desperation→hope; "will not be silenced"→"are not silenced"), so "include genuine
inversions" holds but "not vague drift" is generous. **Brier is 0.1163 if (b)=TRUE, 0.2413 if
(b)=FALSE** — reported for transparency.

## Gates / sanity — ALL PASS

- **G1 gate constraint honoured:** **0** violations — every selected (non-fallback) reconstruction
  has `gate_cos ≥ T` by construction.
- **G2 dose-0 baseline:** 200 unperturbed decodes → faithful **0.94**, altered **0.045** (< 0.10),
  garbled **0.015** (< 0.05). Matches 021's 3.6% semantic-fabrication baseline (ballpark) →
  rubric+pipeline are not manufacturing meaning changes.
- **G3 cosine monotone in interpolation coefficient (orig gate):** 42/4000 partner-tracks (**1.05%**)
  violate by > 0.05 → PASS. (The **self** cosine is deliberately *non*-monotone in α — 3992/4000 —
  because self-consistency does not decay as the decode moves on-manifold toward a real neighbour;
  this non-monotonicity is itself the mechanism behind the self gate's blindness, not an error.)
- **G4 judge reliability:** manager hand-labelled 20 stratified pairs **before** viewing codex →
  **16/20 = 80% agreement**. All 4 disagreements are genuine boundary calls: codex was *stricter* on
  subtle paraphrase (2 cases the manager called faithful, codex called altered) and *more lenient* on
  truncated fragments/citations (2 cases the manager called garbled, codex called altered). V1 vs
  differently-worded V2 rubric on 150 pairs: agreement 0.90, **Cohen's κ = 0.838** (substantial).

## Caveats / limitations

- **Deployability framing is load-bearing.** The alarming ~90% number is the *self-consistency* gate
  (the realistic monitor). The *orig* gate is reliable but presupposes the clean original — if a
  monitor genuinely has ground-truth latents, a tight cosine gate (T ≥ 0.97) is a strong meaning
  check (miss rate ≤ 0.08). The result is "the *deployable* form is unreliable", not "cosine gates
  are useless in every setting".
- **Greedy-only** decode (no beam/nucleus) — the 020/022/023-validated deterministic path. A
  sampling decoder could populate the fluent-wrong region differently.
- **One corpus** (021's cpool = nickypro/sonar-sae, C4-real family), one encoder pipeline.
- **Single LLM judge** (codex, reasoning=low). Full V1 coverage (all 1275 unique reconstructions
  judged, not a subsample); κ=0.84 and 0 parse failures indicate stable labels, but the manager
  hand-check (80%) shows codex is somewhat **lenient on truncated fragments/citations**, which can
  modestly *inflate* the altered count on the orig gate's low-chrF tail (it does not affect the self
  gate story, whose accepted decodes are full fluent sentences, not fragments).
- **Selection is worst-case-per-sentence** (max achievable meaning change subject to the gate), so
  the reported rates are an *upper envelope* of what the gate can miss, not an average-case rate.
- T3-exploratory: a reliability characterisation of a monitoring check, not a promotable mechanism.

## Bottom line

The deployable self-consistency fidelity gate is **not a reliable meaning-preservation check**: for
~90% of sentences it accepts a fluent reconstruction that means something completely different, at
any threshold up to 0.99, and every such case would be caught by a monitor holding the original.
Anyone relying on re-encode self-consistency as a fidelity monitor for this autoencoder should treat
it as a fluency/plausibility check only, **not** as evidence that meaning was preserved.