# RESULT — 021 fabrication-taxonomy

**Verdict: NEGATIVE-vs-C4-magnitude + POSITIVE-taxonomy (certified, T3-exploratory).**
On 1500 clean real free-form sentences, SONAR's greedy round-trip **semantic**-fabrication rate is
**3.6%** (nucleus 4.1%) — an **order of magnitude below** the C4 ~20-30% headline — BUT it is
length-monotone, **entity-substitution-dominated** (72% of fabrications are entity+number), and
**strongly cosine-gate-blind** (65% of fabrications occur at cos>0.95). The judge is reliable and,
if anything, slightly stricter than the manager's hand audit — so the low rate is NOT judge leniency.

## What ran
- **Corpus (out/corpus.json).** 1500 sentences from `w40/data/cpool/cpool_texts_{000,001,002}` (=
  `nickypro/sonar-sae` `data_v2/sent_texts`, verified the SAME free-form family the C4 `real_parascope`
  audit drew from). Clean-sentence filter → 239,181 candidates; shuffled `default_rng(20210802)`,
  tokenized with the SONAR NllbTokenizer, stratified by **SPM token length** into {8-15,16-25,26-40} ×
  **500 each** (all bins full).
- **Round-trip (out/pairs.json, box GPU0, 62s).** Canonical SONAR `TextToEmbeddingModelPipeline` encode
  → z; decode **greedy** (beam_size=1) AND **nucleus** (`TopPSampler(p=0.9)`, `torch.manual_seed(21)`);
  re-encode each reconstruction. Per pair: chrF, cosine(z_orig,z_re), exact-match. Means: **exact_greedy
  0.573**, exact_nucleus 0.531, cos_greedy 0.992, chrF_greedy 95.2.
- **Judge (LOCAL codex `gpt-5.6-sol`, `--effort low`).** 1344 non-exact pairs labeled with 8-category
  RUBRIC_V1 (batch 50, **0 parse errors** across all 3000 labels; exact pairs auto-faithful). 150-pair
  stratified subsample re-judged with differently-worded RUBRIC_V2. Manager hand-labeled 40 stratified
  pairs BEFORE codex (out/calibration_hand.json).

## Headline numbers
| strategy | fab rate | 8-15 | 16-25 | 26-40 | top categories |
|---|---|---|---|---|---|
| greedy  | **0.036** | 0.018 | 0.028 | **0.062** | entity-sub 36, garbled 5, added 4, number 3, dropped 3, role-swap 2, negation 1 |
| nucleus | **0.041** | 0.022 | 0.022 | **0.078** | entity-sub 41, garbled 5, added 5, number 5, role-swap 2, dropped 2, negation 1 |

- **Length-monotone** (greedy strictly increasing; nucleus non-decreasing, 26-40 ≫ 8-15).
- **Entity-substitution dominates.** entity+number = 72% (39/54) of greedy fabrications.
- **Nucleus > greedy** overall and in the long bin (surface stochasticity adds drift).

## Gate blindness (the C4 core — quantified)
Greedy fabrication rate falls sharply with cosine, but the mass sits at high cosine:

| cos bin | n pairs | fab rate |
|---|---|---|
| <0.90 | 10 | 0.700 |
| 0.90-0.95 | 30 | 0.400 |
| 0.95-0.98 | 124 | 0.161 |
| **0.98-1.00** | **1336** | **0.011** |

→ **35/54 = 64.8%** of all greedy fabrications (nucleus 70.5%) occur at **cos>0.95**. A cos>0.95 gate
would MISS ~2/3 of fabrications. Concrete gate-invisible cases: "up to life in prison"→"life in prison"
(cos 0.988, number/scope), "reloading"→"recharging" (0.993), "military"→"army" (0.991). chrF separates
better but overlaps (fab 82.0 vs faithful 95.7).

## Sanity checks / calibration
- **Judge reliability (V1 vs V2, n=150): binary κ=0.885, 98% binary agreement; 8-way κ=0.633.** Taxonomy
  read WITHOUT a low-agreement caveat.
- **Hand vs judge (n=40): 95% 8-way agreement (38/40).** Both disagreements are the judge being
  *stricter* than the manager (greedy:811 "Mark's Square"→"Mark Square" judged entity-substitution;
  nucleus:377 "what does the future of TV hold?"→"what holds the future of TV?" judged role-swap). Manager
  hand-fab-rate on the 40 = 0.000 vs judge 0.050. Binary κ = 0 only because the hand column has no positive
  class (undefined, not disagreement). **Direction: the ~3.6% is if anything an over-estimate vs a careful
  human — the C4 "judges are over-strict, hand-audit revises down" lesson holds, but here the strict-vs-
  lenient gap is tiny and both land ≪ 20-30%.**
- Round-trip identity sanity: 57% exact-match, cos_mean 0.992 (high-fidelity regime).
- GPU0-only guard honored; smoke (6 sents/12 pairs incl. a real codex batch) passed before full.

## Prediction Brier (frozen preds a-e; mean **0.179**)
| pred | p | outcome | brier | note |
|---|---|---|---|---|
| (a) greedy rate ∈[0.15,0.35] | 0.45 | **0** (0.036) | 0.203 | miss — clean sentences fabricate ≪ C4 |
| (b) monotone with length | 0.72 | 1 | 0.078 | hit |
| (c) entity+number >50% of fabs | 0.35 | **1** (72%) | **0.423** | big miss — badly underpriced entity dominance |
| (d) ≥5% of fabs at cos>0.95 | 0.70 | 1 (64.8%) | 0.090 | hit — gate blindness confirmed |
| (e) nucleus > greedy | 0.68 | 1 | 0.102 | hit |

## Honest interpretation — why 3.6% not 20-30%?
Two attributable causes, NOT a contradiction of C4:
1. **Definition.** This measures *semantic* fabrication (faithful synonym/morphology paraphrase EXCLUDED
   as non-fabrication, and human-confirmed as such). The C4 ~20-30% came from a decode-parse / any-content-
   deviation hand-audit, a looser bar. On the same corpus family, "any non-exact" here would be ~43%
   (greedy), and "any-non-exact minus pure-synonym" is what shrinks it to 3.6%.
2. **Input cleanliness.** The clean-sentence filter removed fragments/headers/truncations (the
   `**The Sugar Maple...**\n\n`-style inputs that dominate the raw C4 `rp_real` corpus). Those messy,
   off-distribution inputs fabricate far more; well-formed sentences round-trip at 57% exact / 3.6% fab.
So C4's high rate is real for messy on-manifold text; **for well-formed free-form sentences the semantic-
fabrication rate is ~4%, but it is length-scaling, entity-substitution-shaped, and mostly invisible to a
cosine gate** — the operationally important qualitative claims survive intact.

## Limitations
- Single-judge taxonomy (codex/gpt-5.6-sol) + 40-pair human spot-check; NOT a full human audit. T3.
- "Fabrication" = ANY non-faithful category; the priority-tie-breaking in the rubric assigns one label per
  pair, so co-occurring categories are under-counted (dominant-effect only).
- Corpus is clean well-formed sentences by construction; does NOT speak to fragmentary/noisy inputs (that
  regime is 023 off-manifold-dose / the raw-rp corpus).
- Length stratified by SPM tokens; word-length bins would shift bin boundaries slightly.
- Bin cell counts for fabrications are small (9-39 per cell) → bin rates are indicative, not tight.

## Follow-up worth funding? **Y**
- **022 entropy-signature** and **030 confidence-calibration**: the gate-blind high-cosine fabrications
  (entity-sub at cos>0.99) are the exact cases a token-entropy or decoder-confidence detector must catch —
  this experiment hands them a labeled positive set (out/labels_v1.json + pairs.json).
- **026 decoding-strategy**: nucleus>greedy head-start confirmed here (do not over-claim; beam not tested).
- **029 audit-v2**: run the same corpus through decode-parse+NLI to reconcile the 3.6% (semantic) vs the
  C4 ~20-30% (decode-parse) numbers head-to-head — the definitional gap is now the key open question.
- A raw-`rp_real` (unfiltered, fragment-inclusive) rerun would test cause #2 directly.

## Artifacts
`out/{corpus.json, pairs.json (1500, both strategies + metrics), meta.json, labels_v1.json (3000),
labels_v2.json (150), subsample.json, calibration_hand.json (40), calibration_keys.json, results.json}`.
`src/{build_and_decode.py, run_021.sh, judge_codex.py, analyze.py, HARVEST_NOTES.md}`. Box out ~1 MB (kept,
under cap). No push.