# RESULT — 026 decoding-strategy

**Verdict: VALID (T3-exploratory). All gates pass.** How SONAR's text decoder's DECODING STRATEGY
trades off round-trip fidelity vs diversity, and whether beam search reduces semantic fabrication vs
greedy. Headline: **beam search does NOT reduce semantic fabrication vs greedy** — greedy already sits
at this autoencoder's faithfulness ceiling (~3-4% fab), and beam4/beam8 are statistically
indistinguishable from it on semantics (they buy only a marginal surface-fidelity gain). **Temperature
is the dominant knob**: full-vocab temp-1.0 causes a fidelity CLIFF (cos 0.77, 56% fabrication) and
specifically inflates entity-substitution ~11x. Brier (4 frozen predictions) = **0.2156**.

Judged rates on the fixed 300-id subsample (100/bin); fidelity metrics (exact/chrF/cos) on the full
1500. greedy + nucleus-0.9 labels REUSED from 021 `labels_v1.json` on the SAME 300 ids (not re-judged);
5 NEW strategies judged LOCALLY with codex (gpt-5.6), 021 RUBRIC_V1 verbatim (neutral framing),
batches of 50, **0 parse failures / 859 non-exact jobs**. Exact-match pairs auto-labeled faithful.

## Per-strategy table

| strategy | exact (1500) | chrF | cos mean | cos q05 | faithful% | fab% | entity-sub% | garbled% | self-BLEU |
|----------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| **beam8**   | **.589** | **95.48** | **.9929** | .968 | 96.3 | 3.7 | 2.0 | 0.3 | 1.0 (det) |
| **beam4**   | .586 | 95.42 | .9929 | .968 | 96.0 | 4.0 | 2.3 | 0.3 | 1.0 (det) |
| **greedy**  | .573 | 95.19 | .9919 | .966 | **97.0** | **3.0** | 1.7 | 0.3 | 1.0 (det) |
| nucleus0.9  | .531 | 94.41 | .9915 | .962 | 96.0 | 4.0 | 2.3 | 0.3 | .926 |
| nucleus0.95 | .523 | 94.34 | .9911 | .963 | 95.3 | 4.7 | 3.7 | 0.3 | .914 |
| temp0.7     | .464 | 93.24 | .9892 | .954 | 94.0 | 6.0 | 4.3 | 0.3 | .874 |
| **temp1.0** | **.074** | **73.78** | **.7737** | .483 | **44.0** | **56.0** | **19.0** | **20.0** | .391 |

faithful% / fab% on 300-subsample (fab = 1 − faithful, judge-defined). greedy fab here = 9/300 = 3.0%
(subsample slice of 021's full-1500 3.6%). Rows sorted by faithfulness.

## Findings

1. **Beam ≠ less fabrication.** greedy fab 3.0% (9/300 non-faithful) vs beam8 3.7% (11/300) vs beam4
   4.0% (12/300): indistinguishable at these small counts — greedy is nominally the LOWEST. Beam does
   improve *surface* fidelity (exact +1.3–1.6 pt, chrF +0.2–0.3, cos +0.001) — a mode-seeking/determinism
   effect — but that does not translate into fewer meaning changes. The AE is already near its
   faithfulness ceiling under greedy; there is little semantic headroom for beam to recover.

2. **Temperature CLIFF at temp-1.0.** Removing the nucleus cap and raising temp to 1.0 (full-vocab
   sampling) collapses everything: exact .074, chrF 73.8, cos mean 0.774 (q05 = 0.48 — a large tail of
   totally-off reconstructions), faithful 44%, fab 56%, garbled 20%. temp-0.7 (also p=1.0) degrades
   only mildly (faithful 94%, fab 6%), so the collapse is between 0.7 and 1.0, not linear.

3. **temp-1.0 inflates entity-substitution specifically (~11x): 1.7% → 19.0%.** It is the single
   largest fabrication category at temp-1.0 (57/300), consistent with 021's finding that
   entity-substitution dominates this AE's fabrications. But garbling rises comparably (0.3% → 20%),
   and dropped-content (29) and added-content (13) also spike — so temp-1.0 is broadly destructive, not
   *only* entity-specific. Below temp-1.0, entity-sub climbs monotonically with entropy
   (greedy 1.7 < beam ~2.0-2.3 < nuc0.95 3.7 < temp0.7 4.3), the taxonomy link 021 predicted.

4. **Monotone fidelity/diversity tradeoff.** Ranking by every fidelity axis (faithful, chrF, cos, exact)
   and by diversity (self-BLEU, lower = more diverse) is the SAME order:
   **beam (max faithful, zero diversity) → greedy → nucleus-0.9 → nucleus-0.95 → temp-0.7 → temp-1.0
   (max diversity, min faithful).** Diversity is bought directly with faithfulness; there is no
   free-lunch strategy that raises diversity without losing fidelity.

5. **Faithfulness ranking:** greedy (97.0) ≥ beam8 (96.3) ≈ beam4 ≈ nuc0.9 (96.0) > nuc0.95 (95.3)
   > temp0.7 (94.0) ≫ temp1.0 (44.0). The top five are all within ±1.7 pt on 300 samples (within noise);
   only temp-0.7 and especially temp-1.0 separate. For a faithfulness-maximizing decode, **greedy or
   beam are interchangeable**; the actionable rule is simply *do not sample at temp ≥ ~1.0 full-vocab*.

## Predictions (frozen, PREREG_LITE) → Brier

| # | claim | P | outcome | detail | Brier |
|---|-------|:--:|:--:|-------|:--:|
| a | beam8 faithfulness > greedy by >2 pt | .30 | **FALSE** | Δ = −0.67 pt (beam8 96.3 < greedy 97.0) | .0900 |
| b | temp1.0 fab > greedy by >3 pt | .60 | **TRUE** | Δ = +53 pt (56.0 vs 3.0) | .1600 |
| c | exact ordering beam > greedy > nucleus > temp | .30 | **TRUE** | .586/.589 > .573 > .531/.523 > .464/.074 | .4900 |
| d | beam8 has LOWEST fab of all 7 | .35 | **FALSE** | greedy 3.0% is min; beam8 3.7% | .1225 |

**Brier mean = 0.2156.** (a) The AE's near-ceiling faithfulness left no room for beam to beat greedy by
2 pt — correctly low-prob. (d) fab-count ties at n=300 made "beam8 strictly lowest" a coin-flip; greedy
edged it. (c) hit but was under-confident (P=.30 on a clean monotone ordering) → the dominant Brier
penalty. (b) hit as expected (the one high-prior call).

## Gates

- **G1 determinism — PASS.** Re-decoded greedy == 021 stored greedy on **1500/1500 = 100%** (≥0.99
  required). Soft check: nucleus-0.9 seed-21 reproduction also **100%**. Corpus + pipeline reuse verified.
- **G2 cosine well-defined — PASS.** All re-encode cosines ∈ [−1,1]; greedy cos mean **0.99192** ≈ 021's
  0.9919 (exact match to 4 dp).
- **G3 hand spot-check — PASS.** 15 non-exact beam8 subsample pairs hand-labeled first, then compared to
  codex: **15/15 = 100% agreement** (all faithful-paraphrase). Caveat: beam8 non-exact diffs are all
  synonym/morphology/syntax (e.g. "wore on"→"went on", "presence of Russia"→"Russia's presence",
  "bygone"→"past") so the sample contains no positive-fabrication case — G3 confirms codex does not
  OVER-flag faithful paraphrases, but does not test its fabrication recall here (021's hand-audit, 95%
  agreement over 40 mixed pairs, covers that).

## Caveats / limitations

- **Near-ceiling small counts.** Among the top-5 strategies, fab differences are 9–14 non-faithful of
  300 — not statistically separable. Only temp-1.0 (168/300) and temp-0.7 (18/300) stand out. Claims
  about greedy-vs-beam semantic ordering should be read as "tied at the ceiling," not "greedy wins."
- **Single judge.** One LLM (codex/gpt-5.6) + a 15-pair hand check (all one class). Rates are
  judge-defined; T3 ceiling, no promotion without a human-audit pass.
- **1 corpus, 1 seed** per stochastic primary (seed 21). Self-BLEU uses 3 seeds (21/22/23) on 300 sents.
- **nucleus vs temperature axes are separated** (nucleus varies p at temp 0.6; temp varies temp at p=1.0),
  so temp-0.7 and nucleus-0.9 are not on a common curve — cross-axis comparisons are design-caveated.

## Follow-up

Marginal. The practical answer is settled (greedy/beam interchangeable and near-ceiling; avoid
high-temp full-vocab sampling). A worthwhile extension would be a temp sweep between 0.7 and 1.0 to
locate the collapse knee, and pairing temp-1.0's entity-substitution spike with 021/024's
entity-fabrication mechanism. Not high priority.