# E01: n-gram anatomy of SONAR token states and pooled z — RESULT

Run `full_v4` finished 2026-09-11 00:37 UTC on 4l GPU 2 in 1,024 s. It used 8,000 pile-10k sentences (226k tokens), a
doc-disjoint split stratified by `pile_set_name`, and prereg v1+v2+v3+v4. The breaker pass
(`breaker/BREAKER_REPORT.md`) found no bug. Every registered outcome is unchanged under every robustness variant;
three interpretive claims were narrowed. Outputs are in `out_full_v4/`.

## 4-line debrief
1. **Question:** how much of SONAR's final (L24n) token states, and of the pooled z, is a linear n-gram code:
   own token, ±1/±2 neighbours, order-free bag, and sampled bigram interactions?
2. **Answer:** on content tokens, the own token explains .37 linearly and about .45–.50 non-parametrically. Local
   linear context adds about .02 (group ridge .016). The bag adds under .005. The sampled bigram interactions add
   about .005, but that instrument is nearly blind to generic interactions. About half of the variance is not
   recoverable by any local model at this n. A randomly initialised copy of the same encoder is .94 explained by the
   same linear features. Pooled z is only .23 explained (F4, centred).
3. **So:** training turns SONAR's token states from a near-linear n-gram code into something dominated by a
   non-local or data-hungry component. Reverse-engineering z as a sum of low-order n-gram contributions is ruled out
   at the linear level. The largest recoverable pieces are own-token nonlinearity (+.08–.12) and an unexplained
   ~.5, which is the target of the next study.
4. **Side result:** H3's "role from the reversal-odd component at .76–.86" is a noun-fingerprint artefact. A random
   per-noun Gaussian vector scores .925 under the same prop-grouped CV, and real z_o falls to .51 on noun-disjoint
   splits. See CLAIM_UPDATES U8.

## Registered predictions (content tokens, L24n; primary R² / test-normalised R²_iso)

| P | registered | value | status | p | Brier |
|---|---|---|---|---|---|
| P1 | R²(F0) < .35 | .370 / .191 | metric-dependent, scored FALSE | .55 | .3025 |
| P2′ | Δ_local(trained) − Δ_local(random) ≥ .05 | .016 / .016 | FALSE | .55 | .3025 |
| P3 | Δ_global ≥ .05 | .002 / .002 | FALSE | .50 | .2500 |
| P4 | residual ≥ .30 | .606 / .804 (random-init .062) | TRUE | .65 | .1225 |
| P5 | asymmetry, descriptive | r = .775 | (descriptive) | — | — |
| P7′ | H3 direction on ẑ_o[F6], AUC ≥ .70 | .515 | FALSE | .35 | .1225 |
| P8′ | R²(F0) < .60 by L12 | .337 | TRUE | .60 | .1600 |
| P9 | Δ_interact ≥ .05 and ≥ .03 over C1-F6 | .002; .009 | FALSE | .55 | .3025 |
| P10 | pooled R²(F4) ≥ .50 | .233 | FALSE | .55 | .3025 |
| P11 | pooled R²(F6) − R²(F4) ≥ .05 | .033 (ordered-specific .0055) | FALSE | .45 | .2025 |

**Mean Brier over 9 scored predictions: .230**, which is barely better than a constant .5 forecast (.25). I
systematically over-predicted linearly readable local and global context, and under-predicted how non-n-gram the
trained states are.

## Gates and controls
- **Gates:** G0 .9999936, G1 cos_min .9999997, G3 .438, mean-predictor R²_iso 5e-8. All pass.
- **C1:** Δ_local .019, below its .08 control line. **C1-F6:** F6 − C1F6 = .009.
- **C6 (random-init encoder):** R²(F4) .938, Δ_local .003.
- **C5′:**
  - F2 exact pairs pass (8e-7).
  - F4 passes under the v4 definition (2.2e-6); the v3 all-pairs ratio is 3.6e-2.
  - **F2 on non-exact pairs FAILS** (mean cos .959 against .99). This is definitional: only the 78 objrel pairs
    whose token counts differ fail, and same-length pairs give cos .99999999993.

## Breaker qualifications (post-hoc robustness, not re-scoring)
- **Δ_local:**
  - The pipeline recovers planted effects of .113 and .028 as .112 and .028, and the null gives −.0001.
  - Group ridge gives .016, with a learning-curve asymptote of about .016–.023.
  - Part of Δ_local may be the linear shadow of interaction terms.
- **Interactions:** F6 captures only 4% of a generic planted bilinear bigram term, so **P9 FALSE does not show that
  interactions are small.** The ordered-bigram-specific pooled gain is .0055; the rest of P11's .033 is quadratic
  bag and capacity.
- **Residual:** about .08–.12 of it is own-token and position nonlinearity with no context. The final LayerNorm is
  not the explanation. Adding neighbours nonlinearly buys at most about .02. The remaining ~.5 couldn't be localised
  at n = 162k tokens.
- **Nested gains:** the negative ones are the dimension price of about 2k extra columns (≈ .0078 predicted). Quote
  deltas from group ridge.

## Claim tier
- **T2:** "SONAR L24n token states are poorly described by a linear n-gram code, while a random-init twin is
  well described by one."
  - Evidence: natural text, one model, one data size.
  - Robust to λ scheme, data size and an independent reimplementation.
- **T0:** anything about the *identity* of the unexplained half.