# RESULT — 041 crosscoders over (L12, L24, z): do atoms originate mid-stack?

**Tier: T3-exploratory.** GPU0/phys0. Harvested 2026-08-03. Verdict: **NEGATIVE on the core
hypothesis — dictionary atoms do NOT originate mid-stack and persist to the pooled embedding z; they
are overwhelmingly depth-LOCAL, with a near-vanishing shared fraction. One incidental positive: the
shared encoder measurably improves z reconstruction.**

## What actually ran
Trained ONE BatchTopK crosscoder (shared encoder over concat(L12ₙ, L24ₙ, zₙ) ∈ ℝ³⁰⁷², h=16384, k=32,
250 epochs) with three per-rep unit-norm decoders, plus three single-rep TopK SAE baselines on the same
data. Data = 40,000 pile-10k sentences (36k train / 4k val) encoded once through SONAR with layer hooks
(L12 = layer_idx 11 token-mean; L24 = layer_idx 23 token-mean, pre-final-LN; z = pooled SONAR embedding).
Per-rep 028-style normalization. Wall ≈ 1800 s. `out/`: `results.json`, `atom_stats.npz`,
`crosscoder.pt`, `DONE`. run.log ends `=== 041 chain done Mon Aug 3 00:14:47 ===`.

Reps are genuinely distinct (pairwise cos L12·L24 = 0.758, L12·z = 0.150, L24·z = 0.095; raw norms L12
≈ 1932, L24 ≈ 18357, z ≈ 0.25 — the L24 pre-LN scale pathology the prereg flagged, absorbed by norm).

## Headline numbers
- **Crosscoder trains cleanly**: final train MSE 0.094 (< 0.6× epoch-1 0.815); per-rep val FVU
  **L12 0.259 / L24 0.066 / z 0.643**; L0 = 32; dead fraction 16.3%.
- **Atoms are depth-LOCAL, not shared.** Of 13,706 firing atoms, only **26 are shared across all 3
  reps** (min_r f_r ≥ 0.15) = **0.19%** (predicted ≥25%). Local counts: **z-local 6838, L12-local 11,
  L24-local 0**. Active-per-rep: L12 13129, **L24 only 33**, z 13700.
- **Direction of information flow is z-ward, not L12-ward**: L12 active atoms (13129) < z active atoms
  (13700), the opposite of the "information narrows toward pooling" prediction. L24 barely registers as a
  distinct-firing rep (33 atoms) because it is the easiest to reconstruct (FVU 0.066) and needs little
  dedicated decoder norm.
- **Incidental positive — the shared encoder helps z**: crosscoder z-FVU 0.643 vs the z-only single-SAE
  baseline **0.752** (Δ = 0.109 improvement) — the ONLY rep where the crosscoder beats its baseline by
  >0.10, evidence that cross-depth structure aids the pooled embedding. (L12 and L24 crosscoder FVUs match
  their baselines within 0.007.)
- **Shared vs local interpretability (qualitative spot-check, no formal judge)**: the 26 shared atoms read
  as high-frequency grab-bags (mixed technical/LaTeX/multilingual max-activating sentences); the local
  atoms carry crisper themes (L12: legal/court filings, formal bios, abbreviation-periods; z: medical,
  numeric "four/six", section headers). Shared atoms do NOT look more interpretable than local ones.

## Sanity gates — ALL PASS
- **G1 trains**: final MSE 0.094 < 0.6× 0.815; no NaN/Inf.
- **G2 FVU sane**: every crosscoder rep FVU < 0.70 (max z 0.643); no rep ≥ 1. z-FVU is in/above the 028
  single-SAE ballpark (see context note). Not INSTRUMENT_FAILURE.
- **G3 non-degenerate**: dead fraction 0.163 (< 0.90); max single-atom usage 0.455 (< 0.95).

## FVU context (honest)
The embedding-SAE baseline FVU expectation from the SAE block is ~0.44–0.55. Here the **z single-SAE
baseline is 0.752** and the crosscoder z-FVU is **0.643** — both notably *worse* (higher) than that
context. Likely causes: different corpus (diverse pile-10k vs the campaign's curated corpus), different
h/k, and (for the crosscoder) the shared-encoder constraint. The crosscoder narrows the gap toward the
0.44–0.55 range but does not reach it. z remains the hardest of the three reps to reconstruct.

## Brier vs frozen PREREG_LITE.md predictions
- **(a) P=0.60** — ≥25% of firing atoms shared across all 3 → **FALSE** (0.19%).
- **(b) P=0.45** — L12 has MORE active/distinct atoms than z → **FALSE** (13129 < 13700).
- **(c) P=0.50** — shared atoms more interpretable than local → **FALSE** (qualitative; local themes are
  crisper). Note: p=0.50 so this contributes 0.25 to Brier either way.
- **(d) P=0.50** — crosscoder per-rep FVU within 0.10 of single-SAE baseline for ALL three → **FALSE**
  (L12 Δ0.0004 ✓, L24 Δ0.007 ✓, but **z Δ0.109 ✗** — and z is *better*, not worse).
- **Brier(a–d) ≈ 0.266.** All four predictions missed; the honest read is that the mid-stack-origination
  picture was simply wrong for this representation triple — the dictionary is depth-partitioned.

## Incident (honest disclosure)
SONAR's encoder has a 514-token positional cap. The first launch crashed on a 649-token pile sentence
(`ValueError: ... maximum sequence length (514) ... length 649`). The t42 repair added `MAX_TOK=500`
hard-truncation (keeps EOS) with a drop-count log. Truncation counts from the final run: **4 / 40,000
sequences hard-truncated** in the full encode (0 / 200 in smoke) — a negligible fraction, no material
effect on the dictionary.

## Limitations / what did not run
- One encoder card, one seed, English/pile only.
- Decoder-norm attribution is a proxy for "where a feature writes," not a causal test.
- No formal interpretability judge was run — (c) is a qualitative spot-check read.
- L24 pre-LN anisotropy (raw norm ≈ 18k) is absorbed by normalization but L24's tiny active-atom count
  (33) means its "depth-local" verdict rests on very few atoms.

## Follow-up worth funding? **Weak-Y.**
The clean negative (depth-partitioned dictionary, near-zero shared atoms) is itself informative and the
z-reconstruction-improvement-under-shared-encoding is a genuine lead. A tighter re-run (match the 028
corpus + h/k so z-FVU lands in 0.44–0.55; add a real coherence judge; sweep the share threshold τ) would
firm up whether ANY concepts persist across depth or whether SONAR's dictionary is strictly stratified.
Not urgent.