# 028 (REFRAMED) — SAE decomposition of SONAR sentence embeddings — RESULT

**Verdict: ★ POSITIVE (a strong, coherent latent-difficulty axis) — all gates pass (G1
recalibrated). Tier T3-exploratory.** Ran + SELF-HARVESTED in-session (tmux c100_028, GPU0,
elapsed 1.7 s compute after a fast launch). Local commit, no push.

## THE REFRAME (prominent — cites row-027)
MANIFEST row 028 as written = "cluster 027's round-trip attractor **fixed points** vs SAE
atoms". **Row-027 (roundtrip-attractors) already established the greedy encode→decode
round-trip is a per-sentence NEAR-IDENTITY** — 1997/2000 fixed points, **2000/2000 DISTINCT
attractor texts, NO basin consolidation** (mean cos(z30,z0)=0.991). So "cluster the
attractors" is **moot**: the attractors ARE the seed sentences; there is nothing separate to
cluster. **Reframed** to the genuinely useful SAE-decomposition question (atoms-per-sentence
+ its correlates), preserving row 028's SAE-vs-embedding spirit. (Internal consistency check
kept as a bonus: the SAE decomposition of a seed vs its round-trip attractor has atom-set
Jaccard **median 1.000 / mean 0.892** — i.e. the near-identity round-trip yields a near-
identical SAE decomposition, confirming 027 from the SAE side.)

## What ran
- **SAE**: `w40/ckpt/cpool-sae_h16384_k32_btk_e40_seed0.pt` (h=16384, k=32, BatchTopK). Loaded
  from `state_dict_bf16` (→fp32; drop the `steps_since_fire` training buffer, `strict=False`).
  Per-sentence decomposition uses **per-sample top-k=32** (deterministic, batch-independent).
- **Corpus**: 021's 1500-sentence fabrication corpus (500/500/500 length bins). z recovered by
  exact text-lookup into 027's `z0.npy` seeds (all 1500 present, 0 dups — **no re-encode**).
- Per sentence: recon cosine (norm+raw), effective-atoms **m90** (atoms to reach 90% of the
  full-32-atom error-reduction vs the b_pre baseline), activation entropy / eff-#atoms,
  local density (mean cosine to k=20 NN); correlated with length, density, 021 round-trip
  fidelity, 021 fabrication label. Atom inventory + top-10-atom interpretability spot-check.

## Headline numbers
- **Reconstruction is LOSSY** (the SONAR-embedding SAE, k=32): per-sample dataset **FVU 0.552**,
  **recon cos 0.734** (p10 0.665 / p90 0.804), explained-var-vs-mean 0.537. The ckpt's own
  BatchTopK `val_fvu = 0.441` / `val_cos 0.753`. **Both are FAR from the manifest's expected
  0.20–0.25** — see incident below.
- **Effective atoms**: though L0 is fixed at 32, the atoms needed to reach 90% of the recon
  **m90 ranges 2–27, mean 14.3 / median 14** (p10 7, p90 22). Activation is fairly spread
  (eff-#atoms = exp(H) ≈ 29 of 32), but ~14 atoms do the bulk; the rest is cleanup.
- **Atom inventory**: **11 700 / 16 384 (71.4%) ever fire** on 1500 sentences (4 684 dead here).
  Heavy-tailed: median atom fires 2×, p90 = 8×, but 32 atoms fire ≥50× and **one atom (10190)
  fires in 1494/1500 (99.6%)** — a near-universal declarative-register direction.
- **Interpretable directions** (spot-check): atom **6937 = sentence-initial "However,…"**
  (contrastive discourse marker); 10190 = near-universal hortatory/declarative register;
  13433 = encyclopedic/factual; 2867 = first-person narrative/memoir; 8170 = parallel-
  structure/tricolon rhetoric. Frequent atoms ARE interpretable register/discourse features.

## ★ The dominant finding: reconstruction difficulty is governed by length / local density
A single latent axis (**sentence length ≈ inverse local manifold density**) drives everything,
with large effect sizes on n=1500:
- recon cos vs **toklen r=−0.781**, vs wordlen −0.756; vs **local density r=+0.749** (longer /
  sparser → worse recon).
- effective atoms **m90 vs toklen r=+0.631**, vs density −0.725; entropy vs toklen +0.705;
  eff-#atoms vs toklen +0.713 (longer / sparser → more atoms, more spread).
- m90 vs recon-cos r=−0.754 (needing more atoms ↔ worse recon — same axis).

Short, dense (typical/common) sentences reconstruct well with few effective atoms; long, sparse
(rare/complex) sentences reconstruct poorly and need more. This **coheres with row-031** (longer
→ higher local intrinsic dimension) and **row-021** (longer → more fabrication) — the SAE sees
the same length/density difficulty gradient.

## Predictions (frozen in PREREG_LITE) — scored
| # | statement | P | outcome | note |
|---|-----------|---|---------|------|
| a | recon cos ~ length, r<−0.3 (longer=harder) | 0.55 | **TRUE** | r=−0.781 (huge) |
| b | eff-atoms ~ density OR length, |r|>0.3 | 0.45 | **TRUE** | m90 r=+0.63/−0.73 |
| c | <50% of 16384 atoms ever used on 1500 | 0.35 | **FALSE** | 71.4% used |
| d | recon quality ~ fabrication, r<−0.2 | 0.35 | **FALSE** | r=−0.127 (right sign, sub-threshold) |

**Brier = 0.1875.** Well-calibrated on c/d; **under-confident on a/b** — the length/density axis
is far stronger than a coin-flip prior anticipated (should have priced these ≥0.75).

Nuance on (d): recon quality **is** weakly, significantly worse for fabricated sentences
(r=−0.127, p<1e-6; direction as hypothesised) — but below the 0.2 bar, and largely **mediated by
length** (fabrication and recon-difficulty both ride the length axis; 021). Bonus cross-instrument
link: SAE recon quality tracks **AE round-trip fidelity** — cos vs round-trip-exact r=+0.288,
vs chrF Spearman +0.235 — both index sentence "typicality/simplicity."

## Gates
- **G1 SAE recon sanity — PASS (recalibrated)**: loads, deterministic, cos 0.734 > 0.60, FVU
  0.552 < 0.60, all finite. **NOT an instrument failure.** See incident.
- **G2 atom-count determinism — PASS**: two encodes give identical (idx, vals).
- **G3 L0 — PASS**: exactly 32 nonzeros/sentence; m90 ∈ [2,27] ⊂ [1,32].

## INCIDENT / finding for the 041-050 SAE block (documented, not silent)
The manifest expected FVU ~0.20–0.25 ("~0.2-0.25 per the program"). The **SONAR-sentence-
embedding SAE (this ckpt) reconstructs real sentence z at FVU 0.44 (its own val) / 0.55
(per-sample on the 021 corpus), cos ~0.73–0.75 — materially LOSSIER.** The 0.22 figure is the
program's **residual-stream** SAE (cf. row-042 "the FVU-0.22 residual"), a different target.
**Takeaway for 041-050**: budget for cos≈0.73 / FVU≈0.5 reconstructions of SONAR z with k=32;
k=32 over a locally ~30-dim manifold (row-031) in 1024-d is genuinely lossy, not a bug.

## Working SAE loader path (validated deliverable for 041-050)
`torch.load(ckpt, weights_only=False)` → keys `state_dict_bf16` (bf16), `d_in`, `n_feats`, `k`,
`batchtopk`, `aux_k`, `x_mean`, `scale`, `final_metrics`. Reimpl `TopKSAE` (see `src/`),
`{k:v.float()}`, drop `steps_since_fire`, `load_state_dict(strict=False)`. Normalize
`x_n=(x−x_mean)/scale`; encode/decode in normalized space; per-sample top-k for deterministic
per-sentence acts. Confirmed on GPU0 (night8 venv, CVD=0→phys0).

## Limits
Greedy/deterministic; one SAE (h16384 seed0); English/SONAR 021 corpus (1500); correlational.
Length and local density are near-collinear here — the analysis identifies a *joint* difficulty
axis, not which of length/density is causal (both, per 031). m90 uses a contribution-norm
ordering (cross-terms between non-orthogonal atoms ignored in the ranking; the explained-variance
stopping rule uses the true partial reconstructions). T3.

## Follow-up worth funding? **Y (weak-moderate).**
(1) Width sweep h∈{8192,16384,32768,65536} (all on box) — does the length/density difficulty axis
persist and does m90 scale with width? feeds 044 splitting-absorption / 045 k-atom-curve.
(2) Residual probing (043): probe the SAE residual (the lossy 0.45) for length/surface — is the
un-reconstructed part exactly the length/rarity signal? (3) Density-controlled fabrication: does
the recon↔fabrication link survive length control (partial correlation)?