# RESULT — 042 nonlinear-residual: is the SAE residual structured or noise?

**Tier: T3-exploratory.** GPU0/phys0, tmux c100_042. Harvested 2026-08-03. Verdict: **STRUCTURED —
the embedding-SAE residual r = x_n − x_hat is decisively NOT matched-covariance Gaussian noise.
The two certified instruments both say so loudly: ICA excess kurtosis is ~69× the null band, and
adding even a lossy reconstruction of r back to x_hat recovers +25.3 chrF at decode (the Gaussian
control LOSES 4.4). But the structure is not "nonlinear beyond covariance" in the AE sense (P3
clean fail), and the fresh-SAE dictionary probe was underpowered (its own positive control failed
its gate) — so "residual has dictionary structure" remains open, while "residual carries most of
the decodable sentence content" is the headline.**

## What actually ran
Full chain per PREREG_LITE.md, smoke (2k/500) then full (36k/4k), total ~8 min wall
(encode 102s+14s, analyze 212s, decode 119s). Corpus: 041's 36k/4k pile-10k sentences re-encoded
to z via `TextToEmbeddingModelPipeline` (0 drops; val cross-check vs 041 cached Z: mean cos
0.999996, min 0.984). Residual from the w40 embedding SAE ckpt (h=16384 k=32 BatchTopK, per-sample
top-k). Methods, each on r vs matched-cov Gaussian null(s) vs x_n positive control, all on this one
corpus: (a) fresh TopK SAE h=8192 k=32, 40 ep × 5 fits; (b) PCA-256→FastICA-64 × 8 runs; (c) shallow
AE 1024-512-m-512-1024 vs exact PCA-m, m∈{32,128} × 8 fits; (d) greedy decode of 1000 val sentences
× 5 conditions with paired-bootstrap chrF deltas. run.log ends `=== 042 chain done Mon Aug 3
01:12:48 UTC 2026 ===`; `out/DONE` present. Repo `out/`: results_analyze.json, decode_results.json,
decode_samples.json, decode_texts.json (+ smoke/). Nothing truncated; everything in the prereg ran.

## Regime (the row's constant, honest)
- w40 SAE on THIS corpus: dataset val FVU **0.731** (per-sample mean 0.708), recon cos **0.608**,
  deterministic. The ckpt's own val FVU is 0.441 — this pile corpus is a distinctly **lossier
  regime** than the 021-corpus 0.44–0.55 figure (028 note). Smoke (n=500) measured FVU 0.772,
  which is **outside** the G1 band [0.35, 0.75]; the full-scale 4k value 0.731 sits just inside,
  so G1 formally PASSES — but at the band edge. All comparisons are internal to this corpus, so
  this shifts interpretation ("residual of a very lossy SAE"), not validity.
- Norm profile (lesson 3): ‖r‖/‖x_n‖ mean **0.79** — the residual carries ~4/5 of the vector norm;
  cos(r, x_hat) ≈ −0.05 (roughly orthogonal). This is a residual-dominated regime.
- r is broader-spectrum than x (eff. participation rank 456 vs 280; top-128 eigvals 33% vs 40% of
  var) and more heavy-tailed marginally (mean excess kurtosis 0.539 vs x 0.363 vs null −0.001).

## Numbers per method, each vs its own null
- **(a) fresh SAE (UNCERTIFIED — see G3)**: val FVU r **0.7604** vs null mean **0.7782** → gap
  **+0.018** (predicted ≥0.15). But the positive control only achieved gap +0.079 on x_n (< its
  0.10 gate): 40 epochs of h=8192 k=32 barely separates *real sentence embeddings* from Gaussian,
  so this instrument had no power to certify the residual cell → **INSTRUMENT_FAILURE for the
  dictionary question**, per prereg, not "noise". Direction of the (weak) gap is real-side.
  Qualitative spot-check of top residual atoms is coherent (atom 3804 = LaTeX/math, 6450 = legal
  prose, 133/413 = travel-guide listings, 5315 = facts-with-numbers) — suggestive of topical
  structure in r, unscored.
- **(b) ICA (certified)**: held-out mean |excess kurtosis| of top-16 comps on r **9.997** vs
  3-draw null band max **0.145** → **~69× the band** (predicted ≥2×). Positive control x: 11.98 vs
  band 0.125 (~96×). Massively non-Gaussian coordinate structure in r, nearly as strong as in x
  itself.
- **(c) shallow AE vs exact PCA (certified, key nonlinearity discriminant)**: at m=128,
  Δ = FVU_PCA − FVU_AE on r = **+0.0014** (needed ≥0.03), null Δ = −0.0123, x control Δ = +0.0248
  (> its null −0.0103 → control passes, though itself modest). At m=32: r −0.0015, null −0.0308,
  x +0.0360. So the AE beats PCA clearly on x, and on r only beats *its null's overfitting
  penalty* — **no meaningful nonlinear-manifold structure in r beyond its covariance** by this
  probe.
- **(d) decode (certified, G4 headroom 64.0 chrF)**: n=1000 greedy decodes, chrF mean:
  x **85.5** (exact 25.0%) · x_hat **21.5** (0.2%) · x_hat+r_hat_SAE **36.2** (0.8%) ·
  x_hat+r_hat_AE(m=128) **46.8** (2.9%) · x_hat+Gaussian **17.1** (0.0%).
  Paired-bootstrap deltas vs x_hat: r_hat_AE **+25.30** [24.10, 26.43], r_hat_SAE +14.67
  [13.75, 15.60], Gaussian **−4.42** [−5.18, −3.73]. An AE-m=128 reconstruction of r with FVU
  0.68 more than doubles decode fidelity; matched-norm Gaussian actively hurts. The residual is
  decode-relevant, and heavily so. (Example: x_hat "The problem I gain in testing that does not
  consider the!" → +r_hat_AE "The problem I realized in testing that people don't see the bill!"
  vs ref "…is that people never look at the bill!".)

## Gate outcomes
- **G1 regime/loader**: **PASS at full scale** (FVU 0.731 ∈ [0.35,0.75], cos 0.608 > 0.60,
  deterministic) — but smoke-scale FVU 0.772 breached the band and the full value is at the edge;
  recorded honestly, comparisons unaffected (all internal to this corpus).
- **G2 null validity**: PASS (top-64 eig max rel-err 0.0033 ≪ 5%; identical code path).
- **G3 positive controls**: ICA **PASS**, AE **PASS** (marginal: x-Δ 0.025), SAE **FAIL**
  (x-gap 0.079 < 0.10) → SAE residual cell downgraded to INSTRUMENT_FAILURE per prereg.
- **G4 decode headroom**: **PASS** (headroom 64.0 ≥ 5, chrF(x) 85.5 ≥ 50) → (d) interpretable;
  P4 scored.

## Brier vs frozen predictions
- **P1 (0.75)** SAE null−r FVU gap ≥ 0.15 → **FALSE** (0.018; instrument underpowered — weak
  evidence about the world, but the frozen criterion failed). Brier 0.5625.
- **P2 (0.80)** ICA top-16 ≥ 2× null band → **TRUE** (69×). Brier 0.04.
- **P3 (0.50)** AE m=128 Δ ≥ 0.03 AND ≥ 3× null Δ → **FALSE** (+0.0014). Brier 0.25.
- **P4 (0.55 | G4 PASS)** best decode ≥ x_hat+3 chrF AND beats Gaussian → **TRUE** (+25.3;
  Gaussian −4.4). Brier 0.2025.
- **Brier(P1–P4) = 0.264.** Main miss: P1 — over-confident that a small fresh SAE would separate
  r from its null by 0.15 FVU in a regime where it can barely separate x itself (should have
  power-checked the instrument at prereg time). P4 under-confident: the decode lift is ~8× the
  predicted 3-point threshold.

## Honest verdict
The residual of the embedding SAE is **structured, decode-relevant signal, not noise** — on this
corpus the SAE's reconstruction keeps only ~21% of variance and ~21 chrF of decodability, while a
crude 128-dim AE summary of what it discards restores another +25 chrF. However, the structure
looks **linear-plus-non-Gaussian** (huge ICA kurtosis, near-zero AE-over-PCA advantage): r behaves
like many heavy-tailed directions spread over a broad spectrum, not like a compact nonlinear
manifold. Whether it has *sparse dictionary* structure is unresolved (instrument failure), though
atom spot-checks hint at topical organization.

## Limitations
- One corpus (pile-10k), one SAE ckpt, and a lossier-than-canonical regime (FVU 0.73 at the G1
  band edge); the "residual" here is most of the vector (‖r‖/‖x_n‖ ≈ 0.79), so "residual is
  structured" partially restates "this SAE is very lossy on this corpus".
- Fresh-SAE probe underpowered (40 ep, h=8192, no aux-k); P1's falsification is about the probe.
- AE positive control passed only marginally (0.025), so P3's null result has modest power too.
- FastICA hit max-iters (500) without convergence on most null/x runs (converged on r, 36 iters);
  kurtosis readout is held-out so this biases against, not for, the huge observed ratio.
- Decode conditions share one greedy decoder; chrF vs original only (no semantic judge this row).
- Smoke-scale G1 breach recorded; single seed for most fits (2 null seeds for SAE, 3 for ICA band).

## Follow-up worth funding? **Y (narrow)**
(1) Row 043 residual-probing is already queued and directly extends this: WHAT information lives
in r (probes for length/topic/register/syntax). (2) A properly powered residual dictionary probe
(larger h, aux-k, more epochs, power-certified on x first) to settle P1's open question. (3) The
042 finding reframes the 041–050 block: at FVU ~0.7 the SAE dictionary explains a minority of
sentence content on generic text — worth one row testing whether campaign-corpus FVU 0.44 vs pile
0.73 is a domain-shift effect.