# 043 residual-probing — RESULT

**Verdict: ★ POSITIVE, unambiguous — the SAE residual is a rich, probe-readable carrier of
length, lexical, topic, and syntactic information; on every certified probe family it carries
MORE linearly-decodable content than the reconstruction x_hat itself. All 5 frozen predictions
TRUE (Brier 0.1055). Role cell = INSTRUMENT_FAILURE (positive control failed). Tier
T3-exploratory.** Ran + SELF-HARVESTED in-session (tmux c100_043, CPU only, full run 17 s).

## What ran
- SAE: w40 `cpool-sae_h16384_k32_btk_e40_seed0.pt`, 028 loader (bf16→fp32, per-sample
  top-k=32, x_n=(x−x_mean)/scale). Channels probed: x (=x_n), xh (recon), r (=x−xh),
  ru (=r/‖r‖), **xhr ([xh⊕r] concat — unique-info channel)**.
- Corpus A: 027-local-id, stratified 12,000 of 29,029 (domain×bin cells, seed 0), cached z
  (no re-encode). 70/30 split. Corpus B: agent_patient battery 12,000 (037 SONAR cache,
  12000/12000 hash hits), battery lexical-holdout 'split'.
- Probes: ridge (toklen/wordlen R²), multi-output ridge BoW word-presence AUC, 4-way domain
  logistic (+ per-length-bin), per-family role logistic, 5-way construction-family +
  active/passive voice logistic. Shuffled-label baseline per family; positive controls on x
  certified in the SAME data/split regime (062 lesson).
- Provenance: design adopted from a prior manager killed pre-run by a usage limit (no results
  existed); this manager added xhr channel, syntax probes, G1 band amendment — all pre-run.

## Gates
- G1 instrument: PASS — FVU(A) 0.655 (in amended band [0.35,0.80]), L0=32 everywhere,
  deterministic. FVU(B) 0.826 (templated battery sentences reconstruct notably worse; logged).
- G2 power (on x): domain 0.881 ≥ 0.50 PASS; toklen R² 0.976 ≥ 0.30 PASS; **role pooled AUC
  0.573 < 0.65 FAIL** → role cell INSTRUMENT_FAILURE (below).
- G3 shuffled: domain 0.257 (chance 0.256) PASS; role-shuffled 0.520 PASS (band edge).
  (Smoke's G3 domain miss was a small-n artifact; full n_test=3600 is clean.)

## Headline numbers (test-set, seed 0)
| probe | x | xh | r | ru | xhr | shuf |
|---|---|---|---|---|---|---|
| toklen R² | 0.976 | 0.723 | **0.907** | 0.884 | 0.969 | −0.09 |
| BoW mean AUC (66 words) | 0.981 | 0.878 | **0.937** | 0.938 | 0.971 | 0.493 |
| domain acc (chance .256) | 0.881 | 0.782 | **0.832** | 0.832 | 0.881 | 0.257 |
| family 5-way acc (chance .2) | 1.000 | 0.977 | **1.000** | 1.000 | 1.000 | 0.202 |
| voice AUC | 1.000 | 0.993 | **1.000** | 1.000 | 1.000 | 0.551 |

- **r > xh on EVERY probe family** — at FVU 0.655 the k=32 dictionary keeps a minority of the
  linearly-decodable content; the majority stays in the residual.
- **xhr ≈ x everywhere** (domain 0.881 vs 0.881; BoW 0.971 vs 0.981; toklen 0.969 vs 0.976):
  adding r back restores essentially all probe-accessible info — consistent with 042's decode
  chrF lift, now shown feature-by-feature.
- Length is in both norm AND direction: corr(‖r‖, toklen) = **+0.580** (P2), but ‖r‖ alone
  gives only R² 0.348 while the unit-normalized direction ru gives **0.884** (P3) — length is
  mostly directional, not a norm artifact.
- Confound check (031/028 lesson): domain differences in ‖r‖ (web 22.5 < fiction 26.1 < news
  28.1 < technical 30.3) occur at near-equal mean toklen (20.5–22.5) → topic-in-r is not a
  length proxy; per-length-bin domain acc on r stays high (long .897 / med .853 / short .758).
- Syntax (exploratory): construction family and voice are at/near ceiling in r (1.000) and
  slightly degraded in xh (0.977 / 0.993) — even coarse syntactic frame info partially leaks
  to the residual.

## Predictions (frozen in PREREG_LITE) — scored
- P1 (.85) domain on r ≥ shuffled+10pts: 0.832 vs 0.257 → **TRUE** (0.0225)
- P2 (.70) corr(‖r‖,toklen) ≥ .40: 0.580 → **TRUE** (0.09)
- P3 (.55) toklen R² on ru ≥ .20: 0.884 → **TRUE** (0.2025)
- P4 (.70) BoW AUC r > xh: 0.937 > 0.878 → **TRUE** (0.09)
- P5 (.65) domain xhr ≥ xh+5pts: 0.881 vs 0.782 (+9.9) → **TRUE** (0.1225)
- **Brier 0.1055** (all true; underconfident on P3 — direction signal was huge).

## Role cell: INSTRUMENT_FAILURE (honest null-of-instrument, not a claim about r)
Positive control on x pooled 0.573 (per-family 0.548–0.589) under the battery's lexical-holdout
split < 0.65 gate. The 0.758 reference was measured in a different split regime — exactly the
062 lesson (certify power in the SAME regime); under lexical holdout even raw z barely carries
focal-role here. r values (0.569 pooled) are reported but uninterpretable as evidence.

## Limitations / did NOT run
- BoW candidate pool was 66 words, not ~200 (the 1–30% document-frequency band on this corpus
  admits only 66 after stopword/length filters) — pool property, not truncation.
- Linear probes only (ridge/logistic); "information absent" claims are linear-decodability
  claims. Single seed, single SAE (h16384/k32); no CI/bootstrap on probe metrics.
- xh's deficit conflates dictionary expressivity with k=32 sparsity; no k-sweep here.
- Everything T3-exploratory; single corpus per label type.

## Follow-up worth funding? **Y**
The dictionary misses most probe-readable content at k=32 — a k/width-sweep of "probe recovery
vs FVU" (does the residual's probe content vanish as FVU→0, or plateau?) would say whether
SAE dictionaries systematically privilege some feature types (topic vs length vs lexical) —
directly relevant to interp claims that SAE features "explain" the representation.