Anatomy of a Text Autoencoder: Composition, Canonicity, and the Geometry of SONAR Space

§1 Introduction & setup

SONAR is a frozen text autoencoder: a sentence goes in, a single vector z ∈ ℝ¹⁰²⁴ comes out, and a decoder turns any such vector back into text. That makes it an unusually good proving ground for interpretability methods — every latent direction, every edit, every dictionary feature has a behavioral readout (decode it and read the sentence) that no LLM-internals study has. We applied roughly thirty interpretability techniques to this space: sparse autoencoders (TopK/BatchTopK/Matryoshka), probing, LEACE erasure, activation patching, DAS, intrinsic-dimension estimation, gradient attribution through the decoder, dose–response steering, and a battery of TAE-native methods. The aggregate picture: the data lives on a locally ~24-dimensional but highly curved manifold smeared across ~490 linear dimensions, the decoder is a strong prior rather than a transparent readout, and dictionaries are useful frames rather than canonical atoms — unless you make them canonical on purpose (§6).

This page focuses on five deep-dives: the algebra of composition ("A then B" as an operator on sentence vectors), the literal transcript carried inside a single vector (token identities, positions, order, and exact wording — and its holographic storage), the canonicity of learned dictionaries and the dials that control it, a low-rank but decoder-ignored language fingerprint, and a standing behavioral benchmark (TAE-Bench) that exposes the field's central tradeoff. All experiments ran on precomputed SONAR embeddings of web text (nickypro/llama-3b-embeds, plus a new 834k sentence-level corpus data_v2); every result JSON cited here is public on the HF repo.

Every experiment in the project is framed against five standing questions about the space — Q1 dimensionality, Q2 what directions encode, Q3 token/position vs meaning, Q4 edit & combine, Q5 construct from scratch — and each results section below opens by naming its question. Figure M2 is the roadmap.

§2 The composition operator

Question (Q4 — edit & combine): is there a predictable operator that turns two sentence vectors into the vector of their concatenation — and what is its structure?

Take two consecutive paragraphs A and B, and the SONAR embedding of their concatenation, z_AB. Is there a map [z_A; z_B] → z_AB? Concretely: we fit a ridge regression from the concatenated pair [z_A; z_B] ∈ ℝ²⁰⁴⁸ to the embedding of the concatenated text, and report cosine similarity on held-out pairs. There is, and it is strikingly clean: the full linear map predicts the composed vector at cos 0.889 (R² 0.77), versus 0.63 for simply averaging the two vectors. Decoding the predicted vector yields text nearly as close to the true concatenation as SONAR's own round-trip ceiling (0.941 vs 0.976 SBERT). Two controls pin down what the map is. It is not a learned continuation prior: the same experiment on random cross-paragraph pairs — texts with no continuation relationship at all — scores cos 0.875, essentially unchanged, so the operator implements concatenation, not plausible-next-paragraph. And it is not low-rank: rank-256 reaches only cos 0.52 (Figure 2) — composition uses essentially the whole space. One distribution caveat belongs here rather than buried in §8: pairs drawn from fully held-out shards compose at cos .758, below the same-distribution 0.889 — the operator is real, but its fitted form is distribution-sensitive.

What does the operator look like inside? Splitting the learned map into its two slot blocks W_A and W_B reveals two near-isometric, full-rank, rotation-like maps — flat singular spectra near 1.05 and 0.87, effective ranks 429 and 326 out of 1024, only ~2% diagonal mass, and modest overlap with the identity (cos 0.42/0.32). The inference is direct: if composition were a weighted average, each block would be a scaled identity — diagonal-dominant, with singular values flat at the mixing weight (~0.5) — and the mean baseline would have matched the full map. Instead the blocks are full-rank rotations with 2% diagonal mass: each slot's content is rotated into its own role frame before superposition. This is, empirically, the holographic/VSA binding scheme — role rotations, one per slot, superposed — discovered in a learned representation rather than designed into one. Slot A gets higher gain and dominates the decode (sim-to-A .776 vs sim-to-B .547), which corrupting slot A confirms causally. And riding on top of the rotations is one perfectly interpretable scalar channel: the length axis. PC0 of the composed vector is a literal size-adder, PC0_AB ≈ 0.546·PC0_A + 0.545·PC0_B (R² 0.967) — while the vector norm carries no length information at all (R² < 0).

The operator stacks (approximately associatively: depth-3 left fold .673 ≈ right fold .657 ≈ a directly-trained 3-input map .694), works across languages at ~96% of monolingual quality, and the same map run on swapped inputs produces z_BA at cos 0.881 — while an order-blind control (scoring those swapped-input predictions against z_AB instead) sits at 0.52, so the map genuinely tracks order rather than ignoring it. Order is handled equivariantly by a global rotation, not by an additive low-rank "binding ID" (the AB↔BA delta is 98% of the vector norm with effective dimension 465). Every discourse connective we tested ("However,", "because", "Meanwhile,") composes at cos ~.89 with the same high-rank signature.

§3 Unbinding & capacity

Question (Q4 — edit & combine, with a Q3 readout): can a composed vector be inverted back into its constituents, and how many sentences fit in 1024 dimensions before crosstalk wins?

If composition is invertible superposition, you should be able to unbind. Concretely: we train a linear map z_AB → (ẑ_A, ẑ_B) from the composed vector to both constituent vectors and score the cosine of each recovered slot to its true constituent on held-out compositions. It works: per-slot recovery cos 0.72/0.71 for pairs, decoding to recognizable text at SBERT 0.84/0.76. The number to beat is the similarity baseline — how close z_AB already sits to z_A with no unbinding at all, cos ~0.50 — so the map recovers genuinely slot-specific content, not just the shared gist. Pushing to chains of 3–16 bound sentences traces a clean capacity curve: recovery decays roughly linearly toward a ~0.4 floor by depth 6+, exactly the graceful crosstalk regime vector-symbolic architectures predict for superposed role-bound fillers in a fixed-width vector. One trend died under its own control: an early apparent recovery rise at extreme depth vanished under pool-matched candidate sets (it was a pool-size artifact), which is why every point in Figure 4 is pool-matched. And the honest reading of the floor: recovered deep-chain slots keep the topical frame but lose specifics — "recovery" at cos ~0.4 is gist, not transcript.

§4 Interpretable axes: the size-adder and the language fingerprint

Question (Q2 — what directions encode; Q3 — token/position vs meaning): which attributes live on nameable low-rank directions, which of those does the decoder actually read, and can an edit target one token at one position?

Two scalar/low-rank structures stand out against the otherwise high-rank geometry. The first is the size-adder of §2: sentence length lives on PC0 and adds linearly under composition. The second is a language-of-origin fingerprint: a 6-way linear classifier identifies the source language of a translated sentence's embedding at 84% (chance 17%), yet the structure is tiny — offsets ~17% of data RMS, effective dimension ~4.4, and LEACE removes it with an 8.8% edit.

The surprise is what the decoder does with it: nothing. The test is behavioral on both sides — erase the fingerprint from z and measure character-level agreement of the new decode with the old one; amplify the French component under an English decoder flag and count French stopwords in the decode. If the decoder read the fingerprint, the first edit would have to change the text and the second would pull it toward French (a genuine flip would land near the native-French reference rate of 0.20 stopwords). Neither happens: LEACE-erasing the fingerprint changes decoded text by a character-agreement delta of -0.004; amplifying the French component ×8 produces zero French stopwords (0%) — the decode degrades (SBERT 0.73) but stays resolutely English. Output language is purely flag-controlled; the fingerprint is a vestigial encoder-side code that the decoder never reads. It doesn't impede composition either (cross-language composition runs at ~96% of monolingual quality).

The same erase-and-decode probe works for other linearly-readable attributes: LEACE the attribute direction out of z, decode, and measure the attribute's rate in the decoded text — not just the probe — alongside content similarity. Erasing the first-person direction is an 8.5%-of-norm edit that drives the probe to chance and the decoded first-person rate down 3.5×, while the decodes stay almost verbatim — a clean, surgical concept deletion:

That said, first-person is the favorable case, and erasure ≠ removal in general. The same procedure kills the has_number probe stone dead, yet the decoded number-rate barely moves (0.28→0.25): the decoder regenerates the attribute from nonlinear traces the linear edit never touched. Iterating erase→decode→re-encode→re-erase helps only with diminishing returns (has_number 0.267→0.183 over three rounds, while content similarity decays to ~0.90). What LEACE certifies is the death of the linear probe, not the absence of the attribute.

And directions the decoder does read make clean steering handles. Adding a single SAE feature to an embedding moves the decoded topic — in whatever language the embedding came in, since the features themselves are language-invariant:

Lexical surgery by vector arithmetic

Word identity is itself an editable axis, and how you build the edit vector selects its scope. The measurement: add the difference vector to the sentence embedding, decode, and string-match which occurrences of the word changed. A global bare-word delta (z of the bare target word minus z of the bare source word) swaps every occurrence of the word: full-swap rate 88% on single-occurrence sentences, and on double-occurrence sentences it swaps both occurrences 89% of the time and exactly one 0% of the time. A positional carrier delta (the same subtraction computed inside a carrier sentence at one position) is position-targeted: on double-occurrence sentences it swaps exactly one occurrence 67% of the time (both: 33%). Both edits stay on-manifold (decode similarity to the hand-written expected sentence 0.94 and 0.93). The contrast in scope — both-swapped vs exactly-one-swapped — is what dissociates a position-free word-type code from a position-carrying one. Scope caveat: this is a demonstration, not a corpus result — 6 hand-picked word pairs in hand-constructed carrier sentences (11 single- and 3 double-occurrence frames per pair); §5 tests positional addressing at scale.

§5 The transcript in the vector

Question (Q3 — token/position vs meaning, answered; with the first Q5 baseline): how much of the literal token stream — identities, positions, order, exact wording — is in z, and can any of it be cut out of it?

A new instrument makes Q3 directly testable: capturing the encoder's pre-pooled token states (mean pooling reproduces z at cos > 0.999) lets us ask what each token contributes before the sum. The verdict comes in two stages. Stage one: z faithfully contains the transcript. Individual token states are lexical identity codes — a ridge probe reads the exact token ID from a single state at top-1 75% over a 2000-word vocabulary (shuffled-label control 6%, uniform chance 0.05%), graded by frequency (86%→42% across deciles). Position is linearly present in the states (R² 0.88 absolute, 0.89 relative, surviving same-token and length-stratified controls) — and it survives pooling: per-word “where is word w” probes on z itself reach relative-position R² 0.59 against a length-prior null of 0.0005 (~60×; 45/45 words beat 2× null). And z carries a strong exact-wording code: the teacher-forced decoder prefers the original wording over a length-matched paraphrase of it by 1.2–1.5 nats/token, decisive in both directions for 96% of pairs (a random other text sits at 3.9 nats). If z carried meaning alone, two same-meaning vectors would score the two texts equally; the margin says z knows which words it came from. One honesty note: the comparison is length-matched but paraphrase quality is uncontrolled — part of the margin could reflect residual surface overlap between paraphrase and original rather than a wording code per se.

Word order is in the vector too — and it is in z, not in the decoder's language prior. The design separates the two: encode a scrambled sentence, and the only place its scrambled order exists is z — if the decoder's prior supplied order, the decode would drift back toward grammatical (original) order. Instead the decode follows the scramble: τ-to-scramble 0.93 vs τ-to-original 0.01 (99.2% scramble-wins over 1,500 scrambles), even in the word-salad stratum (0.93). The prior is not entirely silent: it exerts a mild pull back toward grammar when the scramble is already near-grammatical (τ-to-original 0.23 there) — order fidelity is high, not absolute — but it never repairs the order.

Stage two: the transcript is stored holographically — every attempt to localize it fails. Zeroing or removing one token's state before pooling deletes exactly that word only 4%–6% of the time (content vs function words; 79% of decodes don't change at all); content tokens carry 1.6× the state norm of function words, but norm explains decode damage weakly (r = 0.17). A learned “wording subspace” (paraphrase-orbit residuals, length-matched) can be removed at any rank from 8 to 64 without moving the CE wording margin at all — margin-collapse fraction 0.000 at every rank, indistinguishable from removing the meaning subspace or random dimensions (raw margin 1.54 nats). This null rests on 16 held-out paraphrase orbits — consistent across every rank and both removal controls, but low-powered; a subtle low-rank wording component below this test's sensitivity is not excluded. And transplanting B's wording residual onto A imports essentially none of B's words (2.6% vs 2.1% control). Transplanting one token state pre-pooling surfaces the donor word at the matcher's false-positive floor (0.5% single-content vs 0.2% noise floor) while the rest of the sentence survives untouched (SBERT 0.98); the rare successes are position-specific (wrong-position control), so positional addressing exists — the holographic veto just dominates. And the best position-conditioned readout of token states from z works weakly and mainly at the edges: reconstruction cos 0.38 beats shuffled-z 0.26, but reconstructed token-ID is 22.6% vs the 86.9% real-state ceiling (interior deciles ≈ floor; first/last 58%/49%), and re-pooled reconstructions decode at SBERT 0.59 (ceiling 0.98).

The same lesson arrives from the construction direction — the first Q5 baseline. A bag of word vectors fails raw: summed word embeddings decode to degenerate junk (77%), the mean decodes to a single word — but geometric normalization rescues it from zero: mean + corpus-shift + RMS reaches content recall 0.31 / SBERT 0.44 (round-trip ceiling 0.97/0.99). Folding the words through §2's composition operator instead transmits order nearly perfectly (τ 0.92 where scored, only 3% degenerate) while content mass attenuates with depth (recall 0.54 at 4 words → 0.08 at 12, matching the §3 unbinding floor). What a bag is missing is structure — an order/length scaffold — not representable words.

Answer to Q3: z is a positionally-addressed, lossy transcript with a meaning halo, pooled holographically. Token identities, positions, order, and exact wording are all faithfully readable out of the whole vector — and none of them can be cut out of a part of it: no low-rank wording subspace, no per-token locality, only edge-biased weak addressability. Reading the transcript is easy for the decoder and hard for every surgical tool — the same read ≫ write asymmetry the battery found for attributes, now established for the token stream itself.

§6 Dictionaries: the canonicity problem and its dials

Question (Q2 — what directions encode): are dictionary features discovered atoms of the space or an arbitrary frame over it — and can canonicity be bought?

Sparse autoencoders are the best causal handles we found in this space — features transplant content where PCA bands fail, survive encode∘decode round-trips 6.5× better than random directions, and act identically across 10 languages and 4 scripts. But they have a foundational problem. The protocol: train two SAEs identical except for random seed (held-out FVUs match to the third decimal), pair their features by symmetric best-match with both-live filtering, and score the median matched cosine. If features were discovered atoms, a second seed would rediscover them and matches would approach 1. Instead: two seeds reach identical FVU with essentially disjoint dictionaries — median best-match cos 0.149, barely above the 0.122 that a random rotation of the same dictionary scores. Matching at rotation-level means the seeds share a subspace, not features: the dictionary is a frame tiling the manifold, not a set of discovered atoms.

The features a given seed finds are nonetheless individually meaningful — an automated labeling pass over a 192-feature atlas produces short English labels that a blind judge can verify against steered decodes:

That lament turns out to be engineering, not fate. An L2 penalty on the encoder makes seed-alignment a purchasable dial: the fraction of features whose pairwise-matched correlation exceeds 0.5 rises monotonically with the penalty, saturating above .95 at every sparsity level by l2 ≈ 2e-3 — and the reconstruction tax shrinks with k. The sweet spot, k256 at l2 1e-3, gets 96.7% of features seed-aligned at FVU 0.44 — but note the tax is real: the unregularized dictionary sits at FVU .21, so canonicity roughly doubles the reconstruction error. The dial buys identifiability with fidelity; where on that line to stand is a modeling choice, not a free lunch. Two milder dials reproduce in the same direction: sentence-level training data (+64% relative PW-MCC, free of recon cost; row-match 0.19 vs 0.15) and sparse-shift SSAE training on minted paraphrase pairs (~2.2× seed-consistency, PW-MCC median 0.21 vs 0.095, with no regularization at all). Longer training does nothing. Canonical dictionaries are now a recipe, not a hope.

A second dictionary result reframes what they should be trained for. Fine-tuning an SAE through the frozen decoder (text-CE loss) makes FVU worse (.30→.48) while improving every behavioral metric — decoder CE −25%, decode-SBERT up, attribute preservation .82→.92, beating a k512 dictionary with 4× lower FVU. Variance explained and text information are decoupled; for a TAE, dictionary quality should be measured in decode/text space (§7 builds the benchmark). Two caveats keep this honest. First, the gain is behavioral, not geometric: the text-loss dictionary's coverage of the decoder's Jacobian sensitivity subspace is unchanged (.189 vs .185 for its base) — the features behave better without relocating into the decoder-sensitive subspace, so text-loss is not (yet) a fix for the dictionary/decoder misalignment. Second, canonicity under text-loss has only been measured as fine-tune micro-adjustment from a shared warm start (the ~21% of features that move at all stay at cos .988 to their warm-start rows); whether from-scratch text-loss training yields canonical dictionaries is untested. The hierarchical (Matryoshka) variant adds a compositional bonus: its features survive concatenation best and are the most additive (Figure 10).

§7 TAE-Bench and the reconstruction ⊥ causal tradeoff

Question (Q1 — dimensionality; Q5 — construct from scratch): what is the effective geometry behind the benchmark axes, and can we build or improve vectors through the frozen decoder instead of encoding text?

To stop relying on FVU we built TAE-Bench v1: a fixed-seed, fixed-shard behavioral scorecard with four axes — reconstruction (decode SBERT), causal usability (steering potency × decode coherence), manifold round-trip survival, and sparsity hygiene — plus an HF leaderboard that is re-run for every new checkpoint. Every axis is measured in text, on the same fixed seeds and shard: reconstruction decodes the dictionary's reconstruction of z and scores SBERT similarity to the original text; causal usability steers each feature and scores how far the decode moves times how coherent it stays; survival asks how much of a feature direction outlives an encode∘decode round trip. Because rows differ only in dictionary, the trends below are properties of the dictionaries, not the data. Its first law (Figure 11): reconstruction is monotone in k while causal usability runs in the opposite direction — k512 decodes at SBERT 0.90 but scores 2.65 on potency×coherence, while k8 decodes at 0.46 but steers at 4.56. The understand-vs-reconstruct tradeoff is the leaderboard's principal axis. k64 is the all-round optimum (best manifold survival 0.15, pxc 4.07, reconstruction ≈ k128), width hurts both causal axes at matched k, and text-loss dictionaries dominate their base on both behavioral axes (SBERT 0.77 vs 0.74, pxc 3.32 vs 3.27) despite much worse FVU — the third independent instance of behavior and variance-geometry coming apart.

The same geometry/behavior split shows up in the composer. We fine-tune §2's composition map through the frozen decoder, scoring teacher-forced decoder CE of the true concatenation under the predicted vector and SBERT of its decode, with an MSE anchor to the ridge solution. A weak anchor trades cosine for text quality; with a strong anchor (mse ≥ 1) it is a pure win — at mse=1: cos 0.734→0.744, decoder-CE 1.39→0.68, decode-SBERT 0.79→0.89. The anchor coefficient tunes where on the geometry↔behavior line you land.

TAE-Bench's causal axis is grounded in a dose–response census (Figure 13): steering any feature hard enough eventually drives the decoder into a feature-specific single-letter attractor ("f", "n", "e", …). Across all nine dictionaries censused, roughly two-thirds of features are semantic-then-collapse; only a minority (11–30%) steer smoothly at every dose, and immediate cliff collapses get rarer as k grows (14% at k64 → 2% at k512). Census caveat: each dictionary is sampled at 64 features, so the class fractions carry several points of sampling error — the robust finding is that essentially every feature has a collapse dose, not the exact split between classes.

One more piece of low-parameter geometry from the same battery. The setup: texts are generated by a hidden 3-state topic process, and we test whether the exact Bayes posterior over hidden states can be read linearly out of z of the text so far. It can — but an audit-mandated surface baseline disciplines the claim: a bag-of-topic-counts model already explains R² .98 of the posterior, so the honest finding is the order-information residual — z predicts the component of the posterior beyond bag-of-topics at R² .90. The new result is that this structure is literally a simplex: a 3-vertex barycentric model (6 parameters) inverts held-out posteriors at L1 0.30 — better than a full 1024-d linear probe (0.52), both crushing a shuffled-vertex control (1.41), which rules out the barycentric form fitting by accident. The probe retains the calibration edge (KL 0.17 vs 0.61) — the simplex wins on parameter count and L1, not on every metric.

§8 Methods & rigor

Single-author, largely automated experimentation makes self-deception the main failure mode. Three methodological layers guard every number on this page:

• Independent code review with pre-registration. The load-bearing analysis scripts were audited by an independent reviewer both before and after the experiments ran, surfacing 3 FATAL + 14 MAJOR issues in total. One pre-registered FATAL (a leakage bug in the composition evaluation) was fixed before any GPU hour was spent; the audit reversed one earlier claim (the multiclass-DAS "content slot" collapsed to 0/50 with held-out object values — it had memorized per-object directions) and strengthened two others (the belief-state claim survived in honest order-residual form; seed disjointness was fully robust to symmetric matching and rotation baselines).
• Replication batteries on fresh shards. Every §2–§4 headline was independently re-run on held-out data (anatomy eff-rank 419 vs 429, size-adder R² .966, composition .8898, feature survival .1745, language-ID 85.6%) — every one reproduced.
• Pool-matched controls and a fixed-seed benchmark. All unbinding evaluations use pool-matched candidate sets, with determinism reruns; one such control killed an exciting trend (the apparent recovery rise at extreme unbinding depth was a pool-size artifact). TAE-Bench runs every checkpoint on fixed seeds and fixed shards, so leaderboard rows are comparable. Canonicity claims carry baseline and (collapsed) archetypal controls; an early archetypal "success" was correctly rejected as a collapse artifact (FVU 1.95).

Standing uncertainties. Results we report with reduced confidence, collected in one place:

• The wild-corpus rerun of the rank-1 DAS tense coordinate produced 0.0% flips on natural text — but that rerun's templated control also underperformed the original protocol, so the zero may be partly implementation. Either way the "global tense dial" reading is dead: at best the coordinate is template-local.
• The curvature-radius retrodiction (interpolation failure at 0.72× the local-frame decay scale) is directionally consistent but a weak fit (R²≈.33) — a coherence check, not an established law.
• The wording-subspace null (§5) rests on 16 held-out paraphrase orbits — consistent across ranks and controls, but low-powered.
• The wording-CE margin (§5) is length-matched but not paraphrase-quality-controlled; part of it could be residual surface overlap.
• The composition operator's fit is distribution-sensitive (§2): fully held-out shards compose at cos .758 vs .889 in-distribution.

Net: one audit pass found one false claim and strengthened two. Independent review, pre-registered controls, and replication on fresh shards are cheap relative to the cost of believing your own bugs — "the easiest person to trick is yourself" is an executable discipline.

§9 Artifacts & data release

Everything cited on this page is public under nickypro/sonar-sae:

• Dictionaries: BatchTopK SAEs (h16384–65536, k8–512), Matryoshka SAEs, text-loss fine-tunes (v1, v2, seed pair), the L2-canonicity grid (28 paired-seed runs, tae_canon_l2*_k*_e24.json), SSAE shift dictionaries, and decoder-layer transcoders.
• Benchmarks: taebench_leaderboard.json (27 rows, re-run for every new checkpoint) with per-checkpoint scorecards.
• Composition suite: tae_composition*.json, tae_comp_anatomy*.json, tae_comp_unbind*.json (depths 2–16, pool-matched), tae_comp_features*.json, joiner and language variants — each with a _rep replication where applicable.
• Data: data_v2/ — 834k sentence-level SONAR embeddings + texts, with fr/de/zh parallel embeddings for the first six chunks.
• Code & logs: training and analysis scripts in the project repo (parascopes/layerwise/src/), full experiment narrative in INTERP_RESULTS.md, audit trail in CODEX_AUDIT.md, and a 38-experiment designed backlog in EXPERIMENT_BACKLOG.md.

This page was generated by build.py in the same directory; every figure reads the artifact JSONs directly, so the page rebuilds from public data.

§10 A certificate of (partial) understanding

Question (all five): after ~30 techniques — do we understand this codec, and could we prove it?

Everything above reads like progress, and it is — but “we understand SONAR better now” is a vibe, not a measurement. TAE-Bench v2 replaces the vibe with a certificate: understanding is operationalized as seven conjunctive criteria — predict, decompose, edit, construct, mechanism, transfer, calibrate — each scored by protocols with an explicit baseline, ceiling, and pass threshold, evaluated on held-out splits (edit types, perturbation families, and spec shards frozen before any method was tuned). The headline is the minimum over criteria, never the mean: you do not understand a codec you cannot edit, and a mean lets the easy reading half (probes ≈ .99) launder the failing writing half. Anti-gaming clauses do the rest of the work: parameter budgets, so a learned helper cannot simply be a distilled second decoder; no re-encoding the target text, because that is the rewrite pipeline, which “passes” with zero understanding; and an impossibility clause — demonstrating that the object lacks an asked-for structure also passes, so a holographic code is not held hostage to exams about structure it provably does not have. Each clause guards against a pseudo-understanding failure this project met first-hand: the FVU lie (§6), the read≫write gap (§5), the DAS template artifact (§8), seed-disjoint dictionaries (§6). A certificate that can fail is the only kind whose passes mean anything.

The first run adjudicates as follows — the honest language is part of the protocol, and one aspect's verdict is about the test rather than the understanding:

The E1 split — the certificate's one pass, and the failure living inside the same criterion — deserves to be seen at text level:

Verdict: we do not yet fully understand SONAR — and now that statement has numbers. The certified positives are real: edit-controllability, the transcript characterization (§5), the composition algebra (§2), the canonicity dials (§6). The numbered gaps are the research agenda. (1) Spec-only editing at ~.10 is a transport problem: gradient surgery proves every edit is reachable in z with zero collateral, so what is missing is the rule that carries an edit specification to its Δz without descending on the answer's own tokens. (2) The 0.392-nats/token reassembly gap is nonlinear dark-matter accounting: adding the true SAE error recovers the decode ceiling exactly, and the linear predictor of that error closes only half the gap — the missing nats live in the unpredicted, nonlinear part of the residual. (3) Mechanism prediction: ablation-outcome forecasting must rise from 0.47 toward its .6 bar, and the transcoder-swap protocol needs a layer the decoder actually depends on (or per-feature swaps) before MECHANISM can be scored at all. (4) Construction content-mass: order is solved inside the constructor, but content recall holds it at 0.51× the ceiling. The certificate stays standing release over release; the next one either moves these numbers or explains, with an impossibility argument, why they cannot move.

§11 Day 5: the calculus written down, the dark matter read, and a safety loop closed

Question (Q2, Q4 — what directions encode; edit & combine): the night-4 model named a composition operator and a dark-matter content field but did not write either down. Day 5 threw a 17-box fleet at making them measured receipts — and at the one deliverable that matters outside this codec: an end-to-end safety monitor.

The earlier sections leave SONAR with a clean skeleton (composition algebra §2, the transcript §5, canonicity dials §6) and a frank certificate of what is not yet understood (§10): a spec-only edit calculus stuck near .10, and a reassembly gap whose missing nats live in an unpredicted, high-rank residual. Day 5 is the assault on those gaps. Every headline below is paired with the null that catches its trivial explanation, every win is reported next to its scope, and the honest negatives — a refuted feasibility certificate, an adversarially-foolable auditor, a SONAR-specific binding geometry — are as load-bearing as the positives.

§11.1 The composition operator, in closed form — and 20× sparse

The composition map zAB = WAzA + WBzB + b (held-out cos 0.889) decomposes, per slot, into a scaled near-isometric reflective role rotation: a polar split W = Q·S gives an orthogonal factor with det Q = −1 (a reflection in both slots), a moderate stretch (‖S−I‖_rel 0.50), and a real median eigen-angle (0.141 / 0.150 rad — a genuine rotation, where the position clock is ~.007). The two blocks nearly commute (rel-commutator 0.028) but are not a shared base raised to two powers (QB=QAk refuted, residual 0.984) — so the HRR shared-base hypothesis is rejected and there are two genuinely distinct role rotations, exactly as the night-2 work suspected. The best low-parameter closed form — two orthogonal matrices plus two scalars (c·Q per slot, c_A=0.60, c_B=0.58) — recovers cos 0.813 vs the .889 full map, i.e. 91% of the composition is two pure orthogonal role factors plus a length scalar. On the size axis the map is a clean adder: PC0(zAB) ≈ 0.55·PC0(zA) + 0.54·PC0(zB) at R² 0.967. (One §16 correction this absorbs: the night-4 “≥32 eigen-angles at π” belonged to the order-swap involution S, a different, reflection-heavy operator — these slot maps carry only 11–13 near-π axes.)

Asked whether the operator can be made sparse, the answer is a sharp yes — in exactly one basis. Magnitude-pruning in the raw or PC bases sparsifies poorly, but in the operator's own Schur eigenbasis the map is near-permutation (row-participation 1.24 mean): keeping just 5% of entries recovers cos 0.846 — 95% of the full map at 20× compression. The same lesson as the dictionary work: structure is sparse in the right basis, not the raw weights.

§11.2 The reconstruction ladder: content is the dark matter

Rebuilding z term by term — bag → +R_pos → +R_role → +c_i → ceiling — and scoring each term not in vector-cosine but in the decoder's teacher-forced cross-entropy (the behavioral currency, nats/token) gives the definitive decomposition the model promised. Of the 2.616-nat gap between a bag-of-embeddings and the true latent, the position clock closes 6.4%, the role rotation closes -0.1% (a rounding-level regression — refit on disjoint shards R_role comes out near-identity, so its value is binding/order, not pooled recall), and the context field c_i closes 93.7%, collapsing CE onto the ceiling. The clean, constructible algebra is ~6% of what separates a bag from the latent; the rest is the high-rank dark matter — and §11.4 shows it is also exactly what blocks construction.

The cleanest receipt for “meaning lives in the dark matter” is to decode it. Strip the role-skeleton from a sentence's z, form the content residual c_i (69.4% of the energy), and decode each piece alone: the content residual recovers the gist at SBERT 0.851 while the skeleton — all the right words and order, no context — decodes to word-salad (0.514), beating a norm-matched random control (0.400) by +0.451 (paired t= 27.5) — so it is the content direction, not the extra norm, that carries meaning.

§11.3 The safety loop closes — and where the auditor cannot be trusted

This is the deliverable the whole program exists to produce. Wire the two halves into one pipeline: read an LLM's hidden state (Llama-3B, layer 27) → a maghead parascope predicts the next sentence's SONAR z (cos 0.438 to the true latent) → run the calibrated z-auditor on that predicted z. Can we flag the not-yet-written sentence's attributes before it is emitted? Yes: all 6/6 audited attributes recover above chance, mean chain-AUC 0.73 (sentiment 0.82, with 86.5% pre-emission agreement with the true-z verdict) — off a parascope whose z prediction is only cos 0.438. The chain is leakage-clean: the predicted z comes only from the LLM residual via a probe that never sees the next sentence. This is the first end-to-end demonstration that the interpretability read half and the safety audit half compose into a forward-looking monitor.

But the auditor is a monitor, not a guarantee. Red-teaming it with a z-space PGD attack that flips the verdict to ABSENT while the marker word survives in the decode (SBERT≥0.9) shows the single linear probe is fooled on essentially everything (attack-success 1.00 on sentiment). A K=11 random-projection ensemble rescues the active attributes (mean defended success 0.10; sentiment 0.00, fully defended) but not the curved ones (0.51 mean; number stays 0.909, completely undefended). The geometry explains it — concentrated active directions are caught by some projection; diffuse curved signals (number, date) the decode prior re-expresses are hidden from all of them at once. So: ship the ensemble, scope its guarantee to active attributes, and never certify the absence of a curved attribute — an adversary can hide a number while the sentence still states it.

§11.4 Editing becomes deployable — within the lexical-substitution regime

The §10 certificate's central negative was a stuck edit calculus. Day 5 overturns it, then fences it. A cross-attention recontextualization-delta head — which reads the cached original encoder states and predicts only the in-context delta, with no new encoder pass — lands edits at exact-edit@collateral 0.36 (word .30 / entity .44 / number .34), beating the neighbor-MLP plateau (0.233) and even partial re-encode (0.325) — at 1.1% of the partial-reencode FLOPs. Both win-lines met: lands ≥.30 and ≪10% of re-encode cost. The night-4 “irreducible” frame flips to “recoverable cheaply by a learned delta-head from cached states.”

The honest fence (this is the rigor, not a footnote). A robustness gate on six edit types the head never trained on shows it is a memorized lexical-substitution head, not a general edit calculus: held-out-type exact@coll collapses to 0.079 (vs the in-dist .39), with only 1 of 6 unseen types holding ≥.3 — person (0.425), which generalizes precisely because it is also a content-word substitution. Tellingly, for insert-clause / delete-clause / register even the oracle falls to ~0 — those edits are outside the single-slot mean-pool edit model entirely (you cannot insert a clause by perturbing one token-slot's mass), so the failure is the edit parameterization, not the head. The head does compose across k sequential edits (3.9× cheaper at k=4, no re-cache) — but composition is free on cost, not a quality win. Net: deployable, cheap, and even cheaper under multi-edit sessions — for the substitution class it was trained on, full stop.

§11.5 Scope: content is language-universal, the binding geometry is SONAR's own

How far do these claims generalize? Two ways, opposite answers. Content is language-universal: every probed SAE feature steers coherently in all 11 languages tested (6 scripts), with cross-lingual invariance .68–.77 — German, the night-4 caveat, is in fact one of the best transfers (.76); the residual is largest only on typologically-distant scripts (Japanese/Korean/Turkish), never breaking. The composition operator itself ports 79–95% cross-lingually and the role binder is spectrally identical per language. The binding geometry, by contrast, is SONAR-specific. Aligning z to a second pooled encoder with one fitted linear map and asking whether the calculus transfers, a trivial mean-composer (½(A+B)) beats the conjugated role-rotation in both encoders (mpnet: 0.910 vs 0.888; LaBSE: 0.916 vs 0.856) — most of the apparent transfer is shared content (the A-only floor sits at 0.63), and the hub field barely tracks the second encoder's own spikiness. A third encoder (GTR-T5) corroborates from a fresh angle: its W_B median eigen-angle is 0.09 rad vs SONAR's 0.97 — no SONAR-style role-rotation at all, while mean-pool exactness recurs trivially. The additive content field is a generic pooled-encoder property; the reflective role-rotation binding and the dark-matter hub field are SONAR's own geometry. Scope the claims accordingly.

§11.6 What breaks the plateau — training budget, not width

A standing fact across the whole program was a reconstruction plateau at ~0.32 FVU: every larger dictionary at a fixed sparsity sat on the same wall. The flagship dictionary — h131072 (8× the base width) at k=128 — clears it, landing held-out FVU 0.248 (cos 0.875) with a behavioral roundtrip of 0.899 SBERT = 91.3% of the decoder ceiling. An earlier draft credited that win to the 8× width. A pre-registered, matched-budget replication on the flagship's own corpus (v2-sentence, 800k train / disjoint chunk-16 held-out, 34,105 val) refutes the width attribution. At a matched recipe (k=128 BatchTopK, aux_k=512, 40 epochs, batch 2048), the 8×-smaller h16384 reaches FVU 0.225 — well below the flagship's .248 — while h131072 at the same recipe regresses to 0.265 (both hold under a per-sample evaluator too: .237 vs .270). Width hurts held-out reconstruction here: h131072 posts the lowest train MSE (0.121) yet the worst held-out FVU at only 0.03% dead — it is overfitting, not generalizing. The 8×-smaller dictionary also beats the published flagship at the flagship's own batch size (h16384 .240 vs .248 at batch 8192), so 8× width was never required to clear the wall.

The real lever is training budget (data-size × epochs), not dictionary size: widening the training corpus 150k→800k sentences drops h16384's FVU from 0.377 to 0.260 at fixed width, while corpus identity is nearly inert. The old width “control” (the day-5 k64 model at 0.320) was confounded — it moved sparsity and width at once (h131072/k128 vs h32768/k64). Honest scope: this is a reconstruction result only — a wider dictionary could still win on monosemanticity or steering, untested here — and the batch-8192 margin (.240 vs .248) is narrow, with h131072 un-runnable at that batch on 16 GB and no per-width tuning. So the licensed claim is that width was not required to break the plateau, and buys no reconstruction under this recipe — not that width never helps. Nothing here makes the dictionary more canonical.

§18 Night 5: where the meaning lives — c_i deflated, and located

Question (Q1, Q2 — what is c_i, and is the night-4 model a real generator?): the night-4 model named a context field c_i and read its gist off as a near-disjoint dark-matter store. Night 5 took that account apart at the pre-pooled, per-token, causal level — and the honest headline is two-sided.

This night did two things at once, and over-stating either is the live self-trick. It deflated three specific over-readings of c_i — that it is an additively context-accumulated term, that read-out is nonlocal/holographic, and that it is a separable blob that uniquely decodes to the gist disjoint from lexis. And, in the same battery, it converted “c_i matters” into a located, causally-grounded, identifiability-validated positive account: the decodable meaning of z lives in an order-free, intrinsic-heavy c_i field, written by late FFN (L19–23), localizable to a mid-scale ~64–256-neuron handle, globally (not locally) constructed, and not recoverable from surface words. Every positive claim below is paired with its opposing-direction counterweight, and the gating result — A2 identifiability — certifies that the load-bearing claims ride on gauge-rigid objects rather than a pooled-fit artifact, so the chapter is closed, not provisional.

§18.1 Where the meaning lives — the c_i field, not the clock

The sharp test (B1b): how much of z's decodable meaning survives an order-free bag of its own lexical codes E(w_i), no c_i, decoded through the real SONAR decoder? Almost none — SBERT 0.299 / token-id Jaccard 0.189, against a full-z ceiling of 0.985 / 0.916 and a random-word floor 0.045. A bag of exactly the right words decodes to fluent-but-wrong text (a bag about repealing the ACA decodes to “The unity prevented the use of alcohol…”). Of the 0.687 bag→ceiling SBERT gap, order/role adds 0.164 and c_i/composition carries 0.522. The 4-rung ladder (B1c) then isolates which ingredient restores the meaning by toggling the clock (R_pos binding) and c_i independently: an order-free c_i field alone recovers SBERT 0.901 / 69% of tokens, while the position clock alone reaches only 0.463. The clock adds +0.164 over the bag; c_i adds +0.602. The decodable meaning is the c_i field; holographic position-binding is the small term.

§18.2 What writes it — late FFN, a mid-scale neuron handle

The first causal gears test (B4; 1,500 sentences, 49k tokens; per-layer zero+mean ablation of attention-out and FFN-out). Mean-ablating late FFN collapses ‖c_i‖ while lexical content survives — a c_i-specific break, not a generic state break: content-token c-collapse climbs +0.073 (L19) → +0.100 (L21) → +0.135 (L22), hub-concentrated to +0.31 at L23, with lexical cosine moving only −0.004. By contrast attention ablation is flat across all 24 layers (late-attn mean collapse 0.003) and no single head moves c_i (max ~0.006). The pre-registered “late attention writes c_i” is falsified in the informative direction: the writer is late FFN, turning §17.4's correlational attention story into a causal FFN one.

B4b sharpens the writer one level deeper — which FFN neurons, by magnitude attribution over the 40,960 late-FFN units of L19–23. c_i has a mid-scale neuron handle: the ablation curve climbs +5.0% (K8) → +10.8% (K64) → +13.2% (K256) content c-collapse and saturates by ~256 neurons — that 13.2% matching B4's single-layer L22 ceiling. The effect is 18× random (K64 +10.8% vs +0.56%) and z survives (cos 0.96 at K256). So c_i-writing is sparse-ish: ~64–256 neurons (0.16–0.6% of the late-FFN units), L20–22-concentrated. “Single neuron” and “fully distributed” are both rejected. The pre-registered per-neuron hub-detector framing is itself falsified — only 27.5% of top writers are hub-selective and the median writer is mildly anti-hub — so hub-concentration is a field-level property, not per-neuron tuning: the writers' collective output lands disproportionately on hub tokens.

§18.3 What it is NOT — not an LLM field, not locally constructible; and the dictionary view

Two negatives survive at per-token granularity. P6: a per-token frozen Llama-3B predicts c_i at R² 0.077 (≈ the paragraph-pooled 0.081), still below the 0.11 lexical bar and far below the 0.45 same-object threshold — going per-token bought nothing, so c_i is SONAR-internal, not an LLM-legible predictive field. A4: a closed-form encoder-write ladder reproduces §17.4's anchor (VE 0.177) but adding pairwise, cross-attention, and full-transformer terms plateaus at VE 0.163 and never beats the anchor — c_i is irreducible to local (windowed/pairwise) structure and is a global, all-to-all object, exactly what a late-FFN writer over the full bidirectional context produces. (State it as “not locally constructible,” not “uncomputable”: a global model with full context does capture it.)

The first non-pooled SAE (h32768 / k64 BatchTopK) trains to FVU 0.081 — 4.5× more compressible than the pooled-z SAE (0.366), and a bigger h65536 dictionary does not improve it (0.086 plateau). It partitions cleanly into a lexical family (32,734 features, 99.9% / 96% of energy, token-identity-selective beyond frequency) and a small genuine-interior positional family (34 features, 0.1% / 4.1% of energy, all 34 interior, not edge artifacts — the R_pos clock). Diagnostically there are 0 separable context features: token-purity is flat across rarity terciles, so c_i does not form its own dictionary family — it is finely entangled into the lexical family, consistent with it being token-identity-bound and per-token-spiky. This confirms the generative model's families and c_i's non-separability at once.

§18.4 Refinements — capacity, locality, order

Three quantitative refinements. Capacity: one z holds ~25–35 content words at content-recall ≥0.85, with a soft 0.70 knee at ~42 content words and graceful decay — no cliff, edge>interior (first-token recall 0.90, last 0.85, interior 0.74). This corrects the earlier Frady “length-cliff” forecast (measured 0.90 vs forecast ~0.45 at ~60 tokens): SONAR binding is far more capacious than the interference-channel model predicted, and it bounds how much of B1b/B1c's restored meaning is real per-token information rather than decoder saturation (it is real, well within these lengths). Locality is graded by lexicalization: lexical sentiment is most peaked (Gini 0.55), topic intermediate (0.41), implied sentiment least (0.37) but still span-local — none met the distributed falsifier. Order is partially bilinear (A5-fix): a sanity-gated low-rank bilinear comparator reaches Kendall τ 0.36, between linear 0.26 and MLP 0.57 — naming ~32% of the surface-order comparator while genuine deeper-than-bilinear depth remains.

§18.5 Identifiability — what is gauge-rigid vs frame-relative (A2)

A2 is the gating result that lets the chapter stand. The factorization [R_pos, R_role, E, c_i] is partially identifiable — a middle outcome in which neither pre-registered falsifier fired (not the clean “all four objects <10% drift,” not the worse “R_pos drifts wildly → nothing pinned”). The arbiter was deliberately behavioral, not matrix-cosine (which lies here: R_role's cross-seed matrix-cos 0.96 looks stable but is split-specific, while the behavioral compose-agreement 0.9996 is the honest measure). What is gauge-rigid (stop hedging): R_pos — median eigen-angle ~0.011 rad at any init, anchored to the architectural L0 positional encoding; c_i — relative eff-rank drift 2.0% across seeds and exactly orthogonal-gauge invariant; R_role — compose-agreement ≈1.00 across disjoint seeds. What stays frame-relative (keep hedging, these are SONAR-frame readings): R_role's geometry (the “0.87 rad, reflective, det Q = −1” description sign-flips across λ and ~doubles under a recon-preserving non-orthogonal gauge) and c_i's absolute eff-rank magnitude (the “~25” constant is basis-covariant). The decisive consequence: the shared-pooled-fit-artifact worry that hung over the convergent deflations is allayed — the behavioral c_i claims (essential, late-FFN-written, carries-meaning, dilutive) ride on seed- and gauge-stable objects, not a manufactured fit label.

§18.6 Safety — meaning hides in c_i, not the words (SAFE-1)

SAFE-1 is the safety capstone and the mechanism behind §11/§17's auditor-foolability. On a bag-invariant implied-attribute task (implied sentiment via contrast sentences, n=320, balanced) audited three ways, the order-free lexical auditor scores AUC 0.491 = chance while full-z and c_i both score 1.000 — a hidden-meaning gap of 0.509 — and the overt-attribute control shows gap 0.0 (lexis suffices). A worked example: “The mattress was a genuine pleasure, but the rival brand was a constant headache” — the lexical auditor scores benign, the c_i auditor flags it. Audit-relevant meaning lives in the compositional c_i field, not the surface words — precisely why a lexical/surface auditor can be fooled. The load-bearing caveat: a tf-idf bigram model partially cracks the task (AUC 0.991, via clause-order bigrams), so the robust claim is not “no lexical model can audit this” but “order-free lexical features are insufficient; audit on z or the c_i field, which carry the compositional, order-bearing meaning.” For SONAR latents, surface bag-of-words auditing is structurally insufficient for implied meaning, and the audit-relevant signal sits in a field that is essential and located (B1b/B1c), causally late-FFN-written and mid-scale-localizable (B4/B4b — a concrete handle for monitoring or intervention), and gauge-rigid in these behavioral properties (A2). A monitor that reads z or c_i, not a lexical bag, is the supported design.

§19 Reading c_i: what the meaning field actually says

Question (Q3 — can we READ c_i, not just locate it?): §18 found where the decodable meaning lives (the order-free c_i field), what writes it (late FFN), and certified its behavioral claims gauge-rigid — but it never read the field. §19 is the reading chapter: a four-wave battery (a skeptic loop that corrected four of its own over-claims along the way) that asks, property by property, what c_i says and what it cannot say.

The calibrated answer is a middle, and over-reading it in either direction is the live self-trick. c_i is not a solved vector space (a real high-rank holistic residual remains), and it is not opaque (it is probe-readable property-by-property, additive in concepts, and ~0.71-readable with learned topic features). What it is: a precise lexical / CONTENT store — entities, numbers, claims, polarity, event-sets all readable, often exactly — that carries no abstract relational structure: it reads what content is present, not who-did-what-to-whom. That single fact drives both the chapter's safety vulnerability and its positive safety guarantee.

§19.1 How much is readable — and the ~0.71 ceiling

The centerpiece coverage test (R9): reconstruct the order-free field c_pool from a feature set, push the reconstruction through the real SONAR decoder, and score decode-SBERT against the original sentence (so the units match §18's B1c). The ladder: ceiling (true c_pool) 0.90; a learned map from the words recovers 0.50; the words plus every hand-built property (sentiment, tense, number, negation, coreference, NER, 40 topics, discourse — 113 dims) reach only 0.504 — hand properties add essentially nothing (+0.005) — over a raw-bag floor of 0.299. So a learned lexical map gets ~half; enumerable properties don't extend it. But the holistic "other half" is partly bad-feature artifact: replacing hand-properties with learned topic-field features lifts the same decode ladder to 0.71 of the 0.90 ceiling (W9), and no external text encoder beats that ~0.71 (W14: SBERT / Llama / kitchen-sink all cap there). The remaining ~0.19-SBERT residual is meaningful but structureless — high-rank fine lexical/entity specificity smeared over hundreds of dims (98 dims for half its variance, no dominant axis), not a clean missing semantic axis (W1). c_i is a quantified middle: property-readable, mostly lexically-reconstructible, with a hard high-rank remainder.

§19.2 No abstract relational structure — not even possession

Where c_pool is readable it is a near-additive bag-of-concepts (additivity cosine 0.86–0.91), and the binding that should sit on top of the concepts is absent as abstract structure. The decisive test (C3 / R5): train a role probe on active sentences, test on passives. Thematic-role transfer collapses to AUC 0.015 (anti-correlated — the probe really learned "agent = first mention") while surface first-mention transfers at 0.985; nested structure flattens the same way. No relation tested — agent/patient, event-order, comparison, location, causal, recipient, nesting — survives as construction-invariant binding; every one is surface-positional.

The single apparent exception almost fooled the loop. Wave-8's W13 reported possession binding invariantly (cross-form AUC 1.0) — and wave-9's W19 overturned its own chapter: that transfer is carried by the genitive morpheme ('s / of), not an abstract role. Neutralize the clitic ("A's B" → "A owns B") and binding collapses 1.0 → 0.043 (≈ chance). So even possession is a lexical/morpheme readout, not a relation. z has essentially no abstract relational structure — the honest, corrected headline.

§19.3 What it reads precisely vs lexically vs not at all

Resolving the aggregate coverage by content type sharpens the "content store" picture. z reads numbers PRECISELY — exact-digit recovery 1.0 in every magnitude bucket up to 250,000, log₁₀N r² 0.995, robust to 20% noise (precise to read, hard to steer/scrub — W16). It reads style / register lexically — word-choice carries ~all of it, c_pool adds nothing beyond the bag, so it is not an abstract style field (W18). And it reads the event SET precisely (AUC 0.949) but the timeline only as surface order (W17, above). So z is a precise content/lexical store for entities, numbers, style, and event-sets, with genuine abstract reading for only a handful of lexically/morphemically-cued relations (the genitive cue, negation-scope, coreference).

§19.4 A ~16% auditable atom dictionary — readable, not steerable

Is the content a per-neuron code? At the per-token, writer, and LLM-vocabulary level, no — the writers carry position/routing, not nameable semantics (§18's per-token SAE found 0 separable context features). But the pooled, order-free field does surface a real minority code: an SAE over c_pool yields 273 paraphrase-invariant nameable atoms at ~16% of energy — including abstract concepts (contrastive discourse, reported speech) — that are cross-lingual (W23: the same atom fires in en/fr/de/zh, non-EN content AUC 0.987). The honest scoping: these atoms are a real monosemantic, AUDITABLE dictionary (W20: 93.5% pass held-out auto-interp) but NOT causal levers — only 2.5% steer coherently. They are read-out features, not write-in handles. So the field is dominantly lexical, with a genuine auditable ~16%-atom minority recoverable only from the pooled field — neither "no atoms" nor "fully decomposable."

§19.5 Mechanism — the final LayerNorm rank-expands c_i ~53×

White-box capture of the residual stream at every probe point (no ablation) shows c_i crystallizes late and non-additively: cos-to-c_i is 0.007 at embed, 0.247 at L18, 0.169 just before the encoder's final LayerNorm — and 1.0 immediately after it. The single biggest jump in the entire stack is that final model.layer_norm (pre→post cos +0.831; the largest layer-to-layer rise elsewhere is +0.052). This resolves the earlier cos-0.182 puzzle: the raw late-FFN write lives in the un-normalized residual at ~15000× the final norm, and the LayerNorm rescales+recenters it into the c_i direction. W12 sharpens what the LN does: it rank-expands the low-rank pre-LN field (eff-rank 11) ~53× into the high-rank readable c_i (eff-rank 600), and is not linearly invertible (post→pre R² 0.35). So c_i is literally the residual after the encoder's final LayerNorm — and reading should happen post-LN.

§19.6 Safety — trust z for WHAT, not WHO-DID-WHAT (and this generalizes)

The no-abstract-binding result (§19.2) is a concrete adversarial vulnerability. A z-based relational auditor — "is this entity the perpetrator/agent of harm?" — scores AUC 0.90 within-voice but 0.014 cross-voice (W6): a meaning-preserving active→passive paraphrase flips it, because z reads "agent = first mention". A text/voice-aware rule reader stays at 1.0, proving the relation is unchanged — a genuine blind-spot, not a hard problem. W21 confirms it on real directional harm (threat-source / perpetrator / incitement: single-voice cross-AUC 0.037, anti-correlated — it calls the victim the perpetrator), and data-augmentation does not fix it (a novel nominalization still drops to ≈chance); only an explicit dependency-parse agent-slot restores AUC 1.0 (W11/W21). W17 adds the temporal sibling: re-narrating events out of order flips the read timeline.

The positive complement (W22): z is a paraphrase-robust CONTENT auditor — entity / quantity / claim / negation cross-paraphrase AUC 1.0 vs a lexical bag's 0.944 (the decisive case: a synonym-reworded claim with no shared word, z 1.0 vs bag 0.78), while the structure control stays fragile (role cross-voice 0.022). The split is sharp: trust z for "what content is present", not "who-did-what-to-whom".

And this is not a SONAR bug (W30). Re-running both tests on three other deployed sentence encoders — all-MiniLM-L6-v2, all-mpnet-base-v2, and the contrastive-retrieval e5-small-v2 — gives the same profile: all three are paraphrase-robust content stores (content cross-paraphrase AUC 1.0, identical to SONAR) and none reads abstract thematic role cross-construction (0.448 / 0.460 / 0.529 — near chance, including the structure-aware contrastive encoder). So "a precise content store with no abstract relational structure" is a property of mean-pooled sentence embeddings broadly, not of SONAR — pooling a sentence to one vector destroys construction-invariant binding regardless of architecture. The skeptical nuance cuts the SONAR way: SBERT/E5 are if anything more structure-blind (SONAR at least tracks surface position 0.985 and within-construction role 0.774; the others sit near chance, a near-pure lexical bag). The safety corollary is encoder-agnostic: the relational-auditing vulnerability generalizes to any mean-pooled sentence-embedding auditor — switching encoders does not close the hole; an explicit parse does.

§20 Length is a capacity axis, not a structure axis

Question (does sentence length change what z stores or how to read it?): §19 read c_i at a fixed scale. The user's standing questions were sharper: do reconstruction and the readout probes change with length — do you need a length-adaptive auditor — and would a smarter recombination rule beat uniform 1/N pooling? §20 stratifies every load-bearing measurement by encoder-token count (5 bins, vshort 1–4 → vlong 35+) to find out.

§20.1 Reconstruction degrades gracefully — genuine 1/N dilution, not a rare-word artifact

Per-token reconstruction is length-sensitive (per-token-SAE FVU rises 0.062 → 0.235 short→vlong; the "flat-across-length" guess was falsified), but field-level meaning degrades gracefully: the order-free field c_pool still decodes to SBERT 0.811 at ~43 tokens (≥83% of the 0.99 full-z ceiling) — long decodes are faithful paraphrases that drop word-order and list-tail items, not collapses. The decisive deconfound (L1b): on a rare-word-rate-matched subset the SBERT-vs-length curve still falls 0.985 → 0.824 (matched Δ −0.161 vs unmatched −0.181), so 89% of the degradation survives — this is real 1/N pooling dilution, not lexical rarity.

§20.2 Content readouts are single length-invariant directions — no length-adaptation needed

Training probes within each length bin and measuring the cosine between bin-specific weight vectors: numbers (r² 0.92–0.96, weight cosine 0.93–0.997), entity presence (AUC 0.996–1.0, cosine 0.986), and negation (cosine 0.87) are all flat — one length-agnostic global probe loses essentially nothing (pooled-vs-per-bin Δ −0.002 to −0.06), and this replicates on natural v2 sentences. The one length-varying probe is the agent/patient relational one — and it varies because it is near chance: weakly above chance only at short length (0.68) decaying to 0.45–0.51 by vlong, with a cross-length weight cosine of −0.93. That faint short-sentence "role" signal is an anti-correlated surface-position leak, not a thematic-role direction — it sharpens §19's no-binding result rather than overturning it.

§20.3 The dilution is per-token capacity, not the 1/N aggregation rule

SONAR's pooling is exact (z is the token mean), so to make the recombination question decisive we pass each token through a fixed-capacity bottleneck bi = P hi (m∈{64,128}) and ask whether a smarter recombiner beats uniform 1/N. It does not: at the m=64 bottleneck a learned attention-pool ties uniform 1/N to within 0.01 at every length (and DeepSets is worse everywhere); the learned weights show no content-upweighting (Spearman(a_i, c_i-norm) = −0.09), only a mild late-position tilt that buys no FVU. The decode-SBERT secondary metric confirms the non-win is about meaning, not just L2 norm (long bin, m=128: uniform 0.301 / attention 0.300 / DeepSets 0.204). You cannot reweight your way back to diluted content — the loss is at the per-token capacity level, not mis-aggregated.

§21 Three laws of mean-pooled sentence embeddings — and the parascope safety pillar

Question (can the §20 regularities become a predictive, falsifiable theory?): §20's three regularities are turned into three stated laws with closed-form predictions and pre-registered falsifiers, over the same backbone z = (1/N) Σi Rpospi[Rrole E(wi) + ci] — plus a fourth, deployment-facing pillar: does the live parascope monitor inherit z's profile? Each law was tested, several were corrected or rescoped by an adversarial follow-up loop, and the safety claim was honestly bounded.

§21.1 LAW 1 — capacity / signal-to-dilution: one shape, causally generative

Read content off the real pooled z by greedy round-trip decode (no SAE in the loop), bin 1621 items by token count, and fit the Frady superposition recoverability with a single capacity C. It forecasts: a single C fit on the endpoint bins only predicts unseen interior lengths N=60/80/120 at 0.96/0.90/0.75 vs observed 0.978/0.907/0.788 — MAE 0.021, no cliff, no per-fold compounding. (This corrects the old CAPACITY_LEDGER C≈350 ~3× upward: that was the fold-reader's capacity, not the embedding's.) It is causal, not just descriptive (NEW-3): since z is literally the masked-mean of token states, inject one concept token z' = z·N/(N+1) + δ/(N+1) and the injected concept is recovered at the rate Law 1 predicts for N+1 — 0.99/0.96/0.86/0.74 at N≈10/14/23/36 vs predicted 0.998/0.98/0.93/0.84 (mean shortfall −0.047, a hair harder than a real token, ruling out a spare-capacity loophole). Correction (§21.7, A4-lossy): the shape (flat-then-1/N knee, decoder-independent) is robust, but the scalar C is link/metric-dependent (free-V_eff C = 130 decode-SBERT / 26 Jaccard / 3186 cos-to-z, >100× spread) — so read "single shape," not "single constant."

§21.2 LAW 2 — no relational binding: an encoder property, instrument-validated

A linear functional of an additive pool of lexical codes cannot compute the agent⊗patient conjunction without a binding tensor — so relational blindness should be a property of the FORM, not a SONAR quirk. (a) Cross-encoder (C1): on role-balanced active/passive minimal pairs, thematic cross-construction AUC is ≤0.54 across five encoders (SONAR 0.060, LaBSE 0.277, gte-large 0.423, mpnet 0.426, e5-large 0.538) — none reaches the 0.7 falsifier — while content cross-paraphrase AUC is a universal 1.0. (b) Rescope (NEW-1): it is not a mean-pooling artifact — swapping the pooling operator to CLS or last-token leaves cross-construction thematic AUC at chance (mean 0.500 / CLS 0.506 / last 0.502, all ≤0.61), so no-binding is an encoder/training property, deeper than pooling. (c) Instrument-validated (NEW-2): the worry that the bilinear null is just a weak probe is closed by a positive control — a full-power bilinear (order-gate τ=0.79) recovers the one relation known to be bound, the genitive at AUC 1.0 (collapsing to 0.93 under clitic-neutralization, the W19 signature), while agent/patient stays at 0.007: a gap of 0.99. The same fully-powered instrument finds the one binder that exists and only that one — Law 2's nulls are real representational absences, not weak instruments.

§21.3 LAW 3 — capacity, not aggregation: projection-robust

§20.3's "uniform 1/N is near-optimal" used a single random bottleneck projection. A2 sweeps the projection type — learned-P (trained end-to-end jointly with the pooling head, the strongest case for a smart reader+pool), PCA-P (information-preserving top-m singular vectors), and 3 random seeds. A learned attention-pool still ties uniform 1/N to within ΔFVU 0.014 at every length bin under every projection family (DeepSets worse everywhere), and the learned weights never favor content (Spearman(a, c_i-norm) negative in all families, −0.09 to −0.58). So the night-6 tie was not a random-projection artifact: the dilution loss is irreversible at the per-token capacity level.

§21.4 The parascope safety pillar — trust z for WHAT, never for WHO or for TAMPER

The live deployment is a parascope: an LLM hidden state → predicted ẑ → z-auditor chain. Does it inherit z's content-readable/role-blind split? Yes (B1): predicted-ẑ quality matches the champion checkpoint (cos 0.430), a content auditor reads number/entity/claim off ẑ at AUC 0.73–0.85 (mean 0.78, ~88–96% of real-z's readability), and the thematic-role cross-voice AUC is 0.51 ≈ chance — the blind-spot is inherited intact. It is length-robust (B3): the deployed content auditor's ẑ-AUC declines only mildly with N, in lock-step with real-z, with the gap slope ≈ 0 (−0.0002/tok) — no length-dependent blind-zone.

The injection threat is read, not evaded (NEW-6 — corrects §21.7's norm-gate sentence): a budget-respecting (m=1) concept injection is fully visible to the trained content auditor — AUC 1.0 at short N and ≥0.88 even at N≈36, clean-host FPR 0, and the auditor fires at ≥0.88 at every tested magnitude (m=0.5..4). The earlier "a norm/energy check flags over-magnitude m≥2" claim is quantitatively wrong: the injected token is diluted by 1/(N+1), so ‖z'‖ first drops; the norm gate only crosses its threshold at m≈3–4 at short N and is blind to even a 4× write once N≥23. The safety claim survives in revised form — detection does not rely on the norm gate at all; the content auditor is the real defense and reads injections at any magnitude or length.

The honest boundary (NEW-7): can a z-monitor tell a planted concept from a genuinely present one? A discriminator separates smuggled from genuine at AUC 0.989 — which superficially says "tamper detectable" — but the same fair discriminator separates two natural clean sets (genuine vs concept-absent) at AUC 1.0, so the apparent signal is a content confound: tamper-excess over the content baseline is −0.059 (negative). A z-monitor detects content, not tampering.

§21.7 Wave-3/4 stress-tests — one correction, the rest hardened

A critique pass flagged that several §21 claims rested on a single instrument or a saturated metric; five pre-registered adversarial follow-ups (A3, A4-lossy, A5-thematic, B1-noise, B3) returned one upgrade, one correction, and three confirmations. Law 1 — "single shape, link-dependent scale" (A4-lossy): §21.1's "single constant C" is too strong as a number — the absolute C is strongly link/metric-dependent (free-V_eff C = 130 decode-SBERT / 26 token-Jaccard / 3186 cos-to-z, a >100× spread that fails the ±20% cross-metric gate), while the shape (flat-then-knee, monotone 1/N, no cliff, decoder-independent, r ≥ 0.983 across metric families) is robust. Restated: a single-shape signal-to-dilution law whose scale constant is reader/metric-relative. Law 2 — UPGRADED (A5-thematic): the no-binding null holds under the strongest readout the program can show recovers a true bilinear comparator (pooled thematic AUC 0.608 = its linear value; the lone MLP arm crossing 0.70 is a position-only confound killed three ways) — "no binding even under bilinear/nonlinear readout." Two secondary deflations: (i) the "gist is most noise-robust content type" claim survives in direction but its magnitude was ceiling-inflated — matched off a common 0.85 baseline the advantage falls from ~1.24× to ~1.15×, failing the ≥1.3× gate (real but weak ordering, not ~2×); (ii) B4's "content-defined preserved subspace" is more honestly coarseness-defined — fine/rare-lexical directions are partially preserved too (r² 0.35–0.38 vs the content-auditor 0.47), so it is a smooth coarse→fine gradient, not a content/non-content dichotomy. Both refinements trim adjectives; the load-bearing claims (Law 1's single-shape + relational≈0; the parascope's content-readable/role-blind split) stand.

§21.8 Cross-cutting — no-binding is an encoder property, the instrument is validated, the capacity law is causal

Three experiments probing assumptions underneath the three laws, each woven into §21.1–§21.2 above. NEW-1: Law 2 is an encoder/training property, not a mean-pooling artifact — isolating the pooling operator on a fixed backbone (mean / [CLS] / last-token, plus a CLS-trained retrieval encoder) leaves cross-construction thematic AUC at chance everywhere (≤0.603, none clears 0.61), so a non-pooled position-distinct sentence vector buys essentially nothing. NEW-2: the bilinear no-binding instrument is validated on a known positive — the same fully-powered comparator (order-gate τ = 0.79) recovers the genitive at AUC 1.0 (0.93 clitic-neutralized) while agent/patient stays at 0.007 (gap 0.99), closing §21.7's weak-instrument loophole. NEW-3: Law 1 is causally generative — an additive one-token injection z'=z·N/(N+1)+δ/(N+1) is recovered tracking the N+1 curve (mean shortfall −0.047), so the capacity law is a generative budget, not just a curve fit.

§21.9 The safety arc, part I — the LIMIT: a z-monitor detects presence, not planting (NEW-7)

The §21.4 boundary, stated as a law: a z-monitor can flag "money is here" but cannot flag "money was inserted" beyond noticing the host topic differs — which an attacker neutralizes by smuggling into on-topic hosts. Detection holds (genuine-vs-absent 0.99, smuggled-vs-absent 0.94), but the strongest smuggled-vs-genuine discriminator (0.989) merely equals the genuine-vs-absent content baseline (1.0); the tamper-excess over that baseline is −0.059, and the off-manifold signals are weak (‖z‖-shell 0.76, manifold-dist 0.66) and survive a re-norm adversary. Tamper-detection cannot be built on z's content geometry alone — it needs a signal outside z, e.g. the LLM residual that wrote z.

§21.10 The safety arc, part II — the MITIGATION, honestly scoped by its own audit (NEW-8, NEW-9, N10)

The fix (NEW-8) — parascope-consistency. The parascope supplies exactly the out-of-content signal §21.9 named: ẑ = the z predicted from the LLM's residual stream. An additive injection happens downstream of the LLM (at z), so for a tampered z′ the residual-predicted ẑ still reflects what the LLM actually wrote (concept absent) — z′ and ẑ disagree; a genuine host has both read it and agree. On held-out real LLM-paragraph hosts (parascope trained on shards 0–23, tested on 24–26; cos(ẑ,z)=0.403), define discrepancy = (concept-reading of z) − (concept-reading of ẑ). The real, honestly-scoped result is tamper-vs-clean: smuggled and clean drawn from the same negative-host pool (perfectly residual-topic-matched, differing only by the injection) separate at AUC 0.853 — a working detector for "was this z tampered?" when you hold the host fixed.

The self-correction (N10) — the planted-vs-genuine headline was a host-topic confound. An earlier framing reported a smuggled-vs-genuine AUC of 0.787 and read it as a "planted-vs-genuine" detector. A pre-registered audit (N10) fired its falsifier: that 0.787 rode entirely on reading_ẑ — genuine hosts' residual legitimately reads the concept, smuggled hosts' does not, so reading_ẑ alone separates them at AUC 0.919 (R²=0.48 of the discrepancy) — the same host-topic axis §21.9 caught for z-only, re-imported on ẑ. After equalizing the reading_ẑ distribution (bin-match and OLS-residualization agree), the discrepancy no longer separates smuggled from genuine: topic-matched AUC 0.51 — chance (reading_z alone separates smug-vs-gen at only 0.03/0.28/0.18 across money/location/time). So parascope-consistency is not a planted-vs-genuine detector; the genuine-host false-flag rate (NEW-8's mean 0.35) was this same confound surfacing — a discrepancy threshold is partly a reading_ẑ threshold.

The stress-test, corrected (NEW-9 + N10) — an adaptive attacker evades. Against an attacker who controls z (post-SONAR) but not the host's LLM residual: (1) magnitude — sweeping injection magnitude m, attack efficacy and detector AUC move in lockstep (corr +0.986); the only m that drops the detector ≤0.6 also drops efficacy below 0.7, so no magnitude evasion window exists (this finding stands). (2) structural — ẑ = f(LLM residual), out of the z-level attacker's control, so it cannot be forged (recomputing ẑ after tampering z gives max-abs-diff 0.0; this finding stands). (3) host-selection — the real evasion NEW-9 missed. NEW-9's host-selection arm varied ẑ-cos (reconstruction fidelity) — the wrong axis, and reported only a 3.8% "soft spot." The correct adaptive move is to smuggle into hosts whose residual already reads concept-adjacent (top-quartile reading_ẑ): there the smuggled host's reading_ẑ is high too, the discrepancy shrinks, and the detector drops to AUC 0.572 over 25% of hosts (top-half: 0.671 over 50%). The mitigation is therefore partial: a real tamper-vs-untampered detector (0.85, holding host fixed), not a planted-vs-genuine detector (0.51 topic-matched), and evadable by on-topic-host selection — far larger an evasion surface (25%) than first claimed.

§22 The hardening pass: the objective-set capacity dose-response

Question (does a training objective set an autoencoder's verbatim capacity, and by how much?): a controlled scaling ladder — 3 rungs (12M/31M/102M params, d_z=256 frozen) × 2 objectives (D = denoising autoencoder; P = 50/50 DAE + ParaNMT paraphrase) — lets us ask this causally: hold architecture and bottleneck fixed and vary only how strongly the training pairs paraphrase the input. Note this is a different lever from §11.6: that section refuted a dictionary-width attribution for SAE reconstruction ("training budget, not width"); this one is about the effect of the paraphrase training objective on the autoencoder's own verbatim content-word capacity. The two are distinct findings and do not overlap.

§22.1 An ordered dose-response, not a scaling law — and the axis is surface, not semantics

Adding a paraphrase objective causally reduces verbatim content-word fidelity: the content-word "knee" (the sentence length past which exact-wording recovery falls below threshold) drops from D ≈ 9.6 to P ≈ 2.9 content words at 12M, with the ratio replicating at 31M. A same-domain control arm P′ — the identical objective with ParaNMT swapped for weaker SONAR round-trip paraphrases (pre-registered before training) — kills the domain-confound explanation and grades the effect: P′ pairs are weaker paraphrases and P′ knees land between the D and P arms at every length bin. The three arms therefore order monotonically by paraphrase divergence (D > P′ > P), a genuine dose-response. It is emphatically not a "scaling law": three objective arms leave one residual degree of freedom, so the R²=.945 three-point fit certifies a monotone ordering, not a fitted functional form — no per-arm strength or leave-one-arm-out slope CI is claimed.

Extending the curve to 8 dose levels (built by band-filtering the existing paraphrase pools by similarity, no new generation; pre-registered) then corrects the mechanism. The knee tracks surface / verbatim divergence, not semantics: knee-vs-surface-Jaccard fits R² ≈ .91, while within a fixed generator SBERT semantic-similarity is inert — slope 0.11 content-words per sim-unit, R² ≈ .01 across sim .65→.82. The original 3-anchor R²=.945 rode a semantic axis only because each anchor happened to sit in a different surface regime. Pooled across all 8 levels the slope is ~22.5 content-words per unit of surface divergence (CI [17.5, 27.5], containing the old 25.1) — the coefficient survives, its interpretation changes. The same surface-divergence axis also governs the composition-operator rotation (§11): "the knee tracks verbatim demand" and "the role-rotation tracks verbatim demand" are one statement.

§22.2 What the effect is not: not a storage breakpoint, and never a single "3.3×"

Two over-readings are cut by construction. First, the deficit is not a storage / capacity breakpoint. It is already present at 1–2 content words, where a 256-dimensional bottleneck cannot possibly be capacity-bound — so the "knee" is a threshold readout of a smooth quality×length curve, a verbatim-fidelity effect at all lengths, not a point where storage runs out. Second, the magnitude is a bracket between two non-commensurable estimands, reported together and never as a bare "3.3×": a matched-competence estimand (compare arms at equal validation F1) gives ~1.24–1.5× — 1.5× [1.2, 2.1] at 12M/31M, compressing to 1.24 [1.15, 2.90] at 102M — while a matched-budget estimand gives ~2.3–3.3×. These are two different questions (matching on budget launders the P arm's 50% DAE-token cut into the effect; matching on F1 conditions on a post-treatment outcome), so the bracket ~[1.24–1.5×, 2.3–3.3×] is the claim; the lenient "3.3×" is one ParaNMT-strength- and budget-specific corner of it, not a "capacity number." Consistent with the dose-response being coefficient-pinned to competence, the matched-competence component shrinks toward 1 as scale (and competence) grow, while the absolute surface-divergence slope grows.

One withdrawn number, kept visible. The first-reported full-budget C_P knee "8.01 [3.03, 8.47]" is withdrawn as a point estimate: the estimator has a −1.5 to −6.5 content-word bias on the plateau-shaped C_P curve (bin 5–6 recall sits 0.0004 above threshold), with bootstrap CI coverage as low as 0%, so the knee is unidentified in ≈[3, 9] and every downstream clause that consumed 8.01 is re-scored indeterminate. What survives the correction: the qualitative direction (paraphrase cuts verbatim capacity), the monotone D > P′ > P ordering, and the surface-not-semantic axis. Tier: the direction and ordering are T2 (seed-stable, P′-confirmed, domain-confound killed); "scaling law", a single "3.3×", and the C_P 8.01 point are T0 (withdrawn).

Built from public artifacts. Figures: matplotlib; all numbers and text examples read directly from result JSONs on nickypro/sonar-sae.