Decodability Is Not Abstraction: What a Pooled Sentence Embedding Encodes, in One Hundred Pre-Registered Experiments

1. Introduction DONE

1.1 The object, and the question

SONAR is a text autoencoder. Its encoder reads a sentence and produces a single vector z of 1024 numbers; its decoder reads z and writes a sentence back. Nothing else passes between them (Figure D1). That vector is then used as if it were the sentence: as a key for retrieval across two hundred languages, and, in Meta's Large Concept Model line, as the state a language model reasons in, one sentence-vector at a time.

This paper asks a specific question about z: does it store who did what to whom? "The doctor visits the cook" and "the cook visits the doctor" contain the same words. The difference between them is which noun is the agent (the one acting) and which is the patient (the one acted on). Linguists call these thematic roles, and the act of attaching a role to a word is called binding. A representation that binds roles could tell the two sentences apart for any pair of nouns and any way of phrasing the sentence. A representation that does not bind roles can still tell these two sentences apart, by remembering which words came in which order, but it has no general variable that means "agent" (Figure D2).

The distinction matters in practice. If z encodes roles only as a word-order side effect, a retrieval system built on z is one shared word away from confidently returning the reversed proposition (§11). A planner that reasons in z cannot express "swap the roles" (§7). A monitor that reads z cannot see the difference between a claim and its reverse (§8).

1.2 What was known

The question is old. Ettinger et al. (2018) probed a 2400-dimensional autoencoder latent, that is, trained a small classifier to read a property off the vector, and found it much better at word order (92.9%) than at agent-of-verb (63.7%), with no analysis of whether their probe could have found a role code had one existed. ROLE (Soulos et al., 2020) learned role decompositions freely and found sentence encoders barely beat a bag of words. Nikolaev and Padó (2023) showed that sentence-transformer similarity is driven by which nouns appear, with their roles "largely irrelevant". None of these supplied a null that could be read as evidence of absence. That is what we add: a test that holds out sentence structure and vocabulary at the same time, a power check that plants a fake role signal to show the probe would have seen a real one, and a comparison of what a probe can read against what the decoder can read from the same vector.

1.3 How the work was done

Rather than one experiment with many ablations, we ran 100 experiments (hereafter rows, numbered 001–100 and listed in Appendix A) in eleven thematic blocks, roughly one per hour over eight days. Each row froze its predictions before any compute ran and scored them afterwards (§2.3). The campaign was then consolidated by three independent breaker threads with code review (H1–H3). In September it was subjected to five audits and five mechanism experiments run by Claude, and to a twenty-branch iterative loop run by a second model family, Codex (Appendix B). Several campaign-time numbers did not survive that process, and this paper reports the corrected ones.

1.4 The finding

Decodability is not abstraction. We call a fact decodable if some reader can recover it from z, and abstracted if it is stored as one variable that any reader can use on any sentence. Role information is decodable from z in three ways: a decoder head trained to emit the argument triple gets the agent right 99.8% of the time on order-swapped sentences row 003, T3; a role-correct query beats its role-swapped twin in retrieval row 084, T3; and, on a closed bank of four verbs, a reader that is told which noun to ask about reads roles at 0.98–0.99 A2, T3. Yet no probe, whether linear, kernel, MLP or bilinear, finds a role variable that carries over to new sentence structures and new words, at any layer of the encoder, under a certified power check rows 001–020, T3; H1-hardened. Nineteen attempts to install such a variable, by changing the training objective, decorrelating the data, changing the architecture, scaling 44×, adding instructions, cross-attention, latent reasoning, diffusion or speech, produced none on a SONAR-scale vocabulary rows 001–010, 051–059, T3. The single exception, a small model given a thousand-word vocabulary, is what the theory in §5 explains (Figure D4).

Stated in the vocabulary of V-information (Xu et al., 2020): the role information usable by a linear reader is near zero, the role information usable by a 24-layer decoder is large, and nothing in information theory forbids that gap.

Contributions:

1. A certified null on a global linear thematic-role axis in pooled z, with an independent breaker, the retraction of its one apparent counterexample, and a scope statement on what the certificate covers H1-hardened.
2. A mechanism, derived on a toy model and validated on the real 24-layer encoder, with a new positive result that order is stored per queried noun, construction-generally H3-hardened; E02, T2.
3. A linear, composable, causally steerable operator algebra over closed-class markers, with its boundary characterized and its specificity claim withdrawn H2-hardened; A5.
4. A ~460-bit decoder-extractable capacity account with a zero-free-parameter rate-distortion prediction of the knee rows 066–072, 098, T3.
5. The methodology as a runnable benchmark.

2. Setup DONE

2.1 Representations

The primary object is SONAR z (Duquenne et al., 2023): a 1024-dimensional sentence embedding from a 24-layer transformer encoder whose token states are mean-pooled, meaning that the vector for a sentence is the plain average of the encoder's final state for each of its tokens. It is paired with a generative decoder trained on a machine-translation objective with auxiliary MSE and denoising terms. Averaging the layer-24-normed token states ourselves reproduces the shipped pipeline's vector exactly (cosine minimum 1.00000; re-confirmed in September at 0.9999997) H3-hardened; E01, so "pooled z" is the object under study. Because the decoder takes an explicit target-language token, language identity in z is functionally unnecessary, which matters in §4.4.

We also use three ancillary substrates where SONAR cannot answer a question. The TAE ladder is a family of small text autoencoders we train ourselves (smallest rung ~11.6M parameters, d_z 256), used where a question needs training runs, checkpoint series, or sweeps over seeds and bottleneck width. A sparse autoencoder (SAE) tries to rewrite each z as a sum of a few "atoms" drawn from a large learned dictionary, in the hope that the atoms are interpretable pieces of meaning; we train them over z and over encoder-stack states (block E). Block F uses ten external representations (§4.2), and September added one independent inversion system, GTR with vec2text (§11). Mixing substrates is a real limitation, flagged per claim in §3 and §13.

2.2 The binding battery

Our core instrument is a binding battery: a set of test sentences and a protocol for training and testing a probe, a small classifier that reads z and answers one question about the sentence.

Each item is a two-argument proposition, such as "the doctor visits the cook", rendered in one of five constructions: active, passive, cleft ("it is the doctor who visits the cook"), object-relative ("the cook whom the doctor visits"), and nominalization ("the doctor's visit to the cook"). A construction's parity is whether the agent comes before the patient in the sentence: active and cleft are agent-first, passive and object-relative are patient-first. Each item names a focal noun, and the probe's question is: is the focal noun the agent or the patient?

The battery is a grid (Figure D3). A cell trains the probe on one combination of construction and vocabulary and tests it on another. The within-construction ceiling trains and tests on the same construction and the same nouns; it asks whether z carries anything about the item at all (SONAR ≈ 1.0 on within-vocabulary items). The primary cell trains on one construction and tests on another with the nouns held out too: a probe that passes this cell has found a role variable that works regardless of phrasing and regardless of which words are involved, which is what "abstract role variable" means operationally. The parity-flipped and within-vocabulary controls are where a probe that merely reads surface position gives itself away by scoring far below chance: trained on agent-first sentences, a position reader labels every patient-first sentence backwards. Readouts are linear and MLP with 95% bootstrap confidence intervals clustered by proposition. Scores are AUC (area under the ROC curve), where 0.5 is a coin flip and 1.0 is perfect.

Six small embedders tested in row 051 reach only 0.54–0.74 on the ceiling cell, which is why their chance-level primary cells are reported as uninterpretable rather than as nulls.

2.3 Discipline: pre-registration, gates, Brier scores, ledger, tiers

Every row froze its predictions, each with a probability, before any compute ran. At harvest each prediction was scored with a Brier score: the mean squared error between the stated probabilities and what happened. Zero is perfect; 0.25 is what you get from always saying 50%; 1 is confident and wrong. Across the 100 rows the mean is 0.185 and the median 0.179; the best-calibrated rows are 094 (0.003), 091 (0.005) and 018 (0.035). The September follow-ups scored E01 0.230, E02 0.225, E03 0.278, E04 0.180 and E04b 0.150, and the five audit predictions a mean of 0.280.

Gates are positive controls and nulls that decide whether a result may be interpreted at all. A failed gate makes the verdict INSTRUMENT_FAILURE: the numbers are then reported as uninterpretable, not as evidence for anything.

The Bayesian ledger (row 095) aggregates rows into a posterior for each of nine claims by adding up log-odds. Each row's contribution is set by its direction and strength, shrunk by its Brier score, and clamped to ±5 log-odds because rows within a block share stimuli and code. Two controls accompany it: a shuffled-evidence null (500 trials; every claim regresses to its prior) and a prior sensitivity sweep (prior ∈ [0.25, 0.60]; the top claim and top-3 set do not change). It is bookkeeping of the campaign's own evidence, not a promotion mechanism, and its posteriors are not calibrated probabilities that a claim is true.

Evidence tiers and tags. The program's house tiers are: T1, pre-registered and replicated with gated controls; T2, seed-stable but thinner; T3, exploratory; T0, withdrawn or an undefined estimand. Every campaign row is T3. A breaker is an independent re-implementation whose job is to break a result; a claim that survives one is hardened. Codex is an LLM code reviewer and second experimenter used throughout as an independent auditor; in September it also ran its own experimental loop. In-text tags read [row 003, T3] (campaign evidence), [H1-hardened] (survived the August consolidation), [E02, T2] (a September experiment and its tier), and [RETRACTED] or [T0] (withdrawn). September units are lettered by thread: A audits, E mechanism experiments, U unbinding, S sparse-autoencoder branches, G the second inversion system, O operator audits, R literature-gap pilots; every one is listed with its sample size in Appendix B. Claims are numbered C1–C9 (§3). "Readout" below means the same as "probe".

2.4 Six standing cautions

Instrument lessons the program paid for; they apply to every result below.

1. Certify probe power before trusting a null, and say what the certificate covers. A chance-level probe is evidence of absence only if the same probe, on the same representation, in the same transfer regime, recovers a planted signal of known effect size. Row 051 fails this gate and is reported as blocked. A certificate against an additive mean shift says nothing about a rotation or superposition code. In a synthetic check with two orthogonal role matrices, global linear readers scored AUC 0.507 and a query-bilinear reader 0.999 on the same planted code, while both reader classes score 1.000 on an additive plant R01 pilot. Kit: row 090.
2. Check label and slot canonicalization. Focal-swap conventions, label balance, slot-index alignment, and lexical-holdout disjointness are silent failure modes; rows 017 and 020 hit real ones. Kit: row 091.
3. Cross-validate role probes by noun, not only by proposition. A probe cross-validated by proposition while nouns recur across folds measures per-noun fingerprints, not roles. Under that recipe even random noise assigned per noun scores 0.93, so a high score proves nothing about roles E01 breaker. This is what retired one of the consolidation's numbers (§5.2).
4. Norm-profile any difference-of-means or offset construction. Magnitude confounds come in two modes, artifact (dies under cosine) and confound (survives it), needing different fixes; the headline norm↔density r = 0.88 collapsed to 0.21 under cosine. Kit: row 092.
5. Never map a standardized probe direction back to raw z without a variance floor. z has near-dead coordinates that barely vary across sentences. One of our edit directions put most of its weight on two of them, which the decoder never reads, so the edit did nothing [E04].
6. Decode-based claims need validity gates, and even then miss the near-manifold case. When z underdetermines a sentence the decoder invents plausible content (fabrication, §8); on clean sentences this happens 3.6% of the time (row 021), on stressed inputs 20–30%. Vectors that sit where real sentence vectors sit (on the manifold) fabricate without tripping any check we built.

2.5 How much data is behind each number

Three terms first. A seed is the random starting point of a training run or a data split; two seeds are two independent repeats, and a single-seed result could be luck. A bootstrap re-draws the test items at random, with replacement, many times (here usually 1,000) and re-scores each draw; the spread of those scores is the confidence interval, written [low, high]. Clustered by proposition means each draw takes whole propositions (every rendering of "the doctor visits the cook") together, because they are not independent items.

The campaign was cheap by design: one experiment per hour, on one to three RTX A4000 GPUs, with the longest row taking about 75 minutes of GPU time and most taking under ten. That bought breadth, and it cost depth. A read-only audit of every row's pre-registration, result file and outputs (full table in SAMPLE_SIZE_AUDIT.md; compact columns in Appendix A) classified each row's headline evidence as ok (hundreds or more independent items with clustered uncertainty, or an exact count), thin (fewer than about 200 items, or no clustered uncertainty, or a single trained seed where the claim depends on training), or very thin (fewer than about 50 items, or a single seed of a trained model with no repeat). Of the 90 rows that make an empirical claim, 30 are ok, 36 are thin and 24 are very thin.

Three consequences for reading this paper. First, the flagship null does not rest on the thin rows. Block A's single-seed models are individually weak evidence, but the C1 claim is carried by the H1 breaker (five seeds, 1,000 bootstrap resamples, an independent re-implementation), by block F's three-seed batteries, and by the E02 breaker, and the same null recurs in every one of those places. Second, the capacity, operator and safety blocks are where the numbers should be read most cautiously. Every one of the five "independent" confirmations of the 460-bit budget rests on 48–64 items per cell with no interval; the composition result (decode gap 0.000) rests on 24 items per operator pair; and the retrieval result that one shared word overturns a role-correct margin rests on 48 hand-built triples whose query was itself a poor retriever. These are reported at full strength in their sections, with their n beside them, and their section confidence marks reflect the audit. Third, the September follow-ups are better powered than the campaign, but not uniformly. Of the 51 follow-up units, 17 are ok, 21 thin and 10 very thin. The unbinding, count-grammar and token-state anatomy runs use 300–1,024 independent sources with clustered bootstraps. Others are thinner: the query-conditioned order reader rests on one set of 22 test nouns; the first causal test of that reader used 37 propositions and its replacement 160; the multi-event decoder witness rests on 20 base propositions; the theory's checks on the real encoder used 40 token multisets with no intervals; and the SAE seed-stability figures come from six unconverged dictionaries per family. Appendix B carries an n column and an evidence flag for every September unit.

GPU model and wall time were not recorded in 47 of the 100 result files. Where they were, the whole campaign ran on RTX A4000s; no row used more than three of them, and none ran for more than about an hour and a quarter.

4. Results: binding DONEHIGH null and retraction H1-hardened with five seeds; block A rows individually thin; §4.5 is MED

Blocks A (001–010, can binding be induced?), B (011–020, where does role information live?), F (051–060, beyond SONAR), G (061–065, multilingual binding), with the September reader studies.

In one paragraph. We asked whether a probe can read "agent or patient?" off z in a way that carries over to sentence structures and words it never saw. It cannot: it scores at chance in every version of the test we built, at every layer of the encoder, and we know the test had the power to detect a real signal because it detects a fake one we planted. We then tried nineteen ways of training a model to have such a variable, and none worked at SONAR's vocabulary size. What z does carry, strongly, is surface word order, tangled with the specific words. One apparent exception in Japanese and German turned out to be an artefact and is retracted. And in September we found that order is readable from z if the reader is told which noun to ask about, which is what "tangled with the words" predicts.

4.1 The null, and what it certifies

The result. In the primary cell of the battery (train on one construction, test on another, with the nouns held out too) the probe scores at chance for every kind of readout and at every depth of the encoder. The consolidation breaker (H1) re-implemented the battery independently, on real SONAR, with five seeds and 1000 bootstrap resamples, and reproduced the campaign numbers to within ±0.002 H1-hardened:

Why chance here is informative. A probe that scores 0.5 could be failing because there is nothing to find, or because it is too weak to find it. We rule out the second by planting a fake role signal into z, of a known size, and checking that the same probe in the same test recovers it. It does: a global role direction of Cohen's d = 1.0 is recovered at AUC 0.973 (lower confidence bound 0.876), and the smallest effect the probe could have seen is d = 0.70 H1-hardened. So a linear role direction of moderate size would have been found. The MLP probe is weaker in transfer (it overfits the training construction; smallest detectable d = 1.5), but the flagship claim is about linear binding, and row 016 separately certified kernel, focal-conditioned MLP, and bilinear probes at planted amplitudes and found the same null row 016, T3.

What the certificate does not cover. The planted signal is an additive shift. A role code that worked by rotating the representation, or by superposing roles across dimensions, would not be caught by this check (caution 1); §4.5 is what addresses those.

What z does carry. In the cell where a probe that reads surface position must get every answer backwards (same nouns, opposite parity), the linear probe scores 0.266 and the MLP 0.118, far below chance: both are confidently reading position, not role. That is the sharpest positive signature in the campaign: z knows which noun came first H1-hardened.

4.2 Nineteen ways to fail to install binding

The question. If SONAR did not learn a role variable, could a model be made to learn one? Block A tried on the TAE ladder, changing the training objective, the data, and the architecture. Block F asked whether other families of sentence representation already have one.

Changing the objective does not help. A contrastive objective that explicitly penalized confusing a sentence with its role-swapped twin drove its own loss from 1.33 to 0.32, and gained nothing on held-out transfer (0.501, against 0.501 for an untrained model): the model satisfied the objective by memorizing which specific words go together [row 001]. A supervised head that was trained to answer "who is the agent?" reached 100% on held-out sentences at every loss weight, while the transfer battery stayed flat [row 002]. Forcing the decoder to emit an explicit (agent, verb, patient) triple produced 0.998 agent retrieval on order-swapped sentences and exactly chance transfer [row 003].

The one exception is instructive. Holding the objective fixed and widening only the filler vocabulary, role abstraction emerges: with 117 filler words, novel-word retrieval is 0.05; with 1000, it jumps to 0.73 row 006, T3. Abstraction is available to this architecture. It is simply not the cheaper solution at small vocabulary, and §5 explains why.

Other representations do no better. Block F tested nine and found no binding in any rows 051–059, T3:

The only regime shift in the block is driven by the training objective, and it is not binding: training a model to generate translations installs the surface-order code that training it to rank them does not row 052, T3. This agrees with a 2026 finding that supervision from synonym and definition pairs actively degrades relational concepts in sentence encoders (Mohr et al., 2026). The null is what the objective predicts, not an anomaly.

The most uncomfortable number. An LLM judgment panel, asked which of two sentences is closer in meaning to a target, picks the paraphrase over the role-swap 90 times out of 90. Cosine similarity on z picks the role-swap 30 times out of 30, and within a topic the correlation between z-cosine and judged similarity is negative (−0.218) row 060, T3, LLM-proxy only; the human study was designed and never run.

4.3 Where role information lives, and does not

Inside the encoder. Role-consistent attention computation exists: 42 of the encoder's 384 attention heads route from agent to predicate consistently across all constructions, significant after Bonferroni correction and replicated on held-out data row 012, T3. But switching all 42 off changes the decoder's role fidelity by +0.001; the circuit is causally inert row 013, T3. Patching a token's state from the role-swapped sentence flips the decode 95% of the time through layer 22, and 0% at layer 24: the encoder commits late and all at once row 014, T3. The order code is local to the filler tokens at every depth (0.86–0.96) row 017, T3, and a trained attention pooler on frozen states recovers nothing at any depth: the null is a property of the token states, not of averaging as a readout row 015, T3.

What the token states are made of. Two September experiments asked how well simple models describe the final-layer token states. A nested ridge regression predicts each state from (i) its own token, (ii) its two neighbours on each side, (iii) the unordered bag of tokens in the sentence, and (iv) sampled bigram interactions. On trained SONAR, the token's own identity explains R² 0.370 of the variance, the neighbours add 0.019, the bag 0.002, the interactions 0.002, and 0.606 is unexplained. On a randomly initialized twin of the encoder, the same features explain 0.938 E01, T2. Nor is the unexplained part a sentence-wide "broadcast": adding the average of the other tokens' states lifts R² only from 0.474 to 0.509, whereas the random twin goes from 0.624 to 0.972 E03, T2. Training removes the simple structure. About half of what a trained token state contains is not captured by any local or bag-of-words model we could fit at 162k tokens.

4.4 The multilingual exception, retracted

What was claimed. Rows 061/062 reported the campaign's first order-invariant role binding, in languages that mark roles with case rather than word order: German 0.668 and Japanese 0.658 on the strict primary cell, Japanese replicating at 0.696 on a fresh set of nouns. They were the only two rows contradicting C1 in the ledger, and they did not survive.²

Why it was wrong. Each item's probe label had been fixed by the alphabetical order of the focal noun's English gloss, a convention the Japanese encoder cannot see. The H1 breaker ran one faithful pipeline on a third, disjoint set of nouns and obtained 0.451 [0.383, 0.522], chance, while the same pipeline gave 0.696 on 062's nouns and 0.79 on 061's. A swing of 0.45 / 0.70 / 0.79 across three noun sets is incompatible with a stable role code, and it is unlikely to be a power problem: a planted d = 1.0 signal on the third set is recovered at 0.71, above chance though below the 0.9 bar the benchmark uses, and the third set has only 8 test nouns. The mechanism tests refuted their own pre-registration. Swapping the Japanese case particles が↔を, which should invert a genuine case-reader, does not invert the probe (0.78, not ~0.30), and a representation built only from noun+particle chunks does not recover the signal (0.36). The signal was a whole-sentence, noun-set-specific correlation with the arbitrary label H1-hardened, RETRACTED. What is stable across all three noun sets is surface-order reading (anti-transfer 0.10–0.25 in the within-vocabulary cell).

What stands in block G. Language in SONAR is an interface property: a decoder token plus a small, removable mean offset. Steering z along the language direction flips the output language 0% of the time row 064, T3. A 24-item French/English/Chinese pilot run in September is consistent with that. Decoding French directly into Chinese through one vector preserves role reversal and number contrasts, and routing through English text first is a real transformation (cosine 0.92 to the source vector), not an identity multilingual pilot. That is a decodability observation on hand-authored items; it says nothing about the probe null.

4.5 Query-conditioned readers: order is stored per noun MED order T2, single noun set; role T3, predicate-general access open

The question. The unconditioned null leaves two possibilities open. Either role information is absent from z, or it is present but addressed: stored in a way that a reader can only access if it knows which noun to ask about. The September reader studies settle this for word order and leave it open for role.

Order. The reader is bilinear in a query and the latent, s = qᵀUVᵀz, where q is the embedding of the noun being asked about (Figure D8). It was trained on active and passive sentences only, with the training and test nouns disjoint (65 / 22) and the verbs disjoint (18 / 6), then tested on constructions it had never seen. On the leak-free stratum of common nouns (74% of items) it reads surface order on cleft sentences at AUC 0.990, on object-relatives at 0.746–0.922, and on nominalizations at 0.802–0.869, the ranges depending on which families were in training E02, T2. A randomly initialized SONAR reads 0.504 / 0.478 / 0.493, so the code is learned. It is learned late: cleft transfer is 0.822 at layer 20, 0.987 at layer 22 and 0.994 at the final normed layer. The order direction is orthogonal to the construction-parity direction (cosine < 0.032 for all 87 nouns). Readers of the same bilinear form asked about role rather than order do not transfer (object-relative 0.169).

The independent breaker found the result stands, with scope limits that travel with the number. The pooled headline numbers (0.994 / 0.946 / 0.911) are inflated by a proper-name and determiner-case leak and should not be quoted. The order-blind static-bag floor is 0.50 / 0.72 / 0.55 rather than 0.5. Every "unseen noun" figure rests on the same 22 test nouns, and the bootstrap resamples propositions, not nouns, so the stated ±0.03–0.04 intervals understate noun-sampling variance E02 breaker.

What this means for the null. Unconditioned probes fail because order is stored per queried noun: there is a readable answer to "did doctor come first?" but no single direction that means "the agent came first". The binding deficit is on construction parity, which is readable for cleft (1.0) and nominalization (0.95) but not object-relatives (0.56). With only two patient-first families in the battery, a parity probe is effectively a family classifier, so whether parity is readable in general cannot yet be tested cleanly.

Role. Here the picture is genuinely two-sided. A query-conditioned MLP on z reads predicate-conditioned roles on held-out nouns at AUC 0.993 (conjoined clauses) and 0.982 (separate clauses), with wrong-query and missing-z controls at exactly 0.500. But it does so on a closed bank of four verbs shared between training and test A2, T3. On the same multi-event vectors, the native decoder recovers both predicate-conditioned roles correctly on 242 of 320 sources, while a candidate-likelihood reader gets both right on 5 of the 122 opposite-role vectors the decoder accepts (4.1%) binding/multievent, T3. That decoder count rests on only 20 base propositions rendered 16 ways each. We also tried to build a reader that answers "who is the agent of this verb?" for verbs it never saw. That study never reached its final test. On its practice set it read one event per sentence almost perfectly (0.998) but could not assign roles when a sentence contained two events (0.43, below the 0.60 bar we had set), and an added consistency loss made it refuse to answer rather than answer better (0.08). On the one held-out-verb panel that was scored, single events held at 0.99 and two-event assignment fell to about 0.10; the three training seeds disagreed widely (0.17–0.59) and no intervals were computed [binding/predicate_holdout, development only].³ Predicate-general role access is therefore unestablished in either direction. The null is a statement about a class of reader and a way of asking, not about what z contains, and the reader that would settle it has not been built.

Is the readable order code the one the decoder uses? No: readable is not decoder-special. Pushing z along the direction the order reader uses does not make the decoder swap the nouns (0 of the edits at the dose that fools the reader; 7% at double dose). Replacing z outright with the swapped sentence's own vector, which we call the oracle edit, swaps them 98% of the time, so the decoder can be moved; this direction just does not move it E04, T0 as registered. Most of that first result is an instrument artefact (caution 5). A re-registered follow-up on fresh stimuli (160 propositions, 40 new nouns, 8 new verbs, 960 edits) then asked whether the small subspace the reader uses (numerical rank 9) is special to the decoder E04b, T1. It is not. Swapping an item's coordinates in that subspace flips the decoded order in 0.140 of edits; energy-matched structured subspaces, such as the top principal components of the training differences, flip 0.124 and 0.110 (paired difference −0.001 [−0.04, 0.04]). Removing the reader's share from the full swap leaves 0.626 swapping, and removing the same energy from the control subspaces leaves 0.636 and 0.677. What the decoder responds to is how much of the AB→BA difference is moved inside any structured, data-aligned subspace: the same energy spread isotropically over about 334 random dimensions flips only 0.024, and the response is step-like (3% of the squared difference gives 0, a third gives 0.14, all of it gives 0.995).⁴ There is no low-dimensional order handle in z; the decoder reads order from a distributed difference.

5. Theory: why binding fails DONEMED mechanism exact for static pooling; real-encoder checks on 40 multisets without intervals; E01/E03 T2 and E04b T1 well powered

Rows 096, 097 (block L), consolidated in H3.

In one paragraph. Averaging is the culprit, and it is not a bug. If you average a set of context-free word vectors, the order of the words cannot matter: the average of the same three vectors is the same vector (Figure D5, top). SONAR's encoder gives each token a state that depends on its neighbours, so a little order does survive the average, but only in the form "this particular word, in this position, among these particular words". That is exactly what we see: order is readable if you ask about a specific noun, and not as one global direction. The theory below makes that precise, predicts when a small model will learn to bind and when it will not, and was checked against the real encoder.

5.1 The toy account

Row 096 builds a minimal mean-pooled autoencoder over (agent, patient) pairs drawn from a vocabulary of V filler words. Averaging destroys order except for a weak residual channel g = β(e_a − e_p), the difference between the agent's and patient's embeddings scaled by a small β. Write T for the number of training examples and K for the decoder's capacity. The model can solve its task with either of two codes. Bag lookup memorizes the role assignment for each specific pair of words; it is cheap but cannot generalize to a word it has not seen. Role reading learns to read g regardless of which words fill it; it costs a constant amount and generalizes. The number of distinct word pairs is about V²/2, and the memorization budget is min(T, K). Writing γ(β) for the usable signal in the order channel at strength β, the two codes cross over at

V* ≈ √(2 · min(T,K) / γ(β))

Below V*, memorizing pairs is the cheaper way to lower the loss, which predicts the near-zero novel-word accuracy of rows 001 and 006 at 30 and 117 fillers. Above V*, only the role reader lowers the loss, which predicts 006's jump at 1000 fillers. The numerical model falsified the row's own predicted cause: the transition is set not by the amount of data or the model's capacity but by the signal-to-noise ratio of the surviving order channel row 096, T3, Brier 0.136.

Row 097 supplies the formal half. Averaging over positions is the Reynolds operator for the group of all permutations, the linear map that projects any function of a sequence onto the part that does not depend on order. On context-free embeddings it destroys order exactly. On contextual states order survives through one channel only. That channel is the bag of words re-weighted by how much attention each token received. The re-weighting depends on position only through the encoder's relative-position bias. Split that bias into a symmetric part and an antisymmetric part (the part that flips sign when two tokens swap places); the antisymmetric part is the only thing that says who came before whom, and it is precisely 096's β residual row 097, T3, Brier 0.121.

The Reynolds identification is not ours. All linear permutation-invariant readouts, sum and mean pooling named explicitly, factor through the Reynolds operator and erase symmetry-aware components regardless of how expressive the encoder is (Talhi et al., 2026), and Reynolds operators have been built as network layers (Sannai et al., 2021). That theorem covers linear readouts over a linear group action. SONAR's pool is linear only after a nonlinear, position-aware pre-map, so encoder-plus-pool is a learned Janossy pooling of arity greater than one (Murphy et al., 2019) rather than a Deep Sets model.⁵ Every property we observe follows: order is present, local, content-entangled by construction because a k-ary feature is a function of the identities of its k tokens, and it does not compose globally. Two consequences are testable. Injectively representing a 20-token multiset in a 1024-dimensional token space would need on the order of 41,000 latent dimensions (Wagstaff et al., 2019), but that is a sufficiency bound whose violation predicts a cliff, not the smooth knee of §6, which favours the rate-distortion account. And no piecewise-linear Janossy pooling is injective except on multiplicity-free multisets (Reshef and Dym, 2025), predicting that collisions concentrate on sentences with repeated tokens. That order survives in a mean-pooled embedding at all is already published (a probe separates shuffled from unshuffled SBERT inputs at ~89%; Chavan et al., 2023), which is why we claim the mechanism and the locality dissociation, not the discovery.

5.2 Validation on the real 24-layer encoder

H3 tested seven predictions on real SONAR contextual states: five hold, one missed quantitatively, and one was later retracted as a measurement of role H3-hardened. These checks used 40 token multisets with 200 permutations each and reported no intervals; the well-powered part of the anatomy is E01, E03 and E04b (§4.3, §4.5).

• Static pooling is order-blind to precision. Permuting the tokens of a sentence and averaging context-free embeddings changes the result by 5.1·10⁻⁷ (we call the size of that change the shuffle dispersion), and a probe for order within such a bag scores at chance.
• The contextual pool smuggles order, per instance but not globally. Real z recovers order within a fixed set of tokens at AUC 0.997 (shuffle dispersion 0.565), while a probe that must read order across different sentences scores 0.523, and the cross-proposition binding-order probe scores 0.507 at every layer. Order is stored entangled with content: the below-chance parity signature of §4.1, derived from first principles, and the same shape as E02's per-noun order code.
• The reversal-odd component is noun-dependent and order-sensitive. The reversal-odd component of z is the half that flips sign when the two arguments are swapped, (z(a,p) − z(p,a))/2. Role transfer across constructions is below chance (0.26–0.35) on it, a surface reader anti-transferring on a parity flip. The consolidation also reported role as recoverable from this component at 0.76–0.86; that reading is withdrawn.⁶
• β accelerates in L20→L23 and is amplified ~6× by the final LayerNorm, matching the layer-22 commitment and layer-24 cliff of row 014, and E02's L20 → L22 emergence of the order code.

The quantitative miss, and the key refinement. The toy predicts an order subspace of a handful of dimensions; on real SONAR it takes ~23 principal components to reach 90% of the variance (the top 3 give only 30%). Twenty-four nonlinear layers spread the channel wider, and E02's reader subspace is consistent with that (effective rank ≈ 7 for the order direction), though no low-dimensional subspace is causally special to the decoder (§4.5). More importantly, β is small before the final LayerNorm (~0.01–0.05) but amplified ~6× to ~0.29 by it, so "weak β" means weak globally readable signal-to-noise, not weak norm: content-entanglement, not amplitude, is what defeats a linear role reader H3-hardened. Codex review found one real latent bug, a missing zero-variance guard in order_rank, fixed in the original campaign script.

This account also predicts the binding null from outside. Every positive result on entity binding in language models depends on a per-token-position address (Feng and Steinhardt, 2024). Propositional probes recover entity-role bindings at 0.97 exact match, but both of their stages are token-indexed, so after pooling the method is undefined rather than merely harder (Feng et al., 2024). Mean pooling annihilates precisely the address, and the content-keyed fallback that takes over under load in autoregressive models (Gur-Arieh et al., 2026) is exactly the content-entangled channel that survives here.

7. Results: operators and steering DONEMED algebra H2 code-reviewed, but on 24–56 items per operator without intervals; specificity withdrawn

Blocks D (031–040, geometry) and J (078–081, composition operators), consolidated in H2 and audited in September (O1, A5).

In one paragraph. Some things can be done to z with simple vector arithmetic. Add a fixed "negation" vector and the decoder says "does not"; add a "past tense" vector and it says "visited"; add both and it says "did not visit". These offsets are linear, they compose, they can be undone, and they work on natural sentences at a sensible dose. The one thing no offset does is swap who did what to whom: the "voice" offset produces a passive sentence with the roles unchanged (Figure D6). That is the paper's internal consistency check: the edit that fails is exactly the variable §4 could not find. Under audit, one claim from this block, that the edits change nothing else in the sentence, did not hold, and is withdrawn.

7.1 The algebra and its wall

Closed-class morphosyntactic markers, the small set of grammatical words and endings such as not, -ed, above, act on z as approximately linear, invertible offsets that compose additively and in any order. Tense, number, negation, and question are clean linear analogy operators row 033, T3. Five relational marker families (before/after, the vertical above/below, in/out, near/far, bigger/smaller) all linearize at success 1.00 row 078, T3. Adding two single-operator directions gives the same result as fitting the double transform directly (mean additivity 0.982, compose-minus-direct decode gap exactly 0.000), across families and in either order, on 24 held-out items per operator pair with a single judge row 079, T3. An MLP readout never beats the linear one, which is the evidence that these really are linear, and the offsets survive removing the top principal components of a z that is nearly isotropic anyway row 037, T3. They are causal: adding α·v steers negation, tense, number, and a spatial marker from 0 to ~1.0 success by α = 1 on templated stimuli, the edit can be undone exactly, and a random push of the same size does nothing row 080, T3.

The wall. Argument reversal does not linearize: 0/74 argument swaps in the operator zoo, 0/40 in the analogy battery. The voice operator produces correct passive form with the arguments unswapped rows 033, 078, T3. A linear offset can substitute a word; it cannot reverse who-relates-to-whom. The one transform the algebra cannot perform is exactly what pooled z cannot linearly represent (§4). An independent read-side precedent exists: across formal meaning manipulations, role swap is on average the hardest for embedding models to detect (Sentence Smith, 2025).

7.2 Natural text and specificity MED audited; specificity withdrawn; vertical undefined

Code review. Codex review certified the steering core correct (offsets fitted on training data only, clean α arithmetic, no leakage between operators), so the 078/079/080 causal claims stand. It confirmed that composition must be certified on the decoded text, never on cosine similarity: row 079's cosine-additivity check had passed for the wrong reason. It also found one real latent bug. The 080 analysis counted garbled decodes as successes, harmless on templated data but inflating success on natural text. With a fluency gate, negation's apparent 0.90 success at α = 2 is fabrication; the honest peak is 0.70 at α = 1 H2-hardened.

Natural sentences. Applying the same operators to held-out natural sentences (entity-rich, 6–26 words) over a wider range of doses breaks the templated result's "0% garble": as 080 itself predicted, garble appears at high dose and is dose-localized (runaway "…not not not…"). The September audit then re-judged the same 104 natural bases under a rubric that reads the whole proposition rather than two attributes, with Wilson intervals, and the specificity claim did not survive [A5, O1] (two independent joins of the same judgments differ by one negation row, hence the range):⁹

In words: at the standard dose the operator usually installs its marker, but it puts the negation or the tense in the right place only about 40% of the time, and it often changes something else in the sentence. Operator safety is not uniform. Tense is the most robust to over-steering (fluent 0.85 at α = 2, breaks at α = 3) and negation is safe only in a narrow window around α ≈ 1. The vertical operator's natural-text evaluation had no applicable items: none of the 24 sources contains a physical above-relation (all 16 uses of "above" are idiom or discourse), and the edit direction was fixed above→below while the scorer expected a per-item pole flip.¹⁰ Operators are near-identical across seeds on SONAR's fixed basis (cosine 0.98–0.995), the honest contrast to the per-seed-trained ladder row 075, T3; H2-hardened.

Two warnings from outside the program. Steering vectors in language models are anti-steerable on roughly half of inputs across forty datasets, and steerability is largely a property of the dataset (Tan et al., 2024; Braun et al., 2025). An independent group found English-to-Spanish direction vectors in SONAR that are clean, additive and decoder-inert (Hawthorne and Nellessen, 2025). In-distribution steering success demonstrates local linearity, which every smooth map has. What we claim is a local, dose-bounded, decode-certified editing algebra over closed-class markers with a characterized wall, not a global algebra.

7.3 Monitoring the edit

How do you know an edit has pushed z somewhere the decoder cannot handle? The flag that works on natural text is the round-trip re-encode cosine rt_cos: decode the edited vector, re-encode the result, and compare. It falls from 0.99 to 0.29 as the dose goes from α = 0 to 6, and separates fluent from garbled output at AUC 0.845. The nearest-neighbour-to-a-bank metric used in rows 083/087 does not transfer to diverse natural text (AUC 0.779, flat), a real correction to the safety block's monitor choice. The deliverable is latent_rewrite.py, a monitored rewrite primitive with a norm-linter pre-flight and an rt_cos flag that withholds off-manifold outputs. It refuses 91.7% of high-α garble, accepts 100% of fluent α = 1 rewrites, and refuses argument swap by policy on the row-078 wall H2-hardened.

7.4 Geometry, briefly MED two T3 rows

Two block-D negatives are load-bearing. Norm is not meaning [C7]: the length of z tracks sentence length and specificity (partial r 0.41 with perplexity) but scaling z by ±30% changes the decode not at all row 036, T3. The antipode −z is not an opposite: it is indistinguishable from a random vector of the same length and orthogonal to the negation operator row 038, T3. The decoder is nevertheless norm-sensitive at larger scalings (§8.3). Anisotropy in z is best explained by positional outlier dimensions, which are known to matter specifically for mean-pooled sentence embeddings (Luo et al., 2021), rather than by the tied-softmax frequency mechanism that a bottleneck encoder lacks.

13. Limitations DONE

Tier. Every campaign result here is T3-exploratory; nothing has been formally promoted in the program's claims ledger. The consolidation showed C1 is promotable for the linear claim, but that entry has not been written (§15). The September rows carry their own tiers (Appendix B). Every semantic judgment in them is a model judgment; no human reviewed any output in either loop.

Sample sizes. §2.5 and Appendix A give the number of evaluation items behind every row. The campaign's typical row rests on hundreds to low thousands of items, but a minority of headline numbers rest on fewer than fifty, and those are named in §2.5 and marked in the row table. Any number in this paper without a confidence interval beside it should be read with that table open.

What the certificate covers. The binding null is an English cross-construction unconditioned linear-probe null, certified against an additive planted effect. Rotation and superposition codes are not covered by that certificate, and §4.5 shows that a query-conditioned reader recovers order where the unconditioned one cannot. The scope is a reader class and a query regime, not the information content of z.

Single model family, English-centric templated stimuli. SONAR plus one small TAE ladder. Block F's ten external representations extend the binding null specifically; six embedders in row 051 failed their positive-control gate and are reported as blocked; the one second-system replication (GTR/vec2text) covers counts, not roles. The crosslingual claims are the weakest here and two rows have been retracted (§4.4). Content-level and null claims should transfer to other mean-pooling encoders, because the mechanism is architectural. Geometry-level claims (the specific decay constants, the operator directions, any cosine threshold) should not, and SONAR's own authors needed a nonlinear connector and ~14M paired samples to align a second encoder into its space.

Substrate mixing, and one post-hoc number. Some claims blend SONAR and ladder evidence, particularly capacity (C2) and ontogeny (C4); C4 is ladder-only and should be labeled as such wherever quoted. The 460-bit ceiling was not frozen in a prereg and is a decoder-extractable lower bound rather than a mutual-information estimate.

Correlated rows inflate naive evidence. Rows within a block share stimuli, code, and often checkpoints. The ledger's conservative likelihood ratios, Brier shrinkage, and ±5 clamp are mitigations, not solutions. C1's 0.993 is at the clamp and should be read as "the campaign's evidence saturates the ledger's allowance," not as a calibrated probability.

Withdrawn numbers. Four campaign- or consolidation-time results have been withdrawn or rescoped since the previous version: the 061/062 multilingual binding positives (§4.4), the antisymmetric-component role readout (§5.2), the operator specificity figure (§7.2), and the vertical-operator natural-text result (§7.2). Two program headlines that predate the campaign, the unbinding "floor" and the 41–49% silent-flip rate, are re-scoped in §6.2 and §8.2.

Numerical reproducibility. SONAR encodings under PyTorch's high matmul precision are not batch-invariant (maximum relative L2 drift 6.8·10⁻⁴ across batch groupings; highest is invariant to 1.4·10⁻⁶). A pre-registered confirmation of the query-conditioned role reader on 400 fresh sources was closed by that gate rather than by a scientific result. Any claim that reuses frozen readers on newly encoded inputs is not bit-reproducible under the default stack.

Negatives reported at full strength. The auditing block is mostly failures (§8). Row 060 shows z-cosine anti-correlates with (proxy) human similarity judgment within topic. Row 013 shows the role-structured attention we found is causally inert. Row 023 refuted the program's own "fluent lies off-manifold" picture. E04 and E04b found that the readable order direction is not a causal handle.

15. Open problems DONE

What remains undone, as of 11 September 2026.

1. The ledger rerun. bayesian_ledger.py has not been rerun with H1–H3 and the September rows as support and rows 061/062 demoted; until then C8's posterior is stale in a known direction.
2. Formal promotion. No campaign claim has been promoted above T3 in the program's claims ledger. C1 is promotable, with its scope statement, but the entry has not been written.
3. Predicate-general role access. The reader that would decide whether role is addressable across held-out predicates does not exist; the paired-objective attempt induced abstention, and the single-event final panel has not been opened.
4. The decoder's order boundary. E04b ruled out a reader-specific order subspace; whether a single "structured energy" variable predicts the decoded flip across subspace types (E05) is pre-registered and unrun.
5. A rotation-code power certificate on real SONAR. The R01 pilot is synthetic; the binding null is certified against additive plants only.
6. A noise-robust decoder. Every fabrication and audit number was measured with the base SONAR decoder; the Large Concept Model's fine-tuned decoder is the cheapest way to learn whether the fail-closed and coverage results are about the latent or the decoder.
7. Applicable-text operator editing. The physical-relation study's 152 final sources were never decoded; the vertical operator has no natural-text result.
8. Thin rows. The rows §2.5 flags as resting on very few items should be re-run at larger n before any of their numbers is quoted outside this paper.
9. Figures (Appendix C), human calibration for row 060, and TAE-Bench public release (repository, license, fresh-clone GPU smoke test).
10. Blocked rows. Row 009 needs a verified binding teacher; row 051 needs a per-embedder recalibrated battery.
11. Second-corpus replication of the safety block on a non-templated corpus, and a second seed.

Appendix A: the 100 rows DONE

Generated from board_data.json. Outcome is the row's own verdict class: signal (pre-registered effect found), null (no effect, gates permitting interpretation), twist (a real result that contradicted the prereg), blocked. Brier is scored against that row's frozen predictions (lower is better). Lane is the current status: 🟩 hardened (survived consolidation or directly supports a hardened claim), 🟦 shipped (a released artifact), ⬜ T3 (campaign evidence only), 🟥 refuted, ⬛ blocked, 🟨 partial. Follow-up is the row's own answer to "is a follow-up worth funding?" (Y / N / ?). ★ marks a row the campaign flagged as a headline result at harvest time; two starred rows (061, 062) are now refuted.

Appendix B: the September 2026 follow-ups DONE E05 is pre-registered only and excluded

Work after the campaign closed, in three threads. H1–H3 (14 August) are the consolidation breakers. The fable loop (10–11 September; Claude Fable 5.1 with subagents; Codex contributed the plan breaker and one blinded outcome review) ran five audits of the overnight branches and four mechanism experiments. The lit-gap loop (10–11 September; Codex, with Claude as blind reviewer and critic) ran the twenty-branch iterative programme derived from the literature review. Tiers are as in §2.3, evidence flags as in §2.5. Every review in both loops was a model review. Ids in the first column are the ones used in the body's evidence tags; the full per-unit audit is in SAMPLE_SIZE_AUDIT.md.