# Decodability Is Not Abstraction: What a Pooled Sentence Embedding Encodes, in One Hundred Pre-Registered Experiments
**Nicky Pochinkov** · Version 0.3 · 2026-09-11
> *Editorial note.* This version incorporates the September 2026 follow-up work (five audits, five
> mechanism experiments, and a twenty-branch iterative loop; Appendix B), the twenty-strand
> literature review, and a sample-size audit of every experiment (§2.5). Where a number reported in
> the 6 September version has since been corrected or retracted, the corrected statement is in the
> body and the history is in a footnote. The nine diagrams are explanatory; the data figures are not
> yet rendered (Appendix C). Evidence tags are explained in §2.3.
>
> Every section header carries two marks. **Completeness** says whether the section is finished to
> submission standard: ✅ DONE, 🟡 PARTIAL (a named item is owed; see the status box), ⬜ TODO.
> **Confidence** says how far the section's central claims should be trusted, on the program's own
> evidence tiers: ◆ HIGH (hardened by an independent breaker, or T1/T2 with replication), ◆ MED (T3
> evidence that is internally consistent and has replicated where re-run, but has no independent
> breaker, or a hardened claim with a rescoped sub-claim), ◆ LOW (ladder-only, single-row, or a claim
> recently withdrawn or half-refuted). A subsection is marked only where its confidence differs from
> its section's. Sections without a confidence mark make no empirical claim of their own.
## Status at a glance
| § | section | completeness | confidence | what is owed, or what the confidence rests on |
|---|---|---|---|---|
| — | Abstract | 🟡 PARTIAL | — | Over a 250-word venue cap. |
| 1 | Introduction | ✅ DONE | — | Positioned against Ettinger 2018, ROLE, Nikolaev & Padó. |
| 2 | Setup | ✅ DONE | — | Battery, tiers, cautions and the sample-size audit (§2.5): 30 rows ok, 36 thin, 24 very thin. |
| 3 | Results overview | 🟡 PARTIAL | ◆ MED | Ledger not rerun since 2026-08-08; C8 and C1 posteriors stale in a known direction. Tallies exact. |
| 4 | Binding | ✅ DONE | ◆ HIGH | Null and its retraction are H1-hardened (five seeds, 1,000 bootstrap resamples) and reproduced by the E02 breaker; block A's single-seed rows are individually thin but agree. §4.5 is MED: order reader T2 on one 22-noun test set; role side T3 and open. |
| 5 | Theory | ✅ DONE | ◆ MED | The static-pooling half is exact. The real-encoder checks (H3) survived code review but used 40 token multisets with no intervals, and one sub-claim was withdrawn (footnote). E01/E03 (T2, 8,000 sentences) and E04b (T1, 160 propositions) are the well-powered anatomy. Reynolds half is published elsewhere. |
| 6 | Capacity | 🟡 PARTIAL | ◆ LOW | Every capacity cell is 48–64 items with no interval and a single seed; no independent breaker; the 460-bit figure was post hoc and has not been re-frozen. §6.2 is MED: 512–1,024 chains with clustered bootstraps. |
| 7 | Operators | ✅ DONE | ◆ MED | Algebra, wall and causality survived H2 code review and natural-text re-runs, but the composition result rests on 24 items per operator pair and the zoo on 18–28 items per transform, single judge, no intervals. Specificity withdrawn; vertical undefined. |
| 8 | Fabrication | 🟡 PARTIAL | ◆ MED | Near-manifold half of C5 refuted and open; every number measured with the base decoder (noise-robust comparison unrun). Re-scoped flip rate rests on exact replication plus a pre-registered degradation study. |
| 9 | Dictionaries | 🟡 PARTIAL | ◆ MED | Block negatives replicate exactly in September; C6 specificity is LOW and untested at balanced activity; paired dictionaries unconverged. |
| 10 | Training dynamics | 🟡 PARTIAL | ◆ LOW | Ladder-only; no SONAR checkpoint series; C4 posterior 0.568. |
| 11 | Safety | 🟡 PARTIAL | ◆ LOW | 32–96 hand-built items per row, single seed, no intervals; the retrieval demonstration's query was itself a poor retriever. Second system replicates counts (64 sources), not roles. |
| 12 | Instruments | 🟡 PARTIAL | ◆ HIGH | Kits reproduce past Briers exactly (0.003–0.042); TAE-Bench validated locally, not released. |
| 13 | Limitations | ✅ DONE | — | — |
| 14 | Related work | 🟡 PARTIAL | — | Identifiers verified by the literature review; not every work read at primary source by the author. |
| 15 | Open problems | ✅ DONE | — | — |
| — | References | 🟡 PARTIAL | — | Twelve entries cited by title only. |
| A | The 100 rows | ✅ DONE | — | Generated from `board_data.json`, with the sample-size audit's `n` column. |
| B | September follow-ups | ✅ DONE | — | E05 (pre-registered only) excluded. |
| C | Figures | ⬜ TODO | — | Nine explanatory diagrams exist; zero data figures rendered; nine planned. |
**Overall: 9 of 20 sections DONE, 10 PARTIAL, 1 TODO. Confidence: 2 HIGH, 5 MED, 3 LOW. 0 of 9 planned data figures made.**
---
## Abstract 🟡 PARTIAL (over a 250-word venue cap)
A sentence embedding squeezes a whole sentence into one vector, and systems increasingly treat that
vector as if it were the sentence's meaning: as a search key, as the state a "concept-space" language
model plans in, as something a safety monitor can read. We ask what one such vector actually holds.
The object is SONAR's 1024-number sentence vector `z`, produced by averaging the states of a 24-layer
encoder and read back by a decoder. We ran one hundred pre-registered experiments on it, each with
predictions frozen before compute and scored afterwards, followed by an independent attempt to break
each headline and a month of adversarial follow-up. Three findings survive. **First**, `z` does not
store "who did what to whom" as a reusable variable. A classifier asked to tell agent from patient,
and forced to work on sentence structures and words it never trained on, scores at chance (0.51),
while the same classifier detects a planted signal of moderate size at 0.97. Yet the information is
there: a decoder trained to emit the (agent, verb, patient) triple gets the agent right 99.8% of the
time, and a reader that is told which noun to ask about reads word order at up to 0.99 on unseen
words and structures. **Decodability is not abstraction.** **Second**, we explain why. Averaging
token states erases word order exactly for context-free embeddings; the contextual encoder leaks
order back in, but tangled with the words themselves, so it is readable per noun and not as one
global direction. **Third**, grammatical markers such as negation and tense act as linear, composable,
causally effective offsets on `z`, and the one edit that fails, swapping which noun does what (0 of 74), is
exactly what `z` cannot store. We also measure a budget of roughly 460 bits per vector, and predict with no free
parameters the sentence length at which the decoder starts dropping content; and we show, on a
48-item demonstration, that one shared word can flip a role-correct retrieval. All results are on one model family; the null is certified against additive alternatives only;
and a third of the hundred rows rest on fewer than fifty items.
---
## 1. Introduction ✅ DONE
### 1.1 The object, and the question
SONAR is a text autoencoder. Its encoder reads a sentence and produces a single vector `z` of 1024
numbers; its decoder reads `z` and writes a sentence back. Nothing else passes between them (Figure
D1). That vector is then used as if it were the sentence: as a key for retrieval across two hundred
languages, and, in Meta's Large Concept Model line, as the *state* a language model reasons in, one
sentence-vector at a time.
Figure D1. A text autoencoder. Everything the decoder knows about the sentence has to pass through z.
This paper asks a specific question about `z`: does it store **who did what to whom**? "The doctor
visits the cook" and "the cook visits the doctor" contain the same words. The difference between them
is which noun is the *agent* (the one acting) and which is the *patient* (the one acted on). Linguists
call these *thematic roles*, and the act of attaching a role to a word is called *binding*. A
representation that binds roles could tell the two sentences apart for any pair of nouns and any way
of phrasing the sentence. A representation that does not bind roles can still tell *these two*
sentences apart, by remembering which words came in which order, but it has no general variable that
means "agent" (Figure D2).
Figure D2. Binding: what it is, and why it matters for anything built on z.
The distinction matters in practice. If `z` encodes roles only as a word-order side effect, a retrieval
system built on `z` is one shared word away from confidently returning the reversed proposition (§11).
A planner that reasons in `z` cannot express "swap the roles" (§7). A monitor that reads `z` cannot see
the difference between a claim and its reverse (§8).
### 1.2 What was known
The question is old. Ettinger et al. (2018) *probed* a 2400-dimensional autoencoder latent, that is,
trained a small classifier to read a property off the vector, and found it
much better at word order (92.9%) than at agent-of-verb (63.7%), with no analysis of whether their
probe could have found a role code had one existed. ROLE (Soulos et al., 2020) learned role
decompositions freely and found sentence encoders barely beat a bag of words. Nikolaev and Padó (2023)
showed that sentence-transformer similarity is driven by *which* nouns appear, with their roles
"largely irrelevant". None of these supplied a null that could be read as evidence of absence. That is
what we add: a test that holds out sentence structure *and* vocabulary at the same time, a power check
that plants a fake role signal to show the probe would have seen a real one, and a comparison of what
a probe can read against what the decoder can read from the same vector.
### 1.3 How the work was done
Rather than one experiment with many ablations, we ran **100 experiments** (hereafter *rows*,
numbered 001–100 and listed in Appendix A) in eleven thematic blocks, roughly one per hour over eight
days. Each row froze its predictions before any compute ran and scored them afterwards (§2.3). The
campaign was then consolidated by three independent breaker threads with code review (H1–H3). In
September it was subjected to five audits and five mechanism experiments run by Claude, and to a
twenty-branch iterative loop run by a second model family, Codex (Appendix B). Several campaign-time
numbers did not survive that process, and this paper reports the corrected ones.
### 1.4 The finding
**Decodability is not abstraction.** We call a fact *decodable* if some reader can recover it from
`z`, and *abstracted* if it is stored as one variable that any reader can use on any sentence. Role
information is decodable from `z` in three ways: a
decoder head trained to emit the argument triple gets the agent right 99.8% of the time on
order-swapped sentences [row 003, T3]; a role-correct query beats its role-swapped twin in retrieval
[row 084, T3]; and, on a closed bank of four verbs, a reader that is told which noun to ask about
reads roles at 0.98–0.99 [A2, T3]. Yet no probe, whether linear, kernel, MLP or bilinear, finds a role
variable that carries over to new sentence structures and new words, at any layer of the encoder,
under a certified power check [rows 001–020, T3; H1-hardened]. Nineteen attempts to *install* such a
variable, by changing the training objective, decorrelating the data, changing the architecture,
scaling 44×, adding instructions, cross-attention, latent reasoning, diffusion or speech, produced
none on a SONAR-scale vocabulary [rows 001–010, 051–059, T3]. The single exception, a small model
given a thousand-word vocabulary, is what the theory in §5 explains (Figure D4).
Figure D4. The same z, three readers. The information is present; what is missing is a single variable that means "agent" for every sentence.
Stated in the vocabulary of V-information (Xu et al., 2020): the role information *usable by a linear
reader* is near zero, the role information *usable by a 24-layer decoder* is large, and nothing in
information theory forbids that gap.
Contributions:
1. A certified null on a global linear thematic-role axis in pooled `z`, with an independent breaker,
the retraction of its one apparent counterexample, and a scope statement on what the certificate
covers [H1-hardened].
2. A mechanism, derived on a toy model and validated on the real 24-layer encoder, with a new positive
result that order is stored *per queried noun*, construction-generally [H3-hardened; E02, T2].
3. A linear, composable, causally steerable operator algebra over closed-class markers, with its
boundary characterized and its specificity claim withdrawn [H2-hardened; A5].
4. A ~460-bit decoder-extractable capacity account with a zero-free-parameter rate-distortion
prediction of the knee [rows 066–072, 098, T3].
5. The methodology as a runnable benchmark.
---
## 2. Setup ✅ DONE
### 2.1 Representations
The primary object is **SONAR** `z` (Duquenne et al., 2023): a 1024-dimensional sentence embedding
from a 24-layer transformer encoder whose token states are **mean-pooled**, meaning that the vector
for a sentence is the plain average of the encoder's final state for each of its tokens. It is paired
with a generative decoder trained on a machine-translation objective with auxiliary MSE and denoising
terms. Averaging the layer-24-normed token states ourselves reproduces the shipped pipeline's vector
exactly (cosine minimum 1.00000; re-confirmed in September at 0.9999997) [H3-hardened; E01], so
"pooled `z`" *is* the object under study. Because the decoder takes an explicit target-language token,
language identity in `z` is functionally unnecessary, which matters in §4.4.
We also use three ancillary substrates where SONAR cannot answer a question. **The TAE ladder** is a
family of small text autoencoders we train ourselves (smallest rung ~11.6M parameters, `d_z` 256),
used where a question needs training runs, checkpoint series, or sweeps over seeds and bottleneck
width. A **sparse autoencoder** (SAE) tries to rewrite each `z` as a sum of a few "atoms" drawn from a
large learned dictionary, in the hope that the atoms are interpretable pieces of meaning; we train
them over `z` and over encoder-stack states (block E). Block F uses ten external representations (§4.2), and September added one independent
inversion system, GTR with vec2text (§11). Mixing substrates is a real limitation, flagged per claim
in §3 and §13.
### 2.2 The binding battery
Our core instrument is a **binding battery**: a set of test sentences and a protocol for training and
testing a *probe*, a small classifier that reads `z` and answers one question about the sentence.
Each item is a two-argument proposition, such as "the doctor visits the cook", rendered in one of
five **constructions**: active, passive, cleft ("it is the doctor who visits the cook"),
object-relative ("the cook whom the doctor visits"), and nominalization ("the doctor's visit to the
cook"). A construction's **parity** is whether the agent comes before the patient in the sentence:
active and cleft are agent-first, passive and object-relative are patient-first. Each item names a
**focal** noun, and the probe's question is: *is the focal noun the agent or the patient?*
The battery is a grid (Figure D3). A **cell** trains the probe on one combination of construction and
vocabulary and tests it on another. The **within-construction ceiling** trains and tests on the same
construction and the same nouns; it asks whether `z` carries anything about the item at all (SONAR ≈
1.0 on within-vocabulary items). The **primary cell** trains on one construction and tests on another
*with the nouns held out too*: a probe that passes this cell has found a role variable that works
regardless of phrasing and regardless of which words are involved, which is what "abstract role
variable" means operationally. The **parity-flipped and within-vocabulary controls** are where a probe
that merely reads surface position gives itself away by scoring far *below* chance: trained on
agent-first sentences, a position reader labels every patient-first sentence backwards. Readouts are
linear and MLP with 95% bootstrap confidence intervals clustered by proposition. Scores are AUC (area
under the ROC curve), where 0.5 is a coin flip and 1.0 is perfect.
Figure D3. The binding battery. Only the bottom-right cell tests an abstract role variable; the power check says the probe would have found one.
Six small embedders tested in row 051 reach only 0.54–0.74 on the ceiling cell, which is why their
chance-level primary cells are reported as uninterpretable rather than as nulls.
### 2.3 Discipline: pre-registration, gates, Brier scores, ledger, tiers
Every row froze its predictions, each with a probability, before any compute ran. At harvest each
prediction was scored with a **Brier score**: the mean squared error between the stated probabilities
and what happened. Zero is perfect; 0.25 is what you get from always saying 50%; 1 is confident and
wrong. Across the 100 rows the mean is 0.185 and the median 0.179; the best-calibrated rows are 094
(0.003), 091 (0.005) and 018 (0.035). The September follow-ups scored E01 0.230, E02 0.225, E03
0.278, E04 0.180 and E04b 0.150, and the five audit predictions a mean of 0.280.
**Gates** are positive controls and nulls that decide whether a result may be interpreted at all. A
failed gate makes the verdict `INSTRUMENT_FAILURE`: the numbers are then reported as uninterpretable,
not as evidence for anything.
The **Bayesian ledger** (row 095) aggregates rows into a posterior for each of nine claims by adding
up log-odds. Each row's contribution is set by its direction and strength, shrunk by its Brier score,
and clamped to ±5 log-odds because rows within a block share stimuli and code. Two controls
accompany it: a **shuffled-evidence null** (500 trials; every claim regresses to its prior) and a
**prior sensitivity sweep** (prior ∈ [0.25, 0.60]; the top claim and top-3 set do not change). It is
bookkeeping of the campaign's own evidence, not a promotion mechanism, and its posteriors are not
calibrated probabilities that a claim is true.
**Evidence tiers and tags.** The program's house tiers are: **T1**, pre-registered and replicated
with gated controls; **T2**, seed-stable but thinner; **T3**, exploratory; **T0**, withdrawn or an
undefined estimand. Every campaign row is T3. A **breaker** is an independent re-implementation
whose job is to break a result; a claim that survives one is **hardened**. **Codex** is an LLM code
reviewer and second experimenter used throughout as an independent auditor; in September it also ran
its own experimental loop. In-text tags read `[row 003, T3]` (campaign evidence), `[H1-hardened]`
(survived the August consolidation), `[E02, T2]` (a September experiment and its tier), and
`[RETRACTED]` or `[T0]` (withdrawn). September units are lettered by thread: **A** audits, **E**
mechanism experiments, **U** unbinding, **S** sparse-autoencoder branches, **G** the second inversion
system, **O** operator audits, **R** literature-gap pilots; every one is listed with its sample size in
Appendix B. Claims are numbered C1–C9 (§3). "Readout" below means the same as "probe".
### 2.4 Six standing cautions
Instrument lessons the program paid for; they apply to every result below.
1. **Certify probe power before trusting a null, and say what the certificate covers.** A
chance-level probe is evidence of absence only if the same probe, on the same representation, in
the *same transfer regime*, recovers a planted signal of known effect size. Row 051 fails this gate
and is reported as blocked. A certificate against an additive mean shift says nothing about a
rotation or superposition code. In a synthetic check with two orthogonal role matrices, global
linear readers scored AUC 0.507 and a query-bilinear reader 0.999 on the *same* planted code,
while both reader classes score 1.000 on an additive plant [R01 pilot]. Kit: row 090.
2. **Check label and slot canonicalization.** Focal-swap conventions, label balance, slot-index
alignment, and lexical-holdout disjointness are silent failure modes; rows 017 and 020 hit real
ones. Kit: row 091.
3. **Cross-validate role probes by noun, not only by proposition.** A probe cross-validated by
proposition while nouns recur across folds measures per-noun fingerprints, not roles. Under that
recipe even random noise assigned per noun scores 0.93, so a high score proves nothing about roles
[E01 breaker]. This is what retired one of the consolidation's numbers (§5.2).
4. **Norm-profile any difference-of-means or offset construction.** Magnitude confounds come in two
modes, *artifact* (dies under cosine) and *confound* (survives it), needing different fixes; the
headline norm↔density r = 0.88 collapsed to 0.21 under cosine. Kit: row 092.
5. **Never map a standardized probe direction back to raw `z` without a variance floor.** `z` has
near-dead coordinates that barely vary across sentences. One of our edit directions put most of its
weight on two of them, which the decoder never reads, so the edit did nothing [E04].
6. **Decode-based claims need validity gates, and even then miss the near-manifold case.** When `z`
underdetermines a sentence the decoder invents plausible content (*fabrication*, §8); on clean
sentences this happens 3.6% of the time (row 021), on stressed inputs 20–30%. Vectors that sit
where real sentence vectors sit (*on the manifold*) fabricate without tripping any check we built.
### 2.5 How much data is behind each number
Three terms first. A *seed* is the random starting point of a training run or a data split; two
seeds are two independent repeats, and a single-seed result could be luck. A *bootstrap* re-draws the
test items at random, with replacement, many times (here usually 1,000) and re-scores each draw; the
spread of those scores is the confidence interval, written [low, high]. *Clustered by proposition*
means each draw takes whole propositions (every rendering of "the doctor visits the cook") together,
because they are not independent items.
The campaign was cheap by design: one experiment per hour, on one to three RTX A4000 GPUs, with the
longest row taking about 75 minutes of GPU time and most taking under ten. That bought breadth, and
it cost depth. A read-only audit of every row's pre-registration, result file and outputs (full table
in [`SAMPLE_SIZE_AUDIT.md`](SAMPLE_SIZE_AUDIT.md); compact columns in Appendix A) classified each
row's headline evidence as **ok** (hundreds or more independent items with clustered uncertainty, or
an exact count), **thin** (fewer than about 200 items, or no clustered uncertainty, or a single
trained seed where the claim depends on training), or **very thin** (fewer than about 50 items, or a
single seed of a trained model with no repeat). Of the 90 rows that make an empirical claim, **30 are
ok, 36 are thin and 24 are very thin.**
| block | rows | ok | thin | very thin | typical evidence behind a headline |
|---|---|---|---|---|---|
| A: can binding be induced? | 001–010 | 1 | 1 | 8 | one trained ~11.6M-parameter model per condition, one seed; evaluated on the 2,000-sentence battery, whose primary cell is 500 sentences from 50 held-out propositions |
| B: where role information lives | 011–020 | 8 | 2 | 0 | frozen SONAR on the 2,000-sentence battery with 1,000-resample proposition bootstraps |
| C: fabrication | 021–030 | 5 | 4 | 2 | 1,500 sentences, but only 54 fabrication events; row 024's mechanism rests on 38 sentences |
| D: geometry and operators | 031–040 | 2 | 6 | 1 | 40–56 items per transform, one LLM judge, no intervals |
| E: dictionaries | 041–050 | 5 | 4 | 1 | 1,000–5,000 sentences, usually one SAE seed, usually no intervals |
| F: beyond SONAR | 051–060 | 7 | 2 | 1 | the best-powered block: 3,200 items, three readout seeds, clustered bootstraps; row 060 is 100 pairs judged by an LLM proxy |
| G: multilingual | 061–065 | 1 | 4 | 0 | 250-proposition lexical-holdout cells, linear readers only |
| H: capacity | 066–072 | 0 | 5 | 2 | 48–64 items per cell, single seed, no intervals on any knee |
| I: training dynamics | 073–077 | 1 | 2 | 2 | one training run per condition (row 075: eight seeds) |
| J: composition operators | 078–081 | 0 | 2 | 2 | 24–40 held-out items per operator, single judge, no intervals |
| K: safety | 082–089 | 0 | 4 | 4 | 32–96 hand-built items, single seed, no intervals |
| L: instruments and theory | 090–100 | — | — | 1 | no data claim of their own |
Three consequences for reading this paper. First, **the flagship null does not rest on the thin
rows.** Block A's single-seed models are individually weak evidence, but the C1 claim is carried by
the H1 breaker (five seeds, 1,000 bootstrap resamples, an independent re-implementation), by block
F's three-seed batteries, and by the E02 breaker, and the same null recurs in every one of those
places. Second, **the capacity, operator and safety blocks are where the numbers should be read most
cautiously.** Every one of the five "independent" confirmations of the 460-bit budget rests on 48–64
items per cell with no interval; the composition result (decode gap 0.000) rests on 24 items per
operator pair; and the retrieval result that one shared word overturns a role-correct margin rests on
48 hand-built triples whose query was itself a poor retriever. These are reported at full strength in
their sections, with their n beside them, and their section confidence marks reflect the audit.
Third, **the September follow-ups are better powered than the campaign, but not uniformly.** Of the
51 follow-up units, 17 are ok, 21 thin and 10 very thin. The unbinding, count-grammar and token-state
anatomy runs use 300–1,024 independent sources with clustered bootstraps. Others are thinner: the
query-conditioned order reader rests on one set of 22 test nouns; the first causal test of that reader
used 37 propositions and its replacement 160; the multi-event decoder witness rests on 20 base
propositions; the theory's checks on the real encoder used 40 token multisets with no intervals; and
the SAE seed-stability figures come from six unconverged dictionaries per family. Appendix B carries
an `n` column and an evidence flag for every September unit.
GPU model and wall time were not recorded in 47 of the 100 result files. Where they were, the
whole campaign ran on RTX A4000s; no row used more than three of them, and none ran for more than
about an hour and a quarter.
---
## 3. Results overview 🟡 PARTIAL ◆ MED (tallies exact; posteriors predate H1–H3 and the September rows)
**In one paragraph.** The campaign tracked nine claims. In plain words: `z` has no reusable "agent"
variable (C1), and no language supplies one (C8); one vector holds about 460 bits, roughly four
simple facts (C2); grammatical markers can be added and removed by vector arithmetic, but roles
cannot (C3); the length of `z` tracks how long and specific a sentence is, not what it means (C7);
the decoder's inventions can be caught when the vector is obviously corrupt but not when it is subtly
so (C5); a sparse dictionary over `z` finds topics, not sentence structure, and its atoms are
unstable across retrainings (C6); the order code forms in the middle of training (C4); and `z` leaks
enough content to be a retrieval and privacy concern (C9). The table gives each claim's ledger
posterior, the probability the campaign's own evidence-bookkeeping assigns it, and where it stands
after the September follow-ups. Posteriors are from the ledger (row 095, computed
2026-08-08).[^ledger]
| claim | § | statement | posterior | rows | status after the September follow-ups |
|---|---|---|---|---|---|
| **C1** | 4 | No global linear thematic-role axis in pooled `z` | **0.993** (clamped) | 15 | **Hardened.** Scope now explicit: unconditioned readers, additive power certificate. Order *is* readable per queried noun [E02, T2]; role access on held-out predicates is unestablished either way (§4.5). |
| **C8** | 4 | No universal crosslingual role code | **0.484** | 2 | Hardened toward the null; the two contra rows are refuted (§4.4). A 24-item multilingual pilot shows role and number survive direct FR→ZH translation, which bears on decodability, not on the probe null. |
| **C2** | 6 | ~460-bit decoder-extractable budget, ~4-proposition knee | **0.886** | 8 | T3. The number is a decoder-extractable bound, not a mutual-information estimate; the retrieval-cosine "floor" is not a capacity bound (§6.2). |
| **C3** | 7 | Closed-class operators are linear, composable, causally steerable | **0.955** | 10 | Hardened for install rate (T2); **specificity sub-claim withdrawn** [A5, T0]; vertical on natural text is an undefined estimand; scope is local, dose-bounded editing, not a global algebra. |
| **C7** | 7, 8 | ‖z‖ encodes length/specificity, not thematic semantics | **0.690** | 4 | T3, with a sharper statement: halving the norm corrupts one field (location) that a linear reader still recovers, and restoring the norm repairs it (§8.3). |
| **C5** | 8 | Fabrication is off-manifold and gate-catchable | **0.436** | 2 | **Split**: gross off-manifold fabrication is gate-catchable (supported); near-manifold interpolation fabrication is not (refuted); the historical 41–49% flip rate is a floor at zero accepted coverage, not a rate (§8.2). |
| **C6** | 9 | SAE atoms semantic at the frequent core, seed-idiosyncratic tail | **0.620** | 5 | T3. Invariant masks are count- and polarity-specific and *negative* on role; seed instability is the literature's expected outcome; the inference rule was batch-dependent [S1]. |
| **C4** | 10 | Order/role code forms mid-training (substrate-specific ontogeny) | **0.568** | 6 | T3, ladder-only; three ontogenies, not one. |
| **C9** | 11 | `z` leaks propositional / sensitive content (RAG, canaries) | **0.810** | 4 | T3. The count result now replicates on a second inversion system (GTR/vec2text 60/64); the role result does not (40/64) [G1, T3]. |
[^ledger]: The ledger has not been rerun since 2026-08-08. Two entries are therefore stale in a
known direction: C8's 0.484 was computed while rows 061/062 were live contradicting evidence,
and both were refuted by H1; C1 is already at its clamp, so demoting its contra rows cannot move
the number. Rerunning `bayesian_ledger.py` with H1–H3 and the September rows as support, and
061/062 demoted, remains open (§15).
Across the 100 rows: 39 signal, 31 twist (a real result contradicting the prereg), 28 null, 2
blocked; by lane, 23 hardened, 7 shipped artifacts, 65 plain T3, 2 refuted, 2 blocked, 1 partial.
Sixty-three rows flagged a follow-up worth funding; the September work executed, in whole or part,
roughly a dozen of them (Appendix B).
---
## 4. Results: binding ✅ DONE ◆ HIGH (null and retraction H1-hardened with five seeds; block A rows individually thin; §4.5 is MED)
**Blocks A (001–010, can binding be induced?), B (011–020, where does role information live?), F
(051–060, beyond SONAR), G (061–065, multilingual binding), with the September reader studies.**
**In one paragraph.** We asked whether a probe can read "agent or patient?" off `z` in a way that
carries over to sentence structures and words it never saw. It cannot: it scores at chance in every
version of the test we built, at every layer of the encoder, and we know the test had the power to
detect a real signal because it detects a fake one we planted. We then tried nineteen ways of
*training* a model to have such a variable, and none worked at SONAR's vocabulary size. What `z` does
carry, strongly, is surface word order, tangled with the specific words. One apparent exception in
Japanese and German turned out to be an artefact and is retracted. And in September we found that
order *is* readable from `z` if the reader is told which noun to ask about, which is what "tangled with
the words" predicts.
### 4.1 The null, and what it certifies
**The result.** In the primary cell of the battery (train on one construction, test on another, with
the nouns held out too) the probe scores at chance for every kind of readout and at every depth of the
encoder. The consolidation breaker (H1) re-implemented the battery independently, on real SONAR, with
five seeds and 1000 bootstrap resamples, and reproduced the campaign numbers to within ±0.002
[H1-hardened]:
| readout | primary cell (AUC) | within-construction ceiling, same holdout |
|---|---|---|
| linear | **0.509** [0.489, 0.529] | 0.760 |
| MLP | **0.495** [0.471, 0.519] | 0.797 |
**Why chance here is informative.** A probe that scores 0.5 could be failing because there is nothing
to find, or because it is too weak to find it. We rule out the second by planting a fake role signal
into `z`, of a known size, and checking that the same probe in the same test recovers it. It does: a
global role direction of Cohen's d = 1.0 is recovered at **AUC 0.973** (lower confidence bound 0.876),
and the smallest effect the probe could have seen is d = 0.70 [H1-hardened]. So a linear role
direction of moderate size would have been found. The MLP probe is weaker in transfer (it overfits the
training construction; smallest detectable d = 1.5), but the flagship claim is about *linear* binding,
and row 016 separately certified kernel, focal-conditioned MLP, and bilinear probes at planted
amplitudes and found the same null [row 016, T3].
**What the certificate does not cover.** The planted signal is an additive shift. A role code that
worked by *rotating* the representation, or by superposing roles across dimensions, would not be
caught by this check (caution 1); §4.5 is what addresses those.
**What `z` does carry.** In the cell where a probe that reads surface position must get every answer
backwards (same nouns, opposite parity), the linear probe scores **0.266** and the MLP **0.118**, far below chance: both are confidently reading
position, not role. That is the sharpest positive signature in the campaign: `z` knows which noun came
first [H1-hardened].
### 4.2 Nineteen ways to fail to install binding
**The question.** If SONAR did not learn a role variable, could a model be *made* to learn one? Block
A tried on the TAE ladder, changing the training objective, the data, and the architecture. Block F
asked whether other families of sentence representation already have one.
**Changing the objective does not help.** A contrastive objective that explicitly penalized confusing
a sentence with its role-swapped twin drove its own loss from 1.33 to 0.32, and gained nothing on
held-out transfer (0.501, against 0.501 for an untrained model): the model satisfied the objective by
memorizing which specific words go together [row 001]. A supervised head that was trained to answer
"who is the agent?" reached 100% on held-out sentences at every loss weight, while the transfer
battery stayed flat [row 002]. Forcing the decoder to emit an explicit (agent, verb, patient) triple
produced 0.998 agent retrieval on order-swapped sentences and exactly chance transfer [row 003].
**The one exception is instructive.** Holding the objective fixed and widening only the filler
vocabulary, role abstraction *emerges*: with 117 filler words, novel-word retrieval is 0.05; with
1000, it jumps to **0.73** [row 006, T3]. Abstraction is available to this architecture. It is simply
not the cheaper solution at small vocabulary, and §5 explains why.
**Other representations do no better.** Block F tested nine and found no binding in any [rows
051–059, T3]:
| representation | row | result |
|---|---|---|
| 44× scale sweep, GTR-T5 110M → 4.8B | 059 | ceiling 0.601 → 0.591; scale moves nothing |
| instruction conditioning | 053 | +0.004–0.006, indistinguishable from a scrambled-prefix control |
| 1.6B Large Concept Model, which *plans* in SONAR space | 054 | max ceiling shift +0.016 at all three depths |
| Coconut continuous-thought states | 055 | chance |
| text-diffusion latent | 057 | chance in 32 of 32 cells |
| speech encoder | 058 | patterns with text |
| cross-encoder reranker | 056 | **0.468, below chance**; prefers the candidate whose word order matches the query |
| LASER (45M) vs LaBSE (471M) | 052 | LASER lands in SONAR's regime, LaBSE with the small contrastive models |
The only regime shift in the block is driven by the training objective, and it is not binding:
training a model to *generate* translations installs the surface-order code that training it to
*rank* them does not [row 052, T3]. This agrees with a 2026 finding that supervision from synonym and
definition pairs actively degrades relational concepts in sentence encoders (Mohr et al., 2026). The
null is what the objective predicts, not an anomaly.
**The most uncomfortable number.** An LLM judgment panel, asked which of two sentences is closer in
meaning to a target, picks the paraphrase over the role-swap 90 times out of 90. Cosine similarity on
`z` picks the role-swap 30 times out of 30, and within a topic the correlation between `z`-cosine and
judged similarity is **negative (−0.218)** [row 060, T3, LLM-proxy only; the human study was designed
and never run].
### 4.3 Where role information lives, and does not
**Inside the encoder.** Role-consistent attention *computation* exists: 42 of the encoder's 384
attention heads route from agent to predicate consistently across all constructions, significant after
Bonferroni correction and replicated on held-out data [row 012, T3]. But switching all 42 off changes
the decoder's role fidelity by +0.001; the circuit is causally inert [row 013, T3]. Patching a token's
state from the role-swapped sentence flips the decode 95% of the time through layer 22, and 0% at
layer 24: the encoder commits late and all at once [row 014, T3]. The order code is local to the
filler tokens at every depth (0.86–0.96) [row 017, T3], and a trained attention pooler on frozen
states recovers nothing at any depth: the null is a property of the token *states*, not of averaging
as a readout [row 015, T3].
**What the token states are made of.** Two September experiments asked how well simple models
describe the final-layer token states. A nested ridge regression predicts each state from (i) its own
token, (ii) its two neighbours on each side, (iii) the unordered bag of tokens in the sentence, and
(iv) sampled bigram interactions. On trained SONAR, the token's own identity explains R² 0.370 of the
variance, the neighbours add 0.019, the bag 0.002, the interactions 0.002, and 0.606 is unexplained. On
a randomly initialized twin of the encoder, the same features explain 0.938 [E01, T2]. Nor is the
unexplained part a sentence-wide "broadcast": adding the average of the other tokens' states lifts R²
only from 0.474 to 0.509, whereas the random twin goes from 0.624 to 0.972 [E03, T2]. Training
*removes* the simple structure. About half of what a trained token state contains is not captured by
any local or bag-of-words model we could fit at 162k tokens.
### 4.4 The multilingual exception, retracted
**What was claimed.** Rows 061/062 reported the campaign's first order-invariant role binding, in
languages that mark roles with case rather than word order: German 0.668 and Japanese 0.658 on the
strict primary cell, Japanese replicating at 0.696 on a fresh set of nouns. They were the only two rows
contradicting C1 in the ledger, and they did not survive.[^g]
**Why it was wrong.** Each item's probe label had been fixed by the alphabetical order of the focal
noun's English gloss, a convention the Japanese encoder cannot see. The H1 breaker ran one faithful
pipeline on a **third, disjoint set of nouns** and obtained **0.451** [0.383, 0.522], chance, while the
same pipeline gave 0.696 on 062's nouns and 0.79 on 061's. A swing of 0.45 / 0.70 / 0.79 across three
noun sets is incompatible with a stable role code, and it is unlikely to be a power problem: a planted d = 1.0
signal on the third set is recovered at 0.71, above chance though below the 0.9 bar the benchmark
uses, and the third set has only 8 test nouns. The mechanism tests refuted their own pre-registration.
Swapping the Japanese case particles が↔を, which should invert a genuine case-reader, does not invert
the probe (0.78, not ~0.30), and a representation built only from noun+particle chunks does not recover
the signal (0.36). The signal was a whole-sentence, noun-set-specific correlation with the arbitrary
label [H1-hardened, RETRACTED]. What is stable across all three noun sets is surface-order reading
(anti-transfer 0.10–0.25 in the within-vocabulary cell).
[^g]: The campaign-time verdicts for rows 061 and 062 were "signal" and "twist" respectively, and
both were starred as headline results at harvest. They are recorded as refuted on the status
board and in Appendix A. The general principle is that cross-lexical case→role positives are a
false-positive channel from an encoder-invisible label, and that cell is retired as a binding
assay.
**What stands in block G.** Language in SONAR is an *interface* property: a decoder token plus a
small, removable mean offset. Steering `z` along the language direction flips the output language 0%
of the time [row 064, T3]. A 24-item French/English/Chinese pilot run in September is consistent with
that. Decoding French directly into Chinese through one vector preserves role reversal and number
contrasts, and routing through English *text* first is a real transformation (cosine 0.92 to the
source vector), not an identity [multilingual pilot]. That is a decodability observation on
hand-authored items; it says nothing about the probe null.
### 4.5 Query-conditioned readers: order is stored per noun ◆ MED (order T2, single noun set; role T3, predicate-general access open)
**The question.** The unconditioned null leaves two possibilities open. Either role information is
*absent* from `z`, or it is present but *addressed*: stored in a way that a reader can only access if
it knows which noun to ask about. The September reader studies settle this for word order and leave it
open for role.
**Order.** The reader is bilinear in a query and the latent, `s = qᵀUVᵀz`, where `q` is the embedding
of the noun being asked about (Figure D8). It was trained on active and passive sentences only, with the training
and test nouns disjoint (65 / 22) and the verbs disjoint (18 / 6), then tested on constructions it had
never seen. On the leak-free stratum of common nouns (74% of items) it reads surface order on cleft
sentences at AUC **0.990**, on object-relatives at 0.746–0.922, and on nominalizations at 0.802–0.869,
the ranges depending on which families were in training [E02, T2]. A randomly initialized SONAR reads
0.504 / 0.478 / 0.493, so the code is learned. It is learned late: cleft transfer is 0.822 at layer 20,
0.987 at layer 22 and 0.994 at the final normed layer. The order direction is orthogonal to the
construction-parity direction (cosine < 0.032 for all 87 nouns). Readers of the same bilinear form
asked about *role* rather than order do not transfer (object-relative 0.169).
The independent breaker found the result stands, with scope limits that travel with the number. The
pooled headline numbers (0.994 / 0.946 / 0.911) are inflated by a proper-name and determiner-case leak
and should not be quoted. The order-blind static-bag floor is 0.50 / 0.72 / 0.55 rather than 0.5.
Every "unseen noun" figure rests on the same 22 test nouns, and the bootstrap resamples propositions,
not nouns, so the stated ±0.03–0.04 intervals understate noun-sampling variance [E02 breaker].
Figure D8. The unconditioned probe against the query-conditioned reader. Order is stored per noun.
**What this means for the null.** Unconditioned probes fail because order is stored *per queried
noun*: there is a readable answer to "did *doctor* come first?" but no single direction that means
"the agent came first". The binding deficit is on construction parity, which is readable for cleft
(1.0) and nominalization (0.95) but not object-relatives (0.56). With only two patient-first families
in the battery, a parity probe is effectively a family classifier, so whether parity is readable in
general cannot yet be tested cleanly.
**Role.** Here the picture is genuinely two-sided. A query-conditioned MLP on `z` reads
predicate-conditioned roles on held-out nouns at AUC **0.993** (conjoined clauses) and **0.982**
(separate clauses), with wrong-query and missing-`z` controls at exactly 0.500. But it does so on a
closed bank of four verbs shared between training and test [A2, T3]. On the same multi-event vectors,
the native decoder recovers both predicate-conditioned roles correctly on 242 of 320 sources, while a
candidate-likelihood reader gets both right on 5 of the 122 opposite-role vectors the decoder accepts
(4.1%) [binding/multievent, T3]. That decoder count rests on only 20 base propositions rendered 16 ways
each. We also tried to build a reader that answers "who is the agent of *this* verb?" for verbs it
never saw. That study never reached its final test. On its practice set it read one event per
sentence almost perfectly (0.998) but could not assign roles when a sentence contained two events
(0.43, below the 0.60 bar we had set), and an added consistency loss made it refuse to answer rather
than answer better (0.08). On the one held-out-verb panel that was scored, single events held at
0.99 and two-event assignment fell to about 0.10; the three training seeds disagreed widely
(0.17–0.59) and no intervals were computed [binding/predicate_holdout, development only].[^pred]
Predicate-general role access is therefore **unestablished in either direction**. The null is a statement about a class of reader and a way of asking, not about
what `z` contains, and the reader that would settle it has not been built.
**Is the readable order code the one the decoder uses?** No: **readable is not decoder-special.**
Pushing `z` along the direction the order reader uses does not make the decoder swap the nouns (0 of
the edits at the dose that fools the reader; 7% at double dose). Replacing `z` outright with the
swapped sentence's own vector, which we call the *oracle edit*, swaps them 98% of the time, so the
decoder can be moved; this direction just does not move it [E04, T0 as registered]. Most of that
first result is an instrument artefact (caution 5). A re-registered follow-up on fresh stimuli (160
propositions, 40 new nouns, 8 new verbs, 960 edits) then asked whether the small subspace the reader
uses (numerical rank 9) is special to the decoder [E04b, T1]. It is not. Swapping an item's coordinates in that subspace
flips the decoded order in 0.140 of edits; energy-matched structured subspaces, such as the top
principal components of the training differences, flip 0.124 and 0.110 (paired difference −0.001
[−0.04, 0.04]). Removing the reader's share from the full swap leaves 0.626 swapping, and removing the
same energy from the control subspaces leaves 0.636 and 0.677. What the decoder responds to is how much
of the AB→BA difference is moved inside any *structured*, data-aligned subspace: the same energy spread
isotropically over about 334 random dimensions flips only 0.024, and the response is step-like (3% of
the squared difference gives 0, a third gives 0.14, all of it gives 0.995).[^e04] There is no low-dimensional order handle in `z`;
the decoder reads order from a distributed difference.
[^pred]: The 11 September morning version of this paper attributed the 0.998 / 0.426 / 0.077
figures to held-out predicates. The sample-size audit traced them to the seen-predicate
training-noun control split; the held-out validation panel gives 0.993 and ~0.10. The
registered eight-family final panel was never encoded.
[^e04]: The 11 September morning version of this paper reported, descriptively, that swapping
coordinates in the reader's top-64 subspace swapped the decode in 0.78 of edits against 0.00 for
random 64-dimensional subspaces, and flagged it as a candidate causal order subspace. The random
controls in that comparison were not energy-matched; with energy matching on fresh stimuli the
reader subspace does no better than other structured subspaces, and the claim is superseded.
---
## 5. Theory: why binding fails ✅ DONE ◆ MED (mechanism exact for static pooling; real-encoder checks on 40 multisets without intervals; E01/E03 T2 and E04b T1 well powered)
**Rows 096, 097 (block L), consolidated in H3.**
**In one paragraph.** Averaging is the culprit, and it is not a bug. If you average a set of
context-free word vectors, the order of the words cannot matter: the average of the same three vectors
is the same vector (Figure D5, top). SONAR's encoder gives each token a state that depends on its
neighbours, so a little order does survive the average, but only in the form "this particular word,
in this position, among these particular words". That is exactly what we see: order is readable if
you ask about a specific noun, and not as one global direction. The theory below makes that precise,
predicts when a small model will learn to bind and when it will not, and was checked against the real
encoder.
Figure D5. Why averaging loses order, and how a little leaks back, tangled with the words.
### 5.1 The toy account
Row 096 builds a minimal mean-pooled autoencoder over (agent, patient) pairs drawn from a vocabulary
of V filler words. Averaging destroys order except for a weak residual channel `g = β(e_a − e_p)`, the
difference between the agent's and patient's embeddings scaled by a small β. Write T for the number
of training examples and K for the decoder's capacity. The model can solve its
task with either of two codes. **Bag lookup** memorizes the role assignment for each specific pair of
words; it is cheap but cannot generalize to a word it has not seen. **Role reading** learns to read `g`
regardless of which words fill it; it costs a constant amount and generalizes. The number of distinct
word pairs is about V²/2, and the memorization budget is min(T, K). Writing γ(β) for the usable signal in the order channel at
strength β, the two codes cross over at
> **V\* ≈ √(2 · min(T,K) / γ(β))**
Below V\*, memorizing pairs is the cheaper way to lower the loss, which predicts the near-zero
novel-word accuracy of rows 001 and 006 at 30 and 117 fillers. Above V\*, only the role reader lowers
the loss, which predicts 006's jump at 1000 fillers. The numerical model **falsified the row's own
predicted cause**: the transition is set not by the amount of data or the model's capacity but by the
signal-to-noise ratio of the surviving order channel [row 096, T3, Brier 0.136].
Row 097 supplies the formal half. Averaging over positions is the **Reynolds operator** for the
group of all permutations, the linear map that projects any function of a sequence onto the part that
does not depend on order. On context-free embeddings it destroys order *exactly*. On contextual states
order survives through one channel only. That channel is the bag of words re-weighted by how much
attention each token received. The re-weighting depends on position only through the encoder's
relative-position bias. Split that bias into a symmetric part and an antisymmetric part (the part
that flips sign when two tokens swap places); the antisymmetric part is the only thing that says who
came before whom, and it is precisely 096's β residual [row 097, T3, Brier 0.121].
The Reynolds identification is not ours. All linear permutation-invariant readouts, sum and mean
pooling named explicitly, factor through the Reynolds operator and erase symmetry-aware components
regardless of how expressive the encoder is (Talhi et al., 2026), and Reynolds operators have been
built as network layers (Sannai et al., 2021). That theorem covers linear readouts over a linear group
action. SONAR's pool is linear only after a nonlinear, position-aware pre-map, so encoder-plus-pool
is a learned **Janossy pooling of arity greater than one** (Murphy et al., 2019) rather than a Deep
Sets model.[^deepsets] Every property we observe follows: order is present, local, content-entangled
*by construction* because a k-ary feature is a function of the identities of its k tokens, and it does
not compose globally. Two consequences are testable. Injectively representing a 20-token multiset in
a 1024-dimensional token space would need on the order of 41,000 latent dimensions (Wagstaff et al.,
2019), but that is a sufficiency bound whose violation predicts a cliff, not the smooth knee of §6,
which favours the rate-distortion account. And no piecewise-linear Janossy pooling is injective
except on multiplicity-free multisets (Reshef and Dym, 2025), predicting that collisions concentrate
on sentences with repeated tokens. That order survives in a mean-pooled embedding at all is already
published (a probe separates shuffled from unshuffled SBERT inputs at ~89%; Chavan et al., 2023),
which is why we claim the mechanism and the locality dissociation, not the discovery.
[^deepsets]: The 6 September version described SONAR as a Deep Sets model. It is one only on the
static path, where the mean is a genuine group projection.
### 5.2 Validation on the real 24-layer encoder
H3 tested seven predictions on real SONAR contextual states: five hold, one missed quantitatively,
and one was later retracted as a measurement of role [H3-hardened]. These checks used 40 token
multisets with 200 permutations each and reported no intervals; the well-powered part of the
anatomy is E01, E03 and E04b (§4.3, §4.5).
- **Static pooling is order-blind to precision.** Permuting the tokens of a sentence and averaging
context-free embeddings changes the result by **5.1·10⁻⁷** (we call the size of that change the
*shuffle dispersion*), and a probe for order within such a bag scores at chance.
- **The contextual pool smuggles order, per instance but not globally.** Real `z` recovers order
**within a fixed set of tokens at AUC 0.997** (shuffle dispersion 0.565), while a probe that must
read order across *different* sentences scores **0.523**, and the cross-proposition binding-order
probe scores **0.507 at every layer**. Order is stored *entangled with content*: the below-chance parity
signature of §4.1, derived from first principles, and the same shape as E02's per-noun order code.
- **The reversal-odd component is noun-dependent and order-sensitive.** The reversal-odd component
of `z` is the half that flips sign when the two arguments are swapped, `(z(a,p) − z(p,a))/2`. Role
transfer across constructions is **below chance (0.26–0.35)** on it, a surface reader
anti-transferring on a parity flip. The consolidation also reported role as *recoverable* from
this component at 0.76–0.86; that reading is withdrawn.[^odd]
- **β accelerates in L20→L23 and is amplified ~6× by the final LayerNorm**, matching the layer-22
commitment and layer-24 cliff of row 014, and E02's L20 → L22 emergence of the order code.
[^odd]: The 6 September version listed "role is recoverable from the reversal-antisymmetric
component of matched pairs at 0.85 (cleft 0.856)" as decodability evidence in §1 and in this
section. The probe used alphabetical-focal labels under proposition-grouped cross-validation in
which nouns recur across folds; under that exact recipe a random per-noun Gaussian fingerprint
scores 0.925, above the real component at 0.83–0.85, and on noun-disjoint splits the real
component reads 0.51 [E01 breaker, check 5]. The licensed statement is that the odd component is
noun-dependent and order-sensitive, which content-entanglement already implies. The decoder's
0.998 and the retrieval results are unaffected. With alphabetical labels a noun-general test is
ill-posed, which is what motivated the query-conditioned design of §4.5. [T0 as decodability
evidence]
**The quantitative miss, and the key refinement.** The toy predicts an order subspace of a handful of
dimensions; on real SONAR it takes **~23 principal components to reach 90% of the variance** (the top
3 give only 30%). Twenty-four nonlinear layers spread the channel wider, and E02's reader subspace is
consistent with that (effective rank ≈ 7 for the order direction), though no low-dimensional subspace
is causally special to the decoder (§4.5). More importantly, β is small before the final LayerNorm
(~0.01–0.05) but amplified ~6× to ~0.29 by it, so "weak β" means weak *globally readable
signal-to-noise*, **not** weak norm: **content-entanglement, not amplitude, is what defeats a linear
role reader** [H3-hardened]. Codex review found one real latent bug, a missing zero-variance guard in
`order_rank`, fixed in the original campaign script.
This account also predicts the binding null from outside. Every positive result on entity binding in
language models depends on a per-token-position address (Feng and Steinhardt, 2024). Propositional
probes recover entity-role bindings at 0.97 exact match, but both of their stages are token-indexed,
so after pooling the method is undefined rather than merely harder (Feng et al., 2024). Mean pooling
annihilates precisely the address, and the content-keyed fallback that takes over under load in
autoregressive models (Gur-Arieh et al., 2026) is exactly the content-entangled channel that survives
here.
---
## 6. Results: capacity 🟡 PARTIAL ◆ LOW (48–64 items per cell and no intervals on any knee; no breaker; 460 not re-frozen; §6.2 is MED)
**Block H (066–072), the rate-distortion row 098, and the September unbinding branches.**
**In one paragraph.** How much can one vector hold? We measured it by asking the decoder to give back
sentences of increasing length and content, and counting how many bits of sentence-specific
information survive, where a *bit* is one yes/no question's worth of information about which
sentence this was. The answer is about 460 bits, reached at roughly four simple propositions, after
which the decoder starts dropping things (the direct object first). The budget is in bits, not words:
predictable text fits more words than surprising text. A simple rate-distortion model with one measured
rate and one measured capacity predicts where the decoder's quality curve bends, in three different
measurements, with no free parameters. What the number is *not* is a mutual-information estimate, and
an earlier reading of a retrieval-cosine "floor" as a capacity bound has been dismantled.
### 6.1 The budget, in bits
A single `z` carries a bounded, sentence-specific budget of **~460 bits (~0.45 bits per dimension)**,
saturating at roughly **4 propositions** and degrading gracefully past the knee; the budget was
confirmed five independent ways in the block capstone, each on 48–64 items per cell with a single
seed and no interval [rows 066–072, T3].[^bits] The 460 figure is
the saturating ceiling. The *knee* is the length at which the decoder's reconstruction quality starts to bend downward. At
the knee, natural text of three different difficulty tiers carries a constant ~350 bits (352 / 327 /
347, constant to 1.08×), while random, hard-to-predict text hits its knee at *half* the number of
words: the budget is denominated in **bits, not tokens** [row 068, T3].
A typical single FLORES sentence, well below the knee, carries roughly 160–180 bits.
Combining two clauses costs less than the sum of their bits, and past the ceiling the shortfall is
spread evenly across clauses rather than one clause being dropped [row 069, T3]. Digits are not
specially fragile, and the ceiling of about three rare named entities per vector is a capacity limit,
not a fixed number of slots [rows 070, 071, T3]. Pushed past the knee, SONAR deletes the **direct
object first** and keeps time and quantity last, the reverse of a gist-over-detail prior [row 072,
T3]. On the ladder, sentence-specific bits climb 110 → 319 → 404 as the bottleneck grows 16 → 64 →
256 dimensions, saturating at the rank-256 ceiling [row 067, T3]. A **count grammar** used throughout
the September work makes the knee concrete: each sentence states an object, a count, a location and a
tense, so a parser can score each field. One fact decodes cleanly 60 times out of 64, two facts 25 of
64, four facts 0 of 64. Searching more candidate decodes (wider beam search) does not rescue the one- or four-fact cases (59/20/0 at
beam 5, 59/21/0 at beam 10); at two facts the beam-5 difference is −7.8 pp [−18.8, +4.7], unresolved
at this n [count_capability, T3].
[^bits]: The 460-bit ceiling originates in row 066 as a post-hoc quantity that was not frozen in
that row's pre-registration; it has not been re-frozen since. It is a decoder-extractable lower
bound on what one vector transports, not a mutual-information estimate: no distribution-free
high-confidence lower bound on mutual information from N samples can exceed order log N
(McAllester and Stratos, 2020), so no experiment of this size can certify 460 bits as
information in the Shannon sense. The claim also mixes SONAR evidence (066, 068–072) with one
ladder family (067), and has no independent breaker.
**Predicting the knee.** Row 098 closes the block theoretically. With one measured source rate
(r = 4.5 bits per token, how much information a token of this text carries) and one measured
capacity, the length at which the decoder's quality bends should be `L* = C_D / r`. That places all
three measured knees on one rate-distortion curve with no free parameters *given the distortion
measure* [row 098, T3, Brier 0.084]:
| knee | predicted | measured | ratio |
|---|---|---|---|
| sentences of a standard multilingual test set (FLORES), scored by character overlap (chrF) | 1.99 rows | 2.14 rows | 0.93 |
| tokens | 79.6 | 70–83 | 1.10 |
| content words recalled at ≥ 0.85 (an earlier, stricter measurement) | 42.2 tokens | 36.4 tokens | 1.16 |
The gap between 460/4.5 ≈ 100 tokens and the measured ~65–83 is the distortion discount
δ = C_D/C = 0.78. Choosing a surface-overlap distortion is itself the assumption that produces
surface-tracking knees; a semantic distortion would give a different curve over the same latent.
**Two external anchors.** A 2026 bound on embedding-based retrieval gives at most (1 + 1/γ)^d
returnable subsets, which for d = 1024 at unit margin is 1,024 bits (Weller et al., 2026); our ~460 is
conservative against it, not inflated. And single-vector capacity is a property of the
encoder-decoder *pair*: two models of equal dimension differ threefold in how many tokens one vector
holds (Kuratov et al., 2025). The knee here is a measurement of SONAR's decoder, and swapping the
decoder should move it. One wrinkle stands unresolved: bits per sentence are roughly flat across
languages (Japanese 163 vs English 182) even though Japanese round-trip *fidelity* is far worse
(~35 chrF vs 89.7) [rows 063, 066, T3].
### 6.2 What the retrieval floor is not ◆ MED (512–1,024 chains with clustered bootstraps)
If you average the vectors of k sentences into one, can a trained reader pull the j-th sentence back
out? That is the **unbinding** task (Figure D9); k is the *depth*, j the *slot*, and the reader is
*frozen*, that is, trained once and not adapted to each test. Earlier work in this program
read the decay of retrieval quality with depth, from ~0.72 at two items to a plateau near 0.4 by depth
six, as a capacity floor matching vector-symbolic theory. The September unbinding branches dismantle
that reading.[^floor]
Figure D9. The unbinding task. The decay with depth is a property of the reader, not of the vector.
**Most of the floor is baseline similarity.** At matched training size the centred all-slot cosine
falls from 0.663 at depth 2 to 0.134 at depth 16. But at depth 16 a reader that always predicts the
*mean* vector already scores a raw cosine of 0.314, which is 92% of the ridge reader's 0.341. SONAR
vectors all point roughly the same way (the space is anisotropic), so raw cosine is high for any
guess [U1, T3]. Centring the same frozen predictions lifts combined retrieval from 2.84% to 26.03% with
no retraining [U1B]. Training size 256 → 2048 adds +0.071. A simple additive model of pooling predicts retrieval of 0.40 at
depth 16; SONAR's real pooling gives 0.01, so the additive picture is wrong [U4].
**The position effect is a property of the reader, not of the states.** A single global reader
retrieves 94.1% at slot 1 and 1.3% at slot 16 [U2]. Pooling the same token states over the oracle
span retrieves near ceiling at every slot (first-four minus last-four = +0.010). A kernel reader does
no better than linear (−0.0002) [U3]. On a closed grammar, position-supervised heads recover every
field at every position (2025/2048 with routing, 4.9% with a direct query) [A3, T3]. No
information-capacity bound follows from any of these curves.
[^floor]: The historical depth-16 series had a provenance problem: 231 training windows from 16
documents, 35 test windows from 3 documents, 11 of which shared paragraphs and exact strings
with training, 196 training examples against 1024 dimensions, and a string-slicing bug that
reported only 7 of 16 fitted targets at depth 16 [HISTORICAL_SOURCE]. The plateau it seemed to
show is also not predicted by any vector-symbolic mechanism, whose superposition decay is
monotone; the correlated-codebook replacement proposed in the literature review,
c(N) = √((1 + (N−1)μ)/N), fails to produce a 0.4 asymptote either (μ = 0.05 gives 0.224).
Neither asymptote is validated.
---
## 7. Results: operators and steering ✅ DONE ◆ MED (algebra H2 code-reviewed, but on 24–56 items per operator without intervals; specificity withdrawn)
**Blocks D (031–040, geometry) and J (078–081, composition operators), consolidated in H2 and audited
in September (O1, A5).**
**In one paragraph.** Some things *can* be done to `z` with simple vector arithmetic. Add a fixed
"negation" vector and the decoder says "does not"; add a "past tense" vector and it says "visited"; add
both and it says "did not visit". These offsets are linear, they compose, they can be undone, and they
work on natural sentences at a sensible dose. The one thing no offset does is swap who did what to
whom: the "voice" offset produces a passive *sentence* with the roles unchanged (Figure D6). That is the
paper's internal consistency check: the edit that fails is exactly the variable §4 could not find.
Under audit, one claim from this block, that the edits change nothing else in the sentence, did not
hold, and is withdrawn.
Figure D6. What a vector offset can and cannot do to z.
### 7.1 The algebra and its wall
Closed-class morphosyntactic markers, the small set of grammatical words and endings such as *not*,
*-ed*, *above*, act on `z` as approximately linear, invertible offsets that **compose additively and in
any order**. Tense, number, negation, and question are clean linear analogy operators [row 033, T3].
Five relational marker families (before/after, the *vertical* above/below, in/out, near/far,
bigger/smaller) all linearize at **success 1.00** [row 078, T3]. Adding two single-operator directions
gives the same result as fitting the double transform directly (**mean additivity 0.982,
compose-minus-direct decode gap exactly 0.000**), across families and in either order, on 24 held-out items per operator pair with a single judge
[row 079, T3].
An MLP readout never beats the linear one, which is the evidence that these really are linear, and the
offsets survive removing the top principal components of a `z` that is nearly isotropic anyway [row
037, T3]. They are **causal**: adding α·v steers negation, tense, number, and a spatial marker from 0
to ~1.0 success by α = 1 on templated stimuli, the edit can be undone exactly, and a random push of the
same size does nothing [row 080, T3].
**The wall.** **Argument reversal does not linearize: 0/74 argument swaps in the operator zoo, 0/40 in
the analogy battery.** The voice operator produces correct passive *form* with the arguments unswapped
[rows 033, 078, T3]. A linear offset can substitute a word; it cannot reverse who-relates-to-whom.
The one transform the algebra cannot perform is exactly what pooled `z` cannot linearly represent
(§4). An independent read-side precedent exists: across formal meaning manipulations, role swap is on
average the hardest for embedding models to detect (Sentence Smith, 2025).
### 7.2 Natural text and specificity ◆ MED (audited; specificity withdrawn; vertical undefined)
**Code review.** Codex review certified the steering core correct (offsets fitted on training data
only, clean α arithmetic, no leakage between operators), so the 078/079/080 causal claims stand. It
confirmed that composition must be certified on the *decoded text*, never on cosine similarity: row
079's cosine-additivity check had passed for the wrong reason. It also found one real latent bug. The
080 analysis counted garbled decodes as successes, harmless on templated data but inflating success on
natural text. With a fluency gate, **negation's apparent 0.90 success at α = 2 is fabrication; the
honest peak is 0.70 at α = 1** [H2-hardened].
**Natural sentences.** Applying the same operators to held-out *natural* sentences (entity-rich, 6–26
words) over a wider range of doses breaks the templated result's "0% garble": as 080 itself predicted,
garble appears at high dose and is dose-localized (runaway "…not not not…"). The September audit then
re-judged the same 104 natural bases under a rubric that reads the whole proposition rather than two
attributes, with Wilson intervals, and the specificity claim did not survive [A5, O1] (two independent joins of
the same judgments differ by one negation row, hence the range):[^spec]
| operator, α = 1 | target marker installed | full-scope success | collateral propositions changed |
|---|---|---|---|
| negation | 26/40 = 0.65 [0.50, 0.78] | 14–15/39 = **0.36–0.38** [0.25, 0.54] | 19/40 = 0.47 |
| tense | 31/40 = 0.78 [0.62, 0.88] | 16/38 = **0.42** [0.28, 0.58] | 7/40 = 0.17, plus 17 uncertain |
| vertical | 0/24 sources applicable | undefined | 8/24 = 0.33 |
In words: at the standard dose the operator usually installs its marker, but it puts the negation or
the tense in the right place only about 40% of the time, and it often changes something else in the
sentence. Operator safety is not uniform. **Tense is the most robust to over-steering** (fluent 0.85 at
α = 2, breaks at α = 3) and **negation is safe only in a narrow window around α ≈ 1**. The vertical
operator's natural-text evaluation had no applicable items: none of the 24 sources contains a physical
above-relation (all 16 uses of "above" are idiom or discourse), and the edit direction was fixed
above→below while the scorer expected a per-item pole flip.[^vert] Operators are near-identical across
seeds on SONAR's fixed basis (cosine 0.98–0.995), the honest contrast to the per-seed-trained ladder
[row 075, T3; H2-hardened].
[^spec]: The 6 September version stated "specificity stays perfect (0.00 off-target flip at α = 1)"
and "zero off-target collateral". That figure was produced by a two-field coarse-attribute
rubric, not by the data: applying H2's own rule to the same rows still gives 0/35, 0/35 and 0/18.
The specificity sub-claim of C3 is withdrawn [T0]; the install-rate sub-claim stays at T2.
[^vert]: The 6 September version reported "vertical does not transfer to natural polysemous
above/below (success 0.04): it is frame-bound". That number was measured on a task that was
largely undefined, and is replaced by "untested on applicable natural text" [T0, undefined
estimand]. The matched-frame editing study designed to replace it never reached its efficacy
stage (Appendix B, O2). Its 512-string reconstruction bank decoded at 82.2% under the
pre-registered beam-5 primary (84.6% at beam 1), with every parsed error on an object or anchor
noun and none on the relation itself.
**Two warnings from outside the program.** Steering vectors in language models are anti-steerable on
roughly half of inputs across forty datasets, and steerability is largely a property of the dataset
(Tan et al., 2024; Braun et al., 2025). An independent group found English-to-Spanish direction
vectors in SONAR that are clean, additive and **decoder-inert** (Hawthorne and Nellessen, 2025).
In-distribution steering success demonstrates local linearity, which every smooth map has. What we
claim is a local, dose-bounded, decode-certified editing algebra over closed-class markers with a
characterized wall, not a global algebra.
### 7.3 Monitoring the edit
How do you know an edit has pushed `z` somewhere the decoder cannot handle? The flag that works on
natural text is the **round-trip re-encode cosine** `rt_cos`: decode the edited vector, re-encode the
result, and compare. It falls from 0.99 to 0.29 as the dose goes from α = 0 to 6, and separates fluent
from garbled output at **AUC 0.845**. The nearest-neighbour-to-a-bank metric used in rows 083/087
**does not transfer** to diverse natural text (AUC 0.779, flat), a real correction to the safety
block's monitor choice. The deliverable is `latent_rewrite.py`, a monitored rewrite primitive with a
norm-linter pre-flight and an `rt_cos` flag that *withholds* off-manifold outputs. It refuses 91.7% of
high-α garble, accepts 100% of fluent α = 1 rewrites, and refuses argument swap by policy on the
row-078 wall [H2-hardened].
### 7.4 Geometry, briefly ◆ MED (two T3 rows)
Two block-D negatives are load-bearing. **Norm is not meaning** [C7]: the length of `z` tracks
sentence length and specificity (partial r 0.41 with perplexity) but scaling `z` by ±30% changes the
decode not at all [row 036, T3]. The antipode −z is not an opposite: it is indistinguishable from a
random vector of the same length and orthogonal to the negation operator [row 038, T3]. The decoder is
nevertheless norm-sensitive at larger scalings (§8.3). Anisotropy in `z` is best explained by
positional outlier dimensions, which are known to matter specifically for mean-pooled sentence
embeddings (Luo et al., 2021), rather than by the tied-softmax frequency mechanism that a bottleneck
encoder lacks.
---
## 8. Results: fabrication and auditing 🟡 PARTIAL ◆ MED (near-manifold half of C5 open; base decoder only)
**Block C (021–030), rows 086/087 from block K, and the September audit branches.**
**In one paragraph.** When `z` does not pin down a sentence, the decoder fills the gap with something
plausible. We call that *fabrication*, and it matters because anyone reading `z` through the decoder
(an auditor, a monitor, a person) will not be able to tell invented content from transmitted content.
On clean sentences fabrication is rare (3.6%). The bad news is that the filters we built to catch it
(also called gates below) work only on gross corruption. A vector halfway between two real sentence
vectors looks perfectly normal to every filter and decodes fluently, yet invents content 92.5% of the
time (Figure D7). Two earlier headlines are corrected in §8.2 and §8.3.
Figure D7. Gates catch vectors that leave the manifold. The dangerous fabrications never leave it.
### 8.1 What the decoder invents
On *clean* input the fabrication rate is **3.6%**, an order of magnitude below the program's 20–30%
working headline on stressed inputs. It is dominated by named entities, and **65% of it is invisible
to a cosine gate**, because the invented sentence is close to the true one in `z`-space [rows 021,
029, T3]. The mechanism is the decoder's **language-model prior showing through** where `z` fails to
rescue a token the prior finds unlikely [row 024, T3]. The resulting high-entropy spans give a
detector at AUC 0.87 that is redundant with cosine [row 022, T3]. The only published measurement of a
decoder prior overriding its conditioning is an 18.84% average faithfulness drop (Gong et al., 2026),
and the remedy there, contrasting the decoder's logits with and without the conditioning vector, is
directly available to a text autoencoder.
**The auditing results are mostly negative.** The only cosine gate that could be deployed, one that
checks the decode is self-consistent using only the latent, accepts a fluent, completely different
reconstruction ~90% of the time at every threshold [row 025, T3]. An auditor that decodes and then
checks the result with a natural-language-inference model ranks fabrications **below chance (AUC
0.478)**, sharing cosine's blind spot [row 029, T3]. The decoder's own token confidence rank-orders
fidelity (AUC 0.738) but is systematically under-confident [row 030, T3].
### 8.2 C5, split
**The off-manifold half holds.** A density gate, which asks whether a vector sits where real sentence
vectors sit, is a near-perfect fail-closed decoder against gross off-manifold fabrication (reject-ROC
1.000) [row 087, T3]. An ensemble of six detectors catches all seven injected corruption types at AUC
1.000 when allowed to insist on a tight re-encode, which also rejects 82% of benign paraphrases [row
086, T3]. **The near-manifold half fails**: **the midpoint of two real embeddings sits at clean-level
density and fabricates 92.5% of the time, and the gate is blind to it at AUC 0.508** [row 087, T3].
Row 023 adds that there is no "fail-open band" off-manifold where fluent lies live; the only potent
direction is interpolation toward another real embedding [row 023, T3].
**The 41–49% flip rate, re-examined.** The program's earlier headline for decode-then-parse auditing
was that it is safe around cosine 0.94 and fails open around 0.6, with 41–49% of readings flipping
silently. It was re-examined twice in September and does not stand as a rate.[^flip] The historical
audit took a different model's sentence vectors, mapped them into SONAR space with a learned linear
regression (a *surrogate* vector), decoded, and parsed the result. An exact replication reproduced
every legacy number. It then showed that the flips were counted only among the few outputs that
parsed at all, *before* the acceptance filter, and that the filter accepted none of those outputs;
so the rate of accepted wrong readings is undefined, because nothing was accepted. Under a strict criterion that
requires both entities to be present, the valid reads collapse from 78 to 5 and from 82 to 2, and a
blind review of the labelled flips found only 5 of 33 were clear reversals retaining both entities
[audit replication, T3].
A fresh pre-registered degradation study with a frozen gate found the opposite failure. The gate
accepts a decode if it re-encodes within cosine 0.85 of the perturbed latent. At source cosine 0.94,
127 of 128 decodes were correct and all were accepted. At 0.8, 106 of 128 were correct and **none**
were accepted. At 0.6, 34 of 128 were correct and none accepted. Every one of 64 counterfactual latents
was accepted, because a faithful decode of a supplied alternate latent is self-consistent by
construction [R06/R09, T3]. The gate fails *closed* on this perturbation family and cannot certify
original-source integrity on any. Self-consistency and source fidelity are distinct endpoints.
[^flip]: The 41–49% figures (38/78 ridge, 34/82 small-ridge) replicate exactly. What changed is
their denominator and their interpretation: they are direction-flip fractions among the few
parser-valid outputs of a surrogate pipeline, before a gate that accepted none of them. The
dominant result of that audit is loss of auditable content, not a measured reversal rate. The
bootstrap intervals originally embedded in the degradation study's result files coupled
unrelated frames and have been superseded by stratified intervals (8 action frames, 8 number
frames, 10,000 replicates); point estimates were unchanged.
**Three further results locate the blind spot.** Random noise at matched cosine 0.99 and matched norm
produces 0 parsed mismatches in 128, while directions chosen by a reader at the same angle produce 137
(tense 121, count 9, location 7), all accepted by the 0.85 self gate and 109 of 128 by a 0.99 gate
[count_targeted, T3]. The gate is a non-adversarial guarantee only, and the retrieval-attack
literature already manufactures semantically unrelated passages at cosine 0.971 on demand (Ben-Tov
and Sharif, 2025). A filter that can see the original vector rejects every wrong decode we tried (0 of 1,750 across two
vocabularies, at 50% coverage). A filter that sees only the vector it is decoding lets hundreds
through (280 and 174; most were faithful decodes of deliberately substituted vectors, so this is a
property of the filter, not a spontaneous error rate). But a monitor in the field only has the second
kind, and no threshold for it reaches 50% coverage at 5% pessimistic risk [count_monitor,
count_continuous, T3]. And beam search is not the
repair: when the decoder's beam output differs from the true source sentence, the true sentence never
scores higher under the decoder (0 wins and 46–47 losses out of 64 per cohort at the default length
penalty; one win appears under other penalties), so noun substitutions are the model's conditional
probability, not a search failure [lexical_fidelity, T3].
### 8.3 Decoder error is not absent encoding
A dissociation found in September changes how §8's errors should be read. Halving the length of `z`
drops clean fidelity on the count grammar from 284/300 to 148/300. Of the 152 failures, 105 still
parse, and **every one of them changes only the location field, always to the same value** ("table")
[count_monitor, T3]. In a separate restoration run, 96 location-only errors at half norm all vanish
when the norm is restored to the calibration median (154/300 → 283/300, +43.0 pp [36.3, 49.7]), with
every source's parser status identical to clean [count_norm, T3]. A frozen linear reader recovers the
correct location on **all 96** of those failing cases and reads object, count, location and tense
300/300 on the clean set [count_reader, T3]. The decoder's error is a magnitude effect on one field,
not lost content. This is the sharper form of C7: norm is not *meaning*, but the decoder reads it,
and a global magnitude correction repairs what a fabrication gate would have counted as invention.
The Large Concept Model's own tables show a decoder fine-tuned on noised embeddings raises clean
auto-encoding BLEU from 79.5 to 88.0. Every fabrication number here was measured with the base
decoder, and a noise-robust decoder comparison remains the cheapest unrun experiment in this paper.
---
## 9. Results: dictionaries and the SAE residual 🟡 PARTIAL ◆ MED (block negatives replicate; C6 specificity LOW and untested at balanced activity)
**Block E (041–050) and the September S1–S3 branches. Block verdict: the SAE dictionary is a lossy
topical index, not a compositional basis for sentence meaning.**
**In one paragraph.** A sparse autoencoder (SAE) tries to rewrite each `z` as a sum of a few "atoms"
from a large learned dictionary, in the hope that the atoms are interpretable pieces of meaning. On
SONAR, the atoms are real and language-independent, but they index *topics*, not sentence structure.
The part of `z` the dictionary fails to capture (the residual) carries *more* information about the
sentence than the reconstruction does, making the dictionary wider does not help, and the atoms change
from one training seed to the next. The literature says that last point is expected. September added
that what the stable atoms are specific to is polarity and count, not roles, and that the production
dictionary's activations depended on which sentences happened to share a batch.
**Atoms are real and meaning-indexed, not English-indexed.** An English-trained dictionary's atoms
fire on the same sentences in translation: the activation-correlation identity rate is **1.000 across
all five tested language pairs** (German, Japanese, Turkish, Chinese, Arabic), against a null of
0.0035 [row 046, T3]. But semantic frames explain only ~3.3% of `z`'s variance [row 047, T3]: atoms are
topic and lexical-field detectors, not frame-structure detectors.
**The negatives are the substantial part.** **Atoms are depth-local**: in a crosscoder spanning L12,
L24, and `z`, only 26 of 13,706 firing atoms, **0.19%**, are shared across all three [row 041, T3].
**Width buys nothing**: across a 32× width sweep at matched budget, feature splitting is essentially
absent (4.1% → 0.4%, *shrinking* with width) and the fraction of variance unexplained (FVU) is flat
from h = 2048 to h = 65536 [row 044, T3]. **The residual out-informs the reconstruction** on every
certified probe (token length R² 0.907 vs 0.723, word content AUC 0.937 vs 0.878, domain 0.832 vs
0.782) at every width. It is structured, not noise: adding it back lifts decode chrF by **+25.3** where
adding Gaussian noise of the same size *lowers* it [rows 042–044, T3]. Keeping more atoms makes L2
reconstruction *worse* past m = 64 (FVU U-curve 0.69 → 0.83) while decode quality rises monotonically
(chrF 21 → 35). The dictionary's error metric and the decoder disagree about what matters [row 045,
T3].
**Three September results settle what was left open.**
*Specificity.* If the atoms mean anything, the ones that stay fixed when a sentence is paraphrased
should help predict what the paraphrase preserved. An *invariant mask* is that set of atoms. Against
activity-matched controls, the invariant atoms help predict negation (**+0.40**) and count
(**+0.16**) and do not help at all for roles (**−0.09**) on the eight historical dictionaries, with
the same sign pattern in a twelve-dictionary extension and under both inference rules; native `z`
itself scores 0 on role and 1 on polarity and count [S2, T3]. The bank
has 288 clusters in only 16 lexical blocks, which is the unit the bootstrap resamples. The polarity hard-contrast utility of 0.59–0.63
against ≤ 0.10 for all-features is the one real specificity signal. The earlier "invariant masks are
semantically specific" headline is withdrawn: its role bank consisted of word-order swaps on an
order-removed input, and its pooled 0.15 premium was mostly control degradation [A4]. Specificity at
balanced activity remains untested, since three matching designs failed their mass-retention gates.
*Seed stability.* If the atoms change when the dictionary is retrained, they cannot be a stable
vocabulary of meaning. Row 044's cross-seed floor fails (0.151) because low-frequency atoms are
seed-idiosyncratic. Six paired seeds per activation family give reciprocal matched fractions of 0.132
(per-sample TopK) and 0.111 (BatchTopK). Weighted by activation mass the fraction is ≈ 0.46 for both
(an auditor's replay on three of the fifteen seed pairs, not a registered analysis), so about 87% of
features by count and 54% by mass are unmatched under either rule, and there is no family effect
[S3, A4]. The dictionaries were unconverged at 2,760 optimizer steps. The literature says this is the expected outcome: only ~30% of features are
shared across seeds trained on identical data, and TopK-family activations are more seed-dependent
than penalty-based ones (Paulo and Belrose, 2025).
*Inference.* Likewise, if which atoms fire depends on which other sentences happened to be processed
in the same batch, the atoms are not a property of the sentence. The historical atom counts were
computed under per-sample TopK and all eight reproduce exactly. But the production BatchTopK encoder's active set depends on which sentences share a batch:
random regrouping changes the active set for 50–56% of sentences and norm-sorted grouping for
91–94%, across all eight historical checkpoints (45% and 92% in the one-checkpoint pilot). A single
calibrated global threshold gives activations that are unchanged under regrouping on seven of the
eight checkpoints, and differ on one row in 8,192 on the eighth, at the same FVU [S1; R16
pilot].[^batch]
Recoverability has been demonstrated for this latent; identifiability has not. Random-dictionary
baselines match trained ones on interpretability, sparse probing and causal editing in the one study
that checked (Korznikov et al., 2026).
[^batch]: The historical width trend survives the inference change (two-seed mean L1 mass fraction
0.400 at h = 8192 → 0.152 at h = 65536 under threshold, 0.434 → 0.162 under the original rule),
but feature *identity* does not: per-sample and threshold masks agree at Jaccard 0.58–0.65. The
production implementation was not changed. The honest statement of C6 is that atoms are semantic
at the frequent core and seed-idiosyncratic in the tail, and most of what a sentence is never
enters the dictionary at all.
---
## 10. Results: training dynamics 🟡 PARTIAL ◆ LOW (ladder-only; no SONAR checkpoint series)
**Block I (073–077). Block verdict: a stable-core / fragile-shell law. All evidence is ladder-only.**
**In one paragraph.** When, during training, does a text autoencoder acquire the codes described
above? We can only answer this on our own small models, because SONAR ships no intermediate
checkpoints. On those models, the surface-order code forms gradually in the first three-quarters of
training and then freezes; training five times longer produces no late emergence of role abstraction;
and what stays fragile is reconstruction quality, not structure. Different kinds of structure have
different timelines, so "the code forms mid-training" is true of word order and false of dictionary
atoms and operator directions.
The structural and relational code crystallizes early and freezes. The surface-order code and its
anti-transfer signature form **gradually in mid-training, onset ~11% of budget, saturated by ~74%**
[row 073, T3]. There is no late "click" (grokking): 15,000 epochs (5× the standard budget) produce zero delayed
emergence, role abstraction at chance across all 15 checkpoints [row 074, T3]. Compute alone buys no
abstraction.
Ontogeny, the order in which structure appears during training, is **substrate-specific**, which
is why C4 sits at 0.568. SAE atoms are front-loaded: half
the final inventory is matchable at 5.7% of budget, and ~31% of top-frequency atoms are present already
in the *untrained* encoder [row 048, T3]. Operator directions exist essentially at initialization
(negation cosine 0.93, vertical 0.999 at step 0), while their *causal usability* is learned,
decoder-gated, and installs abruptly at steps 2000–4000 [row 081, T3]. There are three ontogenies, not
one.
Path dependence splits the same way. Monotone curricula wreck fidelity while leaving the surface and
relational codes order-robust [row 076, T3]. Objective switching shows reconstruction following the
**last** objective while the surface-order code is imprinted by the **first** [row 077, T3]. Across 8
seeds the anatomy is stable **up to a global rotation** of the `z` basis [row 075, T3]. SONAR has no
released checkpoint series, so none of this has been shown on the model the rest of the paper is
about; the late emergence of the query-conditioned order code across SONAR's *layers* (§4.5) is the
nearest SONAR-side analogue.
---
## 11. Results: safety applications 🟡 PARTIAL ◆ LOW (32–96 hand-built items per row, single seed, no intervals; second system for counts only)
**Block K (082–089) and the September second-system work. Block verdict: the SONAR interface is
defensible against gross attacks and blind to subtle ones; style-transparent and identity-opaque.**
**In one paragraph.** What do the findings above mean for systems that trust `z`? Retrieval over `z`
gets roles right when the words differ, and gets them wrong as soon as the wrong candidate shares a
couple of words with the query, because there is no role variable to fall back on. A cosine-based
monitor cannot both admit paraphrases and reject a negation. `z` does not betray whether a sentence was
in the training set, and it encodes style far better than the identity of the writer. A second, quite
different system for turning vectors back into text reproduces the "counts survive, roles do not"
pattern.
**Retrieval.** In retrieval-augmented generation (RAG), a system fetches the stored sentence whose
vector is closest to a query's and hands it to a language model. Role structure survives a clean
round-trip: a role-correct query retrieves the right proposition over its role-swapped twin 48 times
out of 48, and the retrieved vectors decode back to the exact planted sentence every time [row 084,
T3]. That is fragile. A **single shared
surface word** buys the wrong candidate +0.123 cosine, so about two shared words overpower the +0.159
margin that being role-correct provides. When the distractor wins, the downstream question answerer is
wrong **100% of the time** (0.354 → 0.000, 5/5) [row 085, T3, Brier 0.099]. The demonstration is 48 hand-built triples with a single seed and no interval, and its verbose
query was itself a weak retriever before any distractor was added (§2.5). Taken as a demonstration
rather than a rate, it is the concrete safety consequence of §4: a retrieval system on pooled `z` has no abstract role variable to fall back on when
lexical overlap points the wrong way. The published home for this failure is retrieval evaluation,
where most models rank negated minimal pairs at or below random (Weller et al., 2024; van den Elsen et
al., 2025).
**Monitoring.** A cosine monitor cannot both admit paraphrases and reject meaning flips. A fluent
negation costs *less* cosine (0.069) than a meaning-preserving synonym swap, and closing the hole
requires a threshold of about 0.98, which rejects nearly everything [row 082, T3]. A `z` can hide
**~103 bits per sentence** of covert payload (steganography) past both a cosine and a decode-parse
monitor, though geometric monitors do
see those payloads [row 083, T3]. For comparison, token-space paraphrase defences hold steganography
under 3 bits per kilobyte, and no latent analogue of the paraphrase defence is known. §7.3 and §8.2
show that the bank metric these rows use does not transfer to natural text and that the
self-consistency gate cannot certify source integrity. Any decode-then-parse monitor must also be
benchmarked against a no-decode control, since adding a reading channel can make a monitor worse
(Arnav et al., 2025).
**Privacy.** Two clean negatives. An embedding does not inherently betray training membership: a
normally trained model leaks at AUC 0.546 (chance), an untrained one at 0.522; the signal appears
(AUC 1.000) only under memorization [row 088, T3, Brier 0.088]. And `z` is a **style** encoder, not
an identity encoder. Register (formal versus casual style, 0.998), topic (0.963), sentiment (0.903), and formality (0.873)
are highly linearly readable, while demographic identity collapses to chance once topic is partialled out;
`z` leaks identity *less* than the raw text would [row 089, T3].
**A second system.** C9 rested on one embedder. GTR with a vec2text inverter, a genuinely different
encoder and a decoder-free inversion route, recovers count-grammar content from the vector alone
**60 times out of 64** and fails the role gate at **40 of 64** [G1, T3]. Its norm sensitivity also
replicates in sign: clean 124/128 against 3/128 at half norm on a fresh 128-source main-domain panel
[G3]; the second vocabulary never ran because its calibration failed [G2]. The failure mode differs: GTR's
pooled norm tracks text length at r = −0.91, so a half-norm target is best matched by a long paraphrase
rather than by a field substitution. This is a system-level replication of the count result, not a
controlled decoder-only comparison, and it licenses no role or crosslingual claim. The corpus remains
templated; a non-templated replication is still required before any of this is quoted as a security
property.
---
## 12. Instruments and TAE-Bench 🟡 PARTIAL ◆ HIGH (kits reproduce exactly; benchmark not released)
**Block L (090–095, 099, 100).**
Five instrument kits were built, tested, and documented:
- the **probe-power kit** (row 090), which certifies a probe can detect a planted effect before a
null is trusted and catches exactly the uncertified-probe trap behind rows 051 and 062;
- the **canonicalization checker** (row 091), which catches the real bugs rows 017 and 020 hit
[Brier 0.005];
- the **norm linter** (row 092), whose construction surfaced §2.4's two-mode refinement [Brier 0.031];
- the **prereg engine** (row 094), which reproduces past rows' Brier scores exactly and makes
post-hoc editing detectable via a certificate hash [Brier 0.003];
- the **Bayesian ledger** (row 095) [Brier 0.042].
Row 093 is a useful adversarial null: a Codex loop evolving stimuli to break the binding probe found
**zero genuine breaks in 24 candidates**. The September work added two instruments the campaign
lacked. A check that a decode-then-parse auditor is actually reading the vector: given the right vector it
parses 64 of 64 items correctly, given a swapped or averaged vector 0 of 64 (premium 0.953 [0.891,
1.000]) [R06 pilot]. A rotation-code power check exposes what an additive plant does not certify [R01
pilot].
**TAE-Bench** (row 100) packages the battery, the five kits, the honesty gates, the stimuli (including
093's stress families), a SONAR embedding cache, a checkpoint manifest, and the ledger behind one
entry point. `python run_bench.py` reproduces the flagship `NO_BINDING_CONFIRMED` verdict on CPU,
offline, in minutes. It runs the battery, certifies probe power (plant d = 1.0, CI-lo ≥ 0.9 on its
packaged cell; the H1 transfer-regime certificate of §4.1 reported CI-lo 0.876 on a harder cell) and
canonicalization, and scores four frozen predictions against four honesty gates (exit code 2 =
`INSTRUMENT_FAILURE`). `--encoder mock` is the negative control and **must** fire
`INSTRUMENT_FAILURE`. Row 099 is an interactive `z`-explorer whose every decode is a byte-for-byte
recorded campaign output. TAE-Bench is validated locally only: it has not been pushed to a public
remote, carries no license, and has had no fresh-clone GPU smoke test (§15).[^public]
[^public]: The surrounding material is public. The project overview, this paper's rendered HTML, the
status board, the literature review and the next-steps plan are served at
`https://slop.ie/safety/research/tae-interp/`. The benchmark itself is not.
---
## 13. Limitations ✅ DONE
**Tier.** Every campaign result here is **T3-exploratory**; nothing has been formally promoted in the
program's claims ledger. The consolidation showed C1 is *promotable* for the linear claim, but that
entry has not been written (§15). The September rows carry their own tiers (Appendix B). Every
semantic judgment in them is a model judgment; no human reviewed any output in either loop.
**Sample sizes.** §2.5 and Appendix A give the number of evaluation items behind every row. The
campaign's typical row rests on hundreds to low thousands of items, but a minority of headline
numbers rest on fewer than fifty, and those are named in §2.5 and marked in the row table. Any number
in this paper without a confidence interval beside it should be read with that table open.
**What the certificate covers.** The binding null is an English cross-construction *unconditioned
linear-probe* null, certified against an additive planted effect. Rotation and superposition codes
are not covered by that certificate, and §4.5 shows that a query-conditioned reader recovers order
where the unconditioned one cannot. The scope is a reader class and a query regime, not the
information content of `z`.
**Single model family, English-centric templated stimuli.** SONAR plus one small TAE ladder. Block
F's ten external representations extend the *binding* null specifically; six embedders in row 051
failed their positive-control gate and are reported as blocked; the one second-system replication
(GTR/vec2text) covers counts, not roles. The crosslingual claims are the weakest here and two rows
have been retracted (§4.4). Content-level and null claims should transfer to other mean-pooling
encoders, because the mechanism is architectural. Geometry-level claims (the specific decay
constants, the operator directions, any cosine threshold) should not, and SONAR's own authors needed
a nonlinear connector and ~14M paired samples to align a second encoder into its space.
**Substrate mixing, and one post-hoc number.** Some claims blend SONAR and ladder evidence,
particularly capacity (C2) and ontogeny (C4); C4 is ladder-only and should be labeled as such
wherever quoted. The 460-bit ceiling was not frozen in a prereg and is a decoder-extractable lower
bound rather than a mutual-information estimate.
**Correlated rows inflate naive evidence.** Rows within a block share stimuli, code, and often
checkpoints. The ledger's conservative likelihood ratios, Brier shrinkage, and ±5 clamp are
mitigations, not solutions. C1's 0.993 is *at* the clamp and should be read as "the campaign's
evidence saturates the ledger's allowance," not as a calibrated probability.
**Withdrawn numbers.** Four campaign- or consolidation-time results have been withdrawn or rescoped
since the previous version: the 061/062 multilingual binding positives (§4.4), the
antisymmetric-component role readout (§5.2), the operator specificity figure (§7.2), and the
vertical-operator natural-text result (§7.2). Two program headlines that predate the campaign, the
unbinding "floor" and the 41–49% silent-flip rate, are re-scoped in §6.2 and §8.2.
**Numerical reproducibility.** SONAR encodings under PyTorch's `high` matmul precision are not
batch-invariant (maximum relative L2 drift 6.8·10⁻⁴ across batch groupings; `highest` is invariant to
1.4·10⁻⁶). A pre-registered confirmation of the query-conditioned role reader on 400 fresh sources was
closed by that gate rather than by a scientific result. Any claim that reuses frozen readers on newly
encoded inputs is not bit-reproducible under the default stack.
**Negatives reported at full strength.** The auditing block is mostly failures (§8). Row 060 shows
`z`-cosine anti-correlates with (proxy) human similarity judgment within topic. Row 013 shows the
role-structured attention we found is causally inert. Row 023 refuted the program's own "fluent lies
off-manifold" picture. E04 and E04b found that the readable order direction is not a causal handle.
---
## 14. Related work 🟡 PARTIAL (identifiers verified by the literature review; not every work read at primary source by the author)
**The object.** SONAR (Duquenne et al., 2023) is a 1024-dimensional, 200-language encoder-decoder with
a mean-pooled bottleneck trained on translation with auxiliary MSE and denoising terms. Its successor
OmniSONAR (Janeiro et al., 2026) changes the objective to token-level decoding with a contrastive
term; no public checkpoint has been verified, so every result here is on the 2023 model. The Large
Concept Model (Barrault et al., 2024) plans in SONAR space and is the source of the fragility score
(how much a decode changes under small latent perturbations) and of the noise-robust decoder result
that bounds §8. The program is no longer alone on this encoder. FLiP (Kesiraju et al., 2026) recovers
77.29% of English SONAR lexical content with a rank-512 linear map against an explicitly unordered
target, within two points of our own token-identity figure by a different method. SPARTA (Zinkovich
et al., 2026) finds a single SONAR dimension correlating with token length at r = −0.956. Allesiardo
et al. (2026) build an anomaly detector from perturbation-sensitive SONAR dimensions. Hawthorne and
Nellessen (2025) report decoder-inert language directions. The only prior dictionary-learning study on
SONAR (Pochinkov and Darmawan, 2026) trains BatchTopK, JumpReLU and gated variants at m = 16,384 and
an interpretation dictionary at m = 128k, finds three language-agnostic latents can carry a simple
sentence, and is explicitly correlational; block E is its causal and stability follow-up.
**Binding and thematic roles in sentence vectors.** The dissociation between order and role in a fixed
sentence latent was reported by Ettinger et al. (2018) on a 2400-dimensional sequential denoising
autoencoder (SemRole 63.7%, word order 92.9%, bag-of-words floor 51.3%), without a power analysis.
Tensor-product decompositions (McCoy et al., 2019) and their freely learned successor ROLE (Soulos et
al., 2020) find compositional role structure in synthetic sequence models and none in natural-language
sentence encoders, where ROLE barely beats bag-of-words. ROLE has never been applied to SONAR, whose
single-vector form is what it required. Nikolaev and Padó (2023) give the behavioural statement. Mohr
et al. (2026) find that extensional supervision degrades relational concept families in sentence
encoders and that fine-tuning recalibrates rather than expands the geometry, which makes the null
predicted. In language models, binding is implemented through per-token-position identifiers (Feng
and Steinhardt, 2024), read out by token-indexed propositional probes (Feng et al., 2024), with a
content-keyed fallback under load (Gur-Arieh et al., 2026); the classical framing is Smolensky (1990)
and Greff et al. (2020). Nastase and Merlo (2023; 2024) recover agreement and alternation structure
from transformer sentence embeddings at F1 0.997 within templates and see it degrade under maximal
lexical variation, the same within-multiset versus cross-pair shape we find.
**Probing methodology.** Control tasks (Hewitt and Liang, 2019), minimum-description-length probing
(Voita and Titov, 2020) and the survey in Belinkov (2022) define the reader-relative reading of a
probe. V-information (Xu et al., 2020), its instrument PVI (Ethayarajh et al., 2022) and conditional
probing (Hewitt et al., 2021) give "encoded but not linearly readable" a formal home. Statistical
power in NLP evaluation is treated by Card et al. (2020). Coding relativity, the observation that
Othello-GPT's board state was non-linear until re-parameterised (Nanda et al., 2023) and that some
features are irreducibly multi-dimensional (Engels et al., 2024), is the typological form of caution
1. Apparent word-order signal in shuffle experiments is partly length and unigram statistics, and the
best-known "order does not matter" result rested on a shuffling bug (Ravishankar et al., 2022).
**Pooling and permutation invariance.** Deep Sets (Zaheer et al., 2017) and its limits (Wagstaff et
al., 2019); Janossy pooling (Murphy et al., 2019) and the non-injectivity of its piecewise-linear
forms (Reshef and Dym, 2025); the theorem that linear invariant readouts factor through the Reynolds
operator (Talhi et al., 2026) and Reynolds networks (Sannai et al., 2021). Order is recoverable from
averaged embeddings (Lee et al., 2020; Chavan et al., 2023). Outlier dimensions in mean-pooled
embeddings are positional (Luo et al., 2021); anisotropy is architectural (Godey et al., 2024) and its
tied-softmax mechanism (Gao et al., 2019; Puccetti et al., 2022) does not apply to a bottleneck
encoder.
**Capacity.** Rate-distortion for autoencoders (Alemi et al., 2018), the compression-prediction
equivalence (Delétang et al., 2024), the log N barrier on mutual-information lower bounds (McAllester
and Stratos, 2020), single-vector token capacity as a property of the encoder-decoder pair (Kuratov
et al., 2025), and the dimension bound on embedding retrieval (Weller et al., 2026). Embedding
inversion (Morris et al., 2023) supplies the attack-side view of the same budget. The reproducibility
study of Seputis et al. (2025) shows exact match is not monotone in length once the attacker is held
fixed,[^inv] and Dong et al. (2025) invert 4,112-token prompts from *unpooled* states, isolating
pooling rather than length as the cause of the profile we measure.
[^inv]: An earlier program note cited vec2text's 61% exact match at 32 tokens falling to 8% at 128
as a capacity profile. Those numbers come from two inverters each trained at its own length;
with the attacker fixed, exact match is 4.8% at 32 tokens and 8.1% at 81. The comparison is
withdrawn.
**Steering and latent editing.** Activation addition (Turner et al., 2023), contrastive activation
addition (Panickssery et al., 2024), task arithmetic (Ilharco et al., 2023), function vectors (Todd
et al., 2024) and concept algebra (Wang et al., 2023) are the precedents for an algebra of edits;
none is on a sentence autoencoder. Reliability results (Tan et al., 2024; Braun et al., 2025) bound
what in-distribution success shows. Sentence Smith (2025) provides controlled meaning manipulations
on which role swap is the hardest for embedding models to detect. InterSent (Huang et al., 2023)
measured additive sentence-embedding operators directly and found addition materially worse than
learned operators, which our decode-certified composition result should be read against.
**Fabrication, auditing and monitoring.** Decoder priors overriding conditioning (Gong et al., 2026).
Verbalization methods that score without privileged access (Li et al., 2026), which is why a
latent-ablated null is now part of the audit. Interpretability illusions under distribution shift
(Friedman et al., 2024; Bolukbasi et al., 2021) and overconfidence under shift (Ovadia et al., 2019)
as the general form of a gate that fails open; selective prediction with a learned calibrator (Kamath
et al., 2020) as the one published treatment. Cosine similarity as an unreliable fidelity axis (Steck
et al., 2024; Nastase and Merlo, 2025), and a scalar quality score failing open on exactly the severe
errors that a back-translation check caught, in a clinical setting (Mehandru et al., 2023).
Adversarial retrieval (Ben-Tov and Sharif, 2025) and negation in retrieval (Weller et al., 2024; van
den Elsen et al., 2025). Chain-of-thought monitorability and its failure modes (Korbak et al., 2025;
Arnav et al., 2025; Ferreira et al., 2026); steganographic capacity under paraphrase (Roger and
Greenblatt, 2023; Zhou and May, 2026); positive signalling without positive listening (Lowe et al.,
2019). Vector-mediated agent communication is deployed and unmonitored (Du et al., 2026; Zou et al.,
2026), so every single-vector result here is a lower bound on the real audit problem. Membership and
attribute inference on sentence embeddings: Song and Raghunathan (2020). ParaScopes (Pochinkov et
al., 2025) is the program's own precedent for the distinction between decoding a latent and finding a
variable in it.
**Sparse dictionaries.** BatchTopK (Bussmann et al., 2024), Matryoshka SAEs (Bussmann et al., 2025),
absorption (Chanin et al., 2025), seed instability (Paulo and Belrose, 2025), random baselines
(Korznikov et al., 2026; Heap et al., 2025), and the benchmark critiques (Karvonen et al., 2025;
SynthSAEBench, 2026; "Are Sparse Autoencoder Benchmarks Reliable?", 2026). Evaluation of automated
interpretation (Paulo et al., 2024; Gur-Arieh et al., 2025; "Evaluating Neuron Explanations", 2025;
CHIVE, 2026; "Pitfalls in Evaluating Interpretability Agents", 2026) shaped the September audit
design.
**Latent reasoning.** Continuous-thought models (Hao et al., 2024) and the 2025–26 negative results on
superposition in them; the finding that autoencoding sentence embeddings were the arm that *failed*
as a planning currency; and the two 2026 interpretability studies of latent-reasoning models (Dilgren
and Wiegreffe, 2026; "Interpretability-Guided Intervention", 2026) whose instruments all project
through a tied unembedding onto a token codebook that a pooled sentence latent lacks. If the field is
leaving pooled sentence latents as a planning substrate, the safety motivation for this paper weakens
and the scientific one strengthens: the object becomes a clean, well-instrumented model organism for
what fixed-vector compression does to structure.
---
## 15. Open problems ✅ DONE
What remains undone, as of 11 September 2026.
1. **The ledger rerun.** `bayesian_ledger.py` has not been rerun with H1–H3 and the September rows as
support and rows 061/062 demoted; until then C8's posterior is stale in a known direction.
2. **Formal promotion.** No campaign claim has been promoted above T3 in the program's claims ledger.
C1 is promotable, with its scope statement, but the entry has not been written.
3. **Predicate-general role access.** The reader that would decide whether role is addressable across
held-out predicates does not exist; the paired-objective attempt induced abstention, and the
single-event final panel has not been opened.
4. **The decoder's order boundary.** E04b ruled out a reader-specific order subspace; whether a
single "structured energy" variable predicts the decoded flip across subspace types (E05) is
pre-registered and unrun.
5. **A rotation-code power certificate on real SONAR.** The R01 pilot is synthetic; the binding null
is certified against additive plants only.
6. **A noise-robust decoder.** Every fabrication and audit number was measured with the base SONAR
decoder; the Large Concept Model's fine-tuned decoder is the cheapest way to learn whether the
fail-closed and coverage results are about the latent or the decoder.
7. **Applicable-text operator editing.** The physical-relation study's 152 final sources were never
decoded; the vertical operator has no natural-text result.
8. **Thin rows.** The rows §2.5 flags as resting on very few items should be re-run at larger n
before any of their numbers is quoted outside this paper.
9. **Figures** (Appendix C), **human calibration** for row 060, and **TAE-Bench public release**
(repository, license, fresh-clone GPU smoke test).
10. **Blocked rows.** Row 009 needs a *verified* binding teacher; row 051 needs a per-embedder
recalibrated battery.
11. **Second-corpus replication** of the safety block on a non-templated corpus, and a second seed.
---
## References 🟡 PARTIAL (twelve entries cited by title; author lists to be completed)
Where a work is cited by title, no author list was verified; identifiers were checked against their
index pages during the September literature review.
- Alemi, A. A., Poole, B., Fischer, I., Dillon, J. V., Saurous, R. A., Murphy, K. (2018). Fixing a
broken ELBO. ICML 2018.
- Allesiardo, R., Caubrière, A., Vielzeuf, V. (2026). Forewarned is forearmed. LREC 2026.
arXiv:2606.30196.
- Arnav, B., et al. (2025). CoT red-handed. arXiv:2505.23575.
- Barrault, L., Duquenne, P.-A., Elbayad, M., Kozhevnikov, A., et al. (2024). Large Concept Models:
language modeling in a sentence representation space. arXiv:2412.08821.
- Belinkov, Y. (2022). Probing classifiers: promises, shortcomings, and advances. Computational
Linguistics. arXiv:2102.12452.
- Ben-Tov, M., Sharif, M. (2025). GASLITEing the retrieval. ACM CCS 2025. arXiv:2412.20953.
- Bolukbasi, T., et al. (2021). An interpretability illusion for BERT. arXiv:2104.07143.
- Braun, J., et al. (2025). Understanding (un)reliability of steering vectors. arXiv:2505.22637.
- Bussmann, B., Leask, P., Nanda, N. (2024). BatchTopK sparse autoencoders. arXiv:2412.06410.
- Bussmann, B., Nabeshima, N., Karvonen, A., Nanda, N. (2025). Learning multi-level features with
Matryoshka sparse autoencoders. arXiv:2503.17547.
- Card, D., et al. (2020). With little power comes great responsibility. EMNLP 2020.
arXiv:2010.06595.
- Chanin, D., et al. (2025). A is for absorption. NeurIPS 2025. arXiv:2409.14507.
- Chavan, T., et al. (2023). SenTest. arXiv:2311.17722.
- CHIVE (2026). Would this change your answer? Anthropic Alignment Science.
https://alignment.anthropic.com/2026/chive/
- Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., et al. (2024). Language modeling
is compression. ICLR 2024.
- Dilgren, J., Wiegreffe, S. (2026). Are latent reasoning models easily interpretable? COLM 2026.
arXiv:2604.04902.
- Dong, et al. (2025). Depth gives a false sense of privacy. USENIX Security 2025. arXiv:2507.16372.
- Du, Z., et al. (2026). Interlat. ACL 2026. arXiv:2511.09149.
- Duquenne, P.-A., Schwenk, H., Sagot, B. (2023). SONAR: sentence-level multimodal and
language-agnostic representations. arXiv:2308.11466.
- Engels, J., et al. (2024). Not all language model features are one-dimensionally linear.
arXiv:2405.14860.
- Ethayarajh, K., Choi, Y., Swayamdipta, S. (2022). Understanding dataset difficulty with
V-usable information. ICML 2022. arXiv:2110.08420.
- Ettinger, A., Elgohary, A., Phillips, C., Resnik, P. (2018). Assessing composition in sentence
vector representations. COLING 2018.
- "Evaluating neuron explanations: a unified framework with sanity checks" (2025). ICML 2025.
arXiv:2506.05774.
- Feng, J., Steinhardt, J. (2024). How do language models bind entities in context? ICLR 2024.
arXiv:2310.17191.
- Feng, J., Russell, S., Steinhardt, J. (2024). Monitoring latent world states in language models with
propositional probes. arXiv:2406.19501.
- Ferreira, et al. (2026). Does out-of-sight equal out-of-mind? arXiv:2608.04928.
- Friedman, D., et al. (2024). Interpretability illusions in the generalization of simplified models.
ICML 2024. arXiv:2312.03656.
- Gao, J., et al. (2019). Representation degeneration problem in training natural language generation
models. ICLR 2019. arXiv:1907.12009.
- Godey, N., de la Clergerie, É., Sagot, B. (2024). Anisotropy is inherent to self-attention in
transformers. EACL 2024. arXiv:2401.12143.
- Gong, et al. (2026). Faithful patchscopes. arXiv:2602.00300.
- Greff, K., van Steenkiste, S., Schmidhuber, J. (2020). On the binding problem in artificial neural
networks. arXiv:2012.05208.
- Gur-Arieh, Y., Elhelo, R., Geva, M. (2025). Enhancing automated interpretability with output-centric
feature descriptions. ACL 2025. arXiv:2501.08319.
- Gur-Arieh, Y., Geva, M., Geiger, A. (2026). Mixing mechanisms. ICLR 2026. arXiv:2510.06182.
- Hao, S., et al. (2024). Training large language models to reason in a continuous latent space.
arXiv:2412.06769.
- Hawthorne, Nellessen (2025). Investigating representations in the embedding in SONAR text
autoencoders. LessWrong, 6 September 2025.
- Heap, T., Lawson, T., Farnik, L., Aitchison, L. (2025). Sparse autoencoders can interpret randomly
initialized transformers. arXiv:2501.17727.
- Hewitt, J., Liang, P. (2019). Designing and interpreting probes with control tasks. EMNLP 2019.
arXiv:1909.03368.
- Hewitt, J., Ethayarajh, K., Liang, P., Manning, C. D. (2021). Conditional probing: measuring usable
information beyond a baseline. EMNLP 2021.
- Huang, J. Y., Yao, W., Song, K., Zhang, H., Chen, M., Yu, D. (2023). Bridging continuous and
discrete spaces: interpretable sentence representation learning via compositional operations
(InterSent). EMNLP 2023. arXiv:2305.14599.
- Ilharco, G., et al. (2023). Editing models with task arithmetic. ICLR 2023. arXiv:2212.04089.
- "Interpretability-guided intervention" (2026). ACL 2026. arXiv:2606.01243.
- Janeiro, J. M., Huguet Cabot, P.-L., Tsiamas, I., et al. (2026). Omnilingual SONAR.
arXiv:2603.16606.
- Kamath, A., Jia, R., Liang, P. (2020). Selective question answering under domain shift. ACL 2020.
arXiv:2006.09462.
- Karvonen, A., et al. (2025). SAEBench. ICML 2025. arXiv:2503.09532.
- Kesiraju, S., Yusuf, B., Sedláček, Š., Plchot, O., Schwarz, P. (2026). FLiP: towards understanding
and interpreting multimodal multilingual sentence embeddings. Interspeech 2026. arXiv:2604.18109.
- Korbak, T., et al. (2025). Chain of thought monitorability. arXiv:2507.11473.
- Korznikov, et al. (2026). Sanity checks for sparse autoencoders: do SAEs beat random baselines?
arXiv:2602.14111.
- Kuratov, Y., Arkhipov, M., Bulatov, A., Burtsev, M. (2025). Cramming 1568 tokens into a single
vector and back again. ACL 2025. arXiv:2502.13063.
- Lee, J., Camacho-Collados, J., Espinosa-Anke, L., Schockaert, S. (2020). Capturing word order in
averaging based sentence embeddings. ECAI 2020.
- Li, M., Ceballos Arroyo, A., Rogers, A., Saphra, N., Wallace, B. (2026). Do activation verbalization
methods convey privileged information? ICML 2026. arXiv:2509.13316.
- Lowe, R., et al. (2019). On the pitfalls of measuring emergent communication. AAMAS 2019.
arXiv:1903.05168.
- Luo, Z., Kulmizev, A., Mao, X. (2021). Positional artefacts propagate through masked language model
embeddings. ACL 2021. arXiv:2011.04393.
- McAllester, D., Stratos, K. (2020). Formal limitations on the measurement of mutual information.
AISTATS 2020. arXiv:1811.04251.
- McCoy, R. T., Linzen, T., Dunbar, E., Smolensky, P. (2019). RNNs implicitly implement tensor product
representations. ICLR 2019. arXiv:1812.08718.
- Mehandru, N., Agrawal, S., Xiao, Y., Khoong, E., Gao, G., Carpuat, M., Salehi, N. (2023). Physician
detection of clinical harm in machine translation. EMNLP 2023. arXiv:2310.16924.
- Mohr, Dujany, Souquet, Freitas (2026). Principles of concept representation in sentence encoders.
arXiv:2606.06994.
- Morris, J. X., Kuleshov, V., Shmatikov, V., Rush, A. M. (2023). Text embeddings reveal (almost) as
much as text. EMNLP 2023. arXiv:2310.06816.
- Murphy, R. L., Srinivasan, B., Rao, V., Ribeiro, B. (2019). Janossy pooling. ICLR 2019.
arXiv:1811.01900.
- Nanda, N., Lee, A., Wattenberg, M. (2023). Emergent linear representations in world models of
self-supervised sequence models. arXiv:2309.00941.
- Nastase, V., Merlo, P. (2023). Disentangling continuous and discrete linguistic signals in
transformer-based sentence embeddings. arXiv:2312.11272.
- Nastase, V., Merlo, P. (2024). Are there identifiable structural parts in the sentence embedding
whole? BlackboxNLP 2024. arXiv:2406.16563.
- Nastase, V., Merlo, P. (2025). Testing the assumptions about the geometry of sentence embedding
spaces: the cosine measure need not apply. arXiv:2509.01606.
- Nikolaev, D., Padó, S. (2023). Representation biases in sentence transformers. EACL 2023.
arXiv:2301.13039.
- Ovadia, Y., et al. (2019). Can you trust your model's uncertainty? NeurIPS 2019.
- Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., Turner, A. M. (2024). Steering
Llama 2 via contrastive activation addition. ACL 2024.
- Paulo, G., Mallen, A., Juang, C., Belrose, N. (2024). Automatically interpreting millions of features
in large language models. arXiv:2410.13928.
- Paulo, G., Belrose, N. (2025). Sparse autoencoders trained on the same data learn different
features. arXiv:2501.16615.
- "Pitfalls in evaluating interpretability agents" (2026). arXiv:2603.20101.
- Pochinkov, N., et al. (2025). ParaScopes. arXiv:2511.00180.
- Pochinkov, N., Darmawan, J. R. (2026). Interpretability of text auto encoders using sparse auto
encoders: a sandbox for interpreting neuralese. EACL 2026 submission.
- Puccetti, G., et al. (2022). Outlier dimensions that disrupt transformers are driven by frequency.
Findings of EMNLP 2022. arXiv:2205.11380.
- Ravishankar, V., Abdou, M., Kulmizev, A., Søgaard, A. (2022). Word order does matter (and shuffled
language models know it). ACL 2022. arXiv:2203.10995.
- Reshef, Dym (2025). On the (non) injectivity of piecewise linear Janossy pooling. arXiv:2505.20150.
- Roger, F., Greenblatt, R. (2023). Preventing language models from hiding their reasoning.
arXiv:2310.18512.
- Sannai, A., Kawano, M., Kumagai, W. (2021). Equivariant and invariant Reynolds networks.
arXiv:2110.08092.
- Sentence Smith (2025). Controllable edits for evaluating text embeddings. arXiv:2502.14734.
- Seputis, Li, Langerak, Mihailov (2025). Rethinking the privacy of text embeddings: a reproducibility
study. RecSys 2025. arXiv:2507.07700.
- Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures
in connectionist systems. Artificial Intelligence 46.
- Song, C., Raghunathan, A. (2020). Information leakage in embedding models. CCS 2020.
- Soulos, P., McCoy, R. T., Linzen, T., Smolensky, P. (2020). Discovering the compositional structure
of vector representations with role learning networks. BlackboxNLP 2020. arXiv:1910.09113.
- Steck, H., Ekanadham, C., Kallus, N. (2024). Is cosine-similarity of embeddings really about
similarity? WWW 2024 Companion. arXiv:2403.05440.
- SynthSAEBench (2026). arXiv:2602.14687.
- "Are sparse autoencoder benchmarks reliable?" (2026). arXiv:2605.18229.
- Talhi, Wolf, Monod (2026). Breaking symmetry bottlenecks in GNN readouts. arXiv:2602.05950.
- Tan, D., et al. (2024). Analyzing the generalization and reliability of steering vectors. NeurIPS
2024. arXiv:2407.12404.
- Todd, E., et al. (2024). Function vectors in large language models. ICLR 2024. arXiv:2310.15213.
- Turner, A. M., et al. (2023). Activation addition: steering language models without optimization.
arXiv:2308.10248.
- van den Elsen, et al. (2025). Reproducing NevIR. SIGIR 2025. arXiv:2502.13506.
- Voita, E., Titov, I. (2020). Information-theoretic probing with minimum description length. EMNLP
2020. arXiv:2003.12298.
- Wagstaff, E., et al. (2019). On the limitations of representing functions on sets. ICML 2019.
arXiv:1901.09006.
- Wang, Z., et al. (2023). Concept algebra for (score-based) text-controlled generative models.
arXiv:2302.03693.
- Weller, O., Lawrie, D., Van Durme, B. (2024). NevIR: negation in neural information retrieval. EACL
2024. arXiv:2305.07614.
- Weller, O., Boratko, M., Naim, I., Lee, J. (2026). On the theoretical limitations of embedding-based
retrieval. ICLR 2026. arXiv:2508.21038.
- Xu, Y., et al. (2020). A theory of usable information under computational constraints. ICLR 2020.
arXiv:2002.10689.
- Zaheer, M., et al. (2017). Deep Sets. NeurIPS 2017. arXiv:1703.06114.
- Zhou, May (2026). Conceptual steganography. arXiv:2605.26537.
- Zinkovich, et al. (2026). SPARTA. EACL 2026.
- Zou, J., et al. (2026). LatentMAS. ICML 2026. arXiv:2511.20639.
---
## Appendix A: the 100 rows ✅ DONE
Generated from `board_data.json`. **Outcome** is the row's own verdict class: *signal*
(pre-registered effect found), *null* (no effect, gates permitting interpretation), *twist* (a real
result that contradicted the prereg), *blocked*. **Brier** is scored against that row's frozen
predictions (lower is better). **Lane** is the current status: 🟩 hardened (survived consolidation or
directly supports a hardened claim), 🟦 shipped (a released artifact), ⬜ T3 (campaign evidence only),
🟥 refuted, ⬛ blocked, 🟨 partial. **Follow-up** is the row's own answer to "is a follow-up worth
funding?" (Y / N / ?). ★ marks a row the campaign flagged as a headline result at harvest time; two
starred rows (061, 062) are now refuted.
| n | block | name | outcome | Brier | lane | follow-up | items behind the headline | seeds | evidence |
|---|---|---|---|---|---|---|---|---|---|
| 001 | A | Role-swap contrastive | null | 0.218 | 🟩 hardened | Y | 500 held-out test sentences per pooled… | 1 | **very thin** |
| 002 | A | QA-head dose-response | null | 0.166 | 🟩 hardened | ? | 500 held-out test sentences per dose | 1 | **very thin** |
| 003 | A | Structured decoder | null | 0.062 | 🟩 hardened | ? | 500 held-out test sentences for the transfer null | 1 | **very thin** |
| 004 | A | Shuffled-input DAE | twist | 0.180 | 🟩 hardened | ? | 500 held-out test sentences for surface-cross and… | 1 | **very thin** |
| 005 | A | Passive curriculum | null | 0.078 | 🟩 hardened | ? | 500 held-out test sentences per pooled cell | 1 | **very thin** |
| 006 | A | ★ Lexical-diversity phase transition | signal | 0.306 | 🟩 hardened | ? | 300 novel-filler sentences per dose for the headline… | 1 | **very thin** |
| 007 | A | Two-slot bottleneck | twist | 0.216 | ⬜ T3 | ? | 2k held-out transitive items for the slot-specialization… | 1 | **very thin** |
| 008 | A | Anti-binding mechanism | null | 0.192 | ⬜ T3 | ? | 2000 stimuli probed at 17 checkpoints x 3 seeds | 3 | thin |
| 009 | A | Binding distillation | blocked | 0.022 | ⬛ blocked | Y | 2000 stimuli + 1200 genitive stimuli as positive control | 1 | ok |
| 010 | A | Case-marked scrambled MT | twist | 0.238 | ⬜ T3 | ? | 2k held-out same-vocab sentences + 300 novel-filler… | 1 | **very thin** |
| 011 | B | Pooling-input role code (v4) | null | 0.267 | ⬜ T3 | ? | 2000 sentences + 1200 genitive sentences | 1 | ok |
| 012 | B | ★ Role-consistent attention heads | signal | 0.380 | ⬜ T3 | ? | 2000 sentences | 1 | ok |
| 013 | B | Role-head knockout | null | 0.172 | ⬜ T3 | ? | 2000 sentences , each decoded under baseline + 11… | 1 | ok |
| 014 | B | ★ Swap-patching localization | signal | 0.193 | 🟩 hardened | ? | 43 kept swap-pairs per family after equal-piece-count… | 1 | thin |
| 015 | B | Pooler retrofit | null | 0.092 | ⬜ T3 | ? | 2000 sentences | 1 | ok |
| 016 | B | Nonlinear / kernel probes | null | 0.256 | 🟩 hardened | ? | 2000 sentences | 1 | ok |
| 017 | B | Position × layer heatmap | twist | 0.129 | 🟩 hardened | ? | 2000 sentences | 1 | ok |
| 018 | B | Function-word carriers | null | 0.035 | 🟩 hardened | N | per cell: 100 test items within one family | 1 | thin |
| 019 | B | Decoder-layer mirror | twist | 0.041 | ⬜ T3 | ? | 2000 sentences teacher-forced through the decoder | 1 | ok |
| 020 | B | z-read lens | twist | 0.068 | ⬜ T3 | ? | 2000 sentences teacher-forced | 1 | ok |
| 021 | C | ★ Fabrication taxonomy | signal | 0.179 | ⬜ T3 | Y | 1500 clean sentences | 1 | thin |
| 022 | C | ★ Entropy signature | signal | 0.267 | ⬜ T3 | Y | 1500 sentences with 54 fabrication positives | 3 | thin |
| 023 | C | Off-manifold dose-response | null | 0.388 | ⬜ T3 | Y | 400 sentences x 19 conditions = 7600 greedy decodes for… | 1 | thin |
| 024 | C | ★ LM-prior mechanism | signal | 0.220 | ⬜ T3 | Y | 127 paired fabricated tokens from 38 fabricated sentences | 1 | **very thin** |
| 025 | C | ★ Fidelity-gate reliability | signal | 0.116 | ⬜ T3 | ? | 200 base sentences | 1 | ok |
| 026 | C | Decoding strategy | twist | 0.216 | ⬜ T3 | ? | semantic rates judged on a fixed 300-sentence subsample… | 1 | **very thin** |
| 027 | C | Round-trip attractors | null | 0.168 | ⬜ T3 | ? | 2000 seed sentences iterated up to 30 steps | 1 | ok |
| 028 | C | ★ SAE decomposition | signal | 0.188 | ⬜ T3 | Y | 1500 sentences decomposed with one SAE | 1 | ok |
| 029 | C | ★ Decode-verify auditor (v2) | null | 0.467 | ⬜ T3 | Y | 1500 pairs with 54 fabrication positives | 3 | thin |
| 030 | C | Confidence calibration | twist | 0.181 | ⬜ T3 | Y | 30,742 decoded tokens from 1500 greedy reconstructions | 1 | ok |
| 031 | D | ★ Local intrinsic-dimension field | signal | 0.202 | ⬜ T3 | ? | 29,029 SONAR sentences in a 4-domain x 3-length grid,… | 1 | ok |
| 032 | D | ★ Geodesic vs linear interpolation | signal | 0.262 | ⬜ T3 | ? | 200 pairs x 9 points x 2 methods = 3600 greedy decodes… | 1 | **very thin** |
| 033 | D | ★ Analogy operator battery | signal | 0.254 | 🟩 hardened | Y | 40 held-out test pairs per transform , disjoint vocabulary | 1 | thin |
| 034 | D | ★ Causal negation operator | signal | 0.138 | 🟩 hardened | Y | 56 held-out natural affirmatives for ADD (peak 0.84 at… | 1 | thin |
| 035 | D | Operator curvature | twist | 0.179 | 🟩 hardened | Y | templated pairs: 120 train per region per transform | 1 | ok |
| 036 | D | Norm semantics | twist | 0.249 | ⬜ T3 | Y | Corpus B 1,500 sentences | 1 | thin |
| 037 | D | ★ Anisotropy audit | signal | 0.193 | 🟩 hardened | Y | Corpus A 29,029 sentences | 1 | thin |
| 038 | D | Antipodal decoding | null | 0.151 | ⬜ T3 | ? | 400 seed sentences (antipode judged in full n=400 | 1 | ok |
| 039 | D | ★ Persistent homology | null | 0.056 | ⬜ T3 | N | 1,000-point random subsample of 29,029 z | 1 | thin |
| 040 | D | Whitening robustness | signal | 0.312 | ⬜ T3 | ? | 1,500 orig/reconstruction pairs with 54 labeled… | 1 | thin |
| 041 | E | Crosscoders across depth | null | 0.266 | ⬜ T3 | Y | 4,000 val sentences (FVU | 1 | **very thin** |
| 042 | E | ★ The residual is structured | signal | 0.264 | ⬜ T3 | Y | 1,000 val sentences greedy-decoded x 5 conditions | 1 | ok |
| 043 | E | What the dictionary drops | signal | 0.106 | ⬜ T3 | Y | Corpus A 12,000 sentences, 70/30 split -> 3,600 test | 1 | thin |
| 044 | E | Width buys nothing | null | 0.144 | ⬜ T3 | Y | 36,000 pile z for atom firing/matching | 2 | thin |
| 045 | E | The k-atom curve | twist | 0.146 | ⬜ T3 | Y | 1,000 pile-val sentences greedy-decoded x 12 conditions | 1 | ok |
| 046 | E | ★ Atoms are meaning-indexed | signal | 0.179 | ⬜ T3 | Y | 2,009 FLORES-200 parallel rows x 6 languages | 1 | ok |
| 047 | E | Frames vs atoms | twist | 0.201 | ⬜ T3 | Y | 5,000 FrameNet sentences , 70/30 stratified split ->… | 1 | thin |
| 048 | E | Atom ontogeny | signal | 0.179 | ⬜ T3 | Y | 20,000 owt_val eval rows for activation-correlation… | 2 | thin |
| 049 | E | Paraphrase invariance | twist | 0.179 | ⬜ T3 | Y | 5,000 pairs per cell x 6 cells = 30,000 pairs | 1 | ok |
| 050 | E | Dead-feature necropsy | null | 0.145 | ⬜ T3 | N | 8 w40 training runs x 40 epochs audited | 2 | ok |
| 051 | F | Embedder binding sweep | blocked | 0.220 | ⬛ blocked | Y | 2,000 items + genitive 1,200, per model x 6 models | 3 | ok |
| 052 | F | ★ LASER vs LaBSE: the objective | signal | 0.219 | ⬜ T3 | Y | battery 2,000 + 1,200 items per model | 3 | thin |
| 053 | F | Instructions are inert | null | 0.092 | ⬜ T3 | N | battery 2,000 + 1,200 items x 6 conditions x 2 models =… | 3 | ok |
| 054 | F | The concept-space LM | null | 0.132 | ⬜ T3 | Y | battery 3,200 sentences x 4 representations | 3 | ok |
| 055 | F | Continuous thought | null | 0.243 | ⬜ T3 | N | battery 3,200 sentences x 8 GSM8k representations | 3 | ok |
| 056 | F | ★ Cross-attention doesn't bind either | twist | 0.281 | ⬜ T3 | Y | R-decl 8,000 forced choices | 1 | ok |
| 057 | F | Diffusion latents | null | 0.134 | ⬜ T3 | N | battery 3,200 sentences x 8 representations = 32 rep x… | 3 | ok |
| 058 | F | Speech is text with an accent | signal | 0.166 | ⬜ T3 | N | 3,200 unique battery sentences | 3 | thin |
| 059 | F | Scale moves nothing | null | 0.086 | 🟩 hardened | ? | battery 2,000 + 1,200 items x 4 scales | 3 | ok |
| 060 | F | ★ The judgment anchor | twist | 0.233 | 🟨 partial | Y | 100 rating pairs + 30 forced-choice triplets | 1 | **very thin** |
| 061 | G | ★ Case-marking languages | signal | 0.154 | 🟥 refuted | Y | 1,500 templated stimuli per language x 4 languages | 3 | thin |
| 062 | G | ★ Cross-lingual role transfer | twist | 0.163 | 🟥 refuted | Y | 1,500 sentences per language x 4 languages : 500 train +… | 3 | thin |
| 063 | G | The capacity tax | twist | 0.154 | ⬜ T3 | Y | 64 items per cell x 6 concatenation lengths x 6… | 1 | thin |
| 064 | G | The language vector | null | 0.210 | ⬜ T3 | Y | offset constancy on all 1,012 FLORES devtest rows x 5… | 1 | ok |
| 065 | G | Code-switching monolingualized | twist | 0.220 | ⬜ T3 | Y | 300 code-switched items per cell, 9 cells = 2,700 items,… | 1 | thin |
| 066 | H | ★ The bits budget | twist | 0.295 | ⬜ T3 | Y | 64 items per x 6 lengths = 384 items per language x 3… | 1 | thin |
| 067 | H | The dimension ladder | twist | 0.222 | ⬜ T3 | Y | 192 owt_val chunks x 7 nested prefix lengths per model | 1 | **very thin** |
| 068 | H | ★ The budget is in bits | signal | 0.224 | ⬜ T3 | Y | 64 items per x 7 token bins x 4 tiers = 1,792 items | 1 | thin |
| 069 | H | Conjunction is subadditive | signal | 0.130 | ⬜ T3 | Y | pairs per tier: UNREL 65 / REL 80 / REDUN 96 / SELF 67 /… | 1 | thin |
| 070 | H | Numbers don't cliff | twist | 0.195 | ⬜ T3 | Y | 48 items per cell (digit-count x number-type x context | 1 | **very thin** |
| 071 | H | The entity ceiling | twist | 0.330 | ⬜ T3 | Y | 48 items/cell x 15 cells + REAL cell of 120 FLORES… | 1 | thin |
| 072 | H | ★ What gets deleted first | twist | 0.395 | ⬜ T3 | Y | 48 items/cell x 6 demand levels K=1..6 (288 items | 1 | thin |
| 073 | I | When the order code forms | null | 0.276 | ⬜ T3 | Y | 2000 items per checkpoint | 2 | thin |
| 074 | I | Grokking: none | null | 0.120 | ⬜ T3 | N | 2000 items x 15 checkpoints | 1 | **very thin** |
| 075 | I | ★ Anatomy up to rotation | signal | 0.104 | ⬜ T3 | Y | 8 seeds | 8 | ok |
| 076 | I | ★ Curriculum splits the anatomy | twist | 0.340 | 🟩 hardened | Y | 5 curricula , one run each | 1 | thin |
| 077 | I | Objective hysteresis | twist | 0.421 | 🟩 hardened | Y | 4 conditions , one run each | 1 | **very thin** |
| 078 | J | ★ The operator zoo | signal | 0.125 | 🟩 hardened | ? | 11 transforms | 1 | thin |
| 079 | J | Operators compose | signal | 0.139 | 🟩 hardened | ? | 24 held-out test items per pair x 6 pairs per decode kind | 1 | **very thin** |
| 080 | J | ★ Steering is causal | signal | 0.158 | 🟩 hardened | Y | 90 forward + 50 reverse held-out bases per operator x 4… | 1 | thin |
| 081 | J | When operators install | twist | 0.255 | 🟩 hardened | Y | n_fwd=40 and n_rev=16 held-out bases per operator per… | 2 | **very thin** |
| 082 | K | ★ The cosine gate's blind spot | twist | 0.334 | ⬜ T3 | Y | 96 base/flip pairs per class x 8 classes | 1 | thin |
| 083 | K | Covert capacity, and its catch | twist | 0.207 | ⬜ T3 | Y | 32 held-out test bases | 1 | **very thin** |
| 084 | K | Canaries survive retrieval | signal | 0.207 | ⬜ T3 | Y | 48 canary propositions + 48 role-swap twins in a… | 1 | **very thin** |
| 085 | K | ★ One word breaks RAG | signal | 0.099 | ⬜ T3 | Y | 48 triples x k=0..12 shared-word ladder in a 1248-item… | 1 | **very thin** |
| 086 | K | The ensemble monitor | twist | 0.182 | ⬜ T3 | Y | 96 base sentences x 9 categories = 864 z' scored | 1 | **very thin** |
| 087 | K | Fail-closed decoding | twist | 0.330 | ⬜ T3 | Y | 80 bases x 5 categories = 400 records | 1 | thin |
| 088 | K | Membership is memorization | null | 0.088 | ⬜ T3 | Y | 2000 members / 2000 non-members per ladder arm and… | 1 | thin |
| 089 | K | Style-transparent, identity-opaque | signal | 0.186 | ⬜ T3 | N | per attribute : register 7500, topic 10000, sentiment… | 1 | thin |
| 090 | L | The probe-power kit | signal | 0.115 | 🟦 shipped | Y | synthetic Anchor A n=1000 d=256 | 5 | n/a |
| 091 | L | The canonicalization checker | signal | 0.005 | 🟦 shipped | Y | battery, 4 tasks plus 6 deliberately broken copies | 1 | n/a |
| 092 | L | The norm linter | signal | 0.031 | 🟦 shipped | Y | real anchor: 027/036 SONAR corpus n=29029 z + synthetic… | 1 | n/a |
| 093 | L | Adversarial stimuli evolution | null | 0.238 | ⬜ T3 | Y | 24 codex-generated candidate families | 1 | **very thin** |
| 094 | L | The prereg engine | signal | 0.003 | 🟦 shipped | Y | 4 past rows' frozen predictions reproduced | 1 | n/a |
| 095 | L | The Bayesian ledger | signal | 0.042 | 🟦 shipped | Y | 9 core claims | 1 | n/a |
| 096 | L | Why bag codes win | twist | 0.136 | 🟩 hardened | Y | toy sweep: V in {8,16,32,64,128,256,512} x T in… | 2 | n/a |
| 097 | L | What survives pooling | signal | 0.121 | 🟩 hardened | Y | toy: 60 fixed multisets x 200 permutations each = 12,000… | 1 | n/a |
| 098 | L | The knee from first principles | signal | 0.084 | ⬜ T3 | Y | arithmetic over measured campaign quantities | — | n/a |
| 099 | L | The z-explorer | signal | 0.035 | 🟦 shipped | ? | 5 demos baked from frozen JSON of rows 080/065/071/086/087 | — | n/a |
| 100 | L | TAE-Bench release | signal | 0.132 | 🟦 shipped | Y | packaged stimuli: 2000, genitive 1200, causal 2000,… | — | n/a |
---
## Appendix B: the September 2026 follow-ups ✅ DONE (E05 is pre-registered only and excluded)
Work after the campaign closed, in three threads. **H1–H3** (14 August) are the consolidation
breakers. The **fable loop** (10–11 September; Claude Fable 5.1 with subagents; Codex contributed the
plan breaker and one blinded outcome review) ran five audits of the overnight branches and four
mechanism experiments. The **lit-gap loop** (10–11 September; Codex, with Claude as blind reviewer
and critic) ran the twenty-branch iterative programme derived from the literature review. Tiers are
as in §2.3, evidence flags as in §2.5. Every review in both loops was a model review. Ids in the
first column are the ones used in the body's evidence tags; the full per-unit audit is in
[`SAMPLE_SIZE_AUDIT.md`](SAMPLE_SIZE_AUDIT.md).
| id | thread | question | headline | Brier | tier | §§ | items behind the headline | evidence |
|---|---|---|---|---|---|---|---|---|
| H1 | consolidation | Does the binding null survive an independent re-implementation and a third lexicon? | linear 0.509, MLP 0.495; planted d = 1.0 at 0.973; Japanese third lexicon 0.451 | — | hardened | 4.1, 4.4 | English: 2,000 agent/patient items + 1,200 genitive items | thin |
| H2 | consolidation | Do the operators survive code review and natural text? | core correct; negation peak 0.70 at α = 1; `rt_cos` AUC 0.845; `latent_rewrite.py` | — | hardened | 7.2, 7.3 | Natural bases per alpha cell: negation 40, tense 40, vertical 24… | **very thin** |
| H3 | consolidation | Do the toy predictions hold on the real encoder? | static dispersion 5.1·10⁻⁷; within-multiset 0.997 vs cross-pair 0.523; ~23 PCs; 6× LN gain | — | hardened | 5.2 | Binding stimuli: 2,000 items , per-layer mean-pooled states | thin |
| A1 | fable audit | Meta-prediction: at least one overnight branch is broken | all five branches replicate; every change is a scope change | 0.562 | — | 2.3 | 5 overnight branches | n/a |
| A2 | fable audit | Direct query-conditioned role reader on held-out nouns | CONJ 0.993 [0.989, 0.995], SEP 0.982 [0.975, 0.987]; controls exactly 0.5; closed 4-verb bank | 0.160 | T3 | 4.5 | 200 test propositions on 32 disjoint nouns x 32 sources x 4 rows =… | thin |
| A3 | fable audit | Object-query slot routing on a closed grammar | oracle and routed 2025/2048; direct query 4.9% | 0.303 | T3 | 6.2 | 128 base groups x 4 cyclic rotations = 512 carriers, 2,048 present… | thin |
| A4 | fable audit | Per-sample vs BatchTopK feature stability; S2 headline | count-weighted 0.132 vs 0.111; mass-weighted ≈ 0.46 both; polarity utility 0.59–0.63 | 0.250 | T3 | 9 | 12 SAEs -> 15 seed pairs per family for count-weighted matched… | thin |
| A5 / O1 | fable audit / lit-gap | Operator specificity under a full-proposition rubric | negation full success 15/39, tense 16/38; collateral 47% / 17%; vertical 0/24 applicable | 0.122 | T0 (specificity), T2 (install) | 7.2 | 104 reused H2 natural sources x 2 arms = 208 judged rows | thin |
| G1–G3 | both | GTR/vec2text as a second inversion system | count 60/64, role 40/64; half-norm 3/128 | mixed | T3 | 11 | G1: 64 count sources and 64 role strings = 16 proposition blocks x 4 forms; G3: 128 fresh paired source groups, main domain only (clean 124/128,… | G1 **very thin**; G3 ok |
| E01 | fable | Are final token states a linear n-gram code? | own-token R² 0.370, residual 0.606; random-init twin 0.938; breaker: role-from-odd is a noun fingerprint | 0.230 | T2 | 2.4, 4.3, 5.2 | 8,000 pile-10k sentences / 226,392 tokens | ok |
| E02 | fable | Query-conditioned order reader across constructions | cleft 0.990, objrel 0.75–0.92, nominal 0.80–0.87 on common nouns; random-init ≈ 0.5; L20 → L22 emergence | 0.225 | T2 | 4.5, 5.2 | 2,000 binding_death items | thin |
| E03 | fable | Is the residual a sentence broadcast? | +0.035 trained vs +0.348 random-init | 0.278 | T2 | 4.3 | E01's 8,000 sentences | ok |
| E04 | fable | Is the reader direction a causal handle? | 0.000 at matched dose, 0.072 at 2×; oracle 0.982 | 0.180 | T0 (registered) | 2.4, 4.5 | 37 common-noun test propositions x {active, passive, cleft} x 2… | **very thin** |
| E04b | fable | Is the reader's rank-9 order subspace special to the decoder? | swap 0.140 vs energy-matched structured controls 0.124 / 0.110; isotropic 0.024; oracle 0.995 | 0.150 | T1 | 4.5 | 160 fresh propositions x {active, passive, cleft} x 2 directions =… | thin |
| R01 pilot | lit-gap pilot | Does an additive power certificate cover rotation codes? | global linear 0.507, query-bilinear 0.999 on the same planted rotation code | — | pilot | 2.4, 12 | 2,000 synthetic test examples per seed x load | ok |
| R06 pilot | lit-gap pilot | Latent-access positive control for decode-then-parse | 64/64 correct latent, 0/64 counterfactual, premium 0.953 [0.891, 1.000] | — | pilot | 12 | 64 items = 32 counterfactual pairs, 4 families x 16, only 48 unique… | thin |
| R06/R09 | lit-gap pilot | Frozen-gate degradation audit | 0.94: 127/128 correct, all accepted; 0.8: 106/128 correct, none accepted; counterfactuals 64/64 accepted | — | T3 | 8.2 | Same 64 items x 2 noise seeds = 128 per noise level | thin |
| R16 pilot | lit-gap pilot | Is BatchTopK inference batch-dependent? | random regrouping changes 45% of supports; fixed threshold identical under regrouping | — | pilot | 9 | 2,048 held-out c_pool rows from the historical 20,000 validation pool | thin |
| audit replication | lit-gap | Exact replication of the historical 41–49% flip audit | replicates; gate accepted 0 surrogates; strict pairs 5 and 2; blind review 5/33 clear reversals | ≤ 0.04 | T3 | 8.2 | Exact replication: 1,411 source rows / 211 groups x 8 arms | **very thin** |
| count_capability | lit-gap | Fact-count knee on the count grammar, with beam ablation | 60/25/0 by fact count; beam 5 and 10 do not move it | mixed | T3 | 6.1 | 64 fresh source clusters x 3 fact counts x 2 arms = 384 decodes | thin |
| count_monitor, count_continuous, count_targeted | lit-gap | Self vs source-aware gates; targeted attacks | 0.85 self gate accepts 379 mismatches; source-aware 0 at 50% coverage; targeted 137 vs isotropic 0 | mixed | T3 | 8.2, 8.3 | count_monitor: Main: 300 fresh source clusters x 7 arms = 2,100 outputs (.85 self…; count_targeted: 128 fresh source tuples x 13 arms = 1,664 outputs | count_monitor ok; count_targeted thin |
| count_norm, count_reader | lit-gap | Norm restoration and a frozen field reader | 96 location-only errors → 0 on restoration (+43.0 pp); reader 96/96 and 300/300 | mixed | T3 | 8.3 | count_norm: 300 main + 300 domain fresh sources x 8 arms = 4,800 outputs; count_reader: 300 fresh main sources per field | count_norm ok; count_reader ok |
| lexical_fidelity | lit-gap | Does a length penalty or the source string beat the beam? | source string 0 wins / 45–47 losses | mixed | T3 | 8.2 | 128 OLD calibration source vectors (64 original + 64 fresh | thin |
| binding/multievent, binding/predicate_holdout | lit-gap | Reader ladder, predicate holdout, multi-event decoder witness | order 0.959 vs role 0.526 on a fresh construction; decoder 242/320 vs candidate reader 4.1%; predicate-held-out two-event 0.426 | mixed | T3 | 4.5 | binding/multievent: Candidate-likelihood reader: 200 base propositions -> 3,200 sources…; binding/predicate_holdout: NO RESULT.md | binding/multievent **very thin**; binding/predicate_holdout **very thin** |
| U1, U1B, U2, U3, U4, HISTORICAL_SOURCE | lit-gap | Unbinding depth curves and the historical series | mean-baseline 92% of ridge raw cosine; centring 2.8% → 26.0%; span pooling near ceiling at every position | mixed | T3 | 6.2 | U1: 1,024 held-out test master chains , all slots scored at depths…; U2: 512 of the 1,024 U1 test chains x 16 cyclic rotations = 8,192…; HISTORICAL_SOURCE: Source-only reconstruction of the historical natural-chain depth… | U1 ok; U2 ok; HISTORICAL_SOURCE **very thin** |
| S1, S2, S3 | lit-gap | SAE inference rule, seed pairs, semantic specificity | batch-dependent supports 45–92%; threshold identical under regrouping; utility polarity +0.40, count +0.16, role −0.09 | mixed | T3 | 9 | S1: 8 frozen checkpoints = a census; S2: 6 paired training seeds per family -> 15 dependent seed pairs per…; S3: Bank of 288 clusters x 4 strings = 1,152 strings in 16 lexical blocks | S1 ok; S2 thin; S3 thin |
| O2 | lit-gap | Physical-relation editing on natural text | calibration failed 17/64 then 55/64; 152 final sources never decoded; a 512-source frame bank decoded at 84.6% with zero relation errors | — | unrun | 7.2 | '17/64': operator_transfer calibration, 32 above/below pairs = 64… | n/a |
| multilingual pilot | pilot | FR→ZH direct vs English text pivot | direct = via-English 13–14/24; pivot cosine 0.92; role and number survive | — | pilot | 4.4 | 24 assistant-authored FR/EN/ZH triples | **very thin** |
---
## Appendix C: figures ⬜ TODO (no figures rendered)
No figures are rendered. Each entry names the data that would produce it.
| # | Figure | What it shows | Data source |
|---|---|---|---|
| **F1** | **The certified null and its scope** | Binding battery cells side by side (within-construction ceiling, primary cell, within-vocab flipped parity, genitive control) for linear and MLP readouts with cluster-bootstrap CIs; the planted-signal power curve with the d = 0.70 MDE marker; and the R01 rotation-code panel showing what the additive certificate does not cover. | `../consolidation/H1-binding-breaker/out/`; `next-steps/lit-gap-followup/binding/` |
| **F2** | **Order is stored per noun** | E02 transfer AUC by target construction and training family, common-noun stratum, with the static-bag floors (0.50 / 0.72 / 0.55) and random-init SONAR; inset: layer profile L20 → L22 → L24n. | `next-steps/fable-loop-20260910/experiments/E02-xor-binding/out_full_v3/` |
| **F3** | **The retraction** | Japanese primary cell across three disjoint lexicons (0.79 / 0.696 / 0.451) with the particle-swap and bag-of-chunks tests and the third-lexicon planted recovery at 0.71. | `../consolidation/H1-binding-breaker/out/`; `061-case-marking/out/`; `062-crosslingual-role-transfer/out/` |
| **F4** | **The smuggling channel** | Static vs contextual pooling on fixed multisets (dispersion, within-bag and cross-pair AUC), the order-subspace PC spectrum, and layer-wise β with the final-LayerNorm gain. | `../consolidation/H3-theory-real-sonar/out/` |
| **F5** | **Capacity and rate-distortion** | Three measured knees on one R(D) curve against `L* = C_D / r` at r = 4.5 bits/token with δ = 0.78 annotated; inset: count grammar 60/25/0 by fact count at three beam widths. | `066-bits-budget/out/`, `098-knee-theory/out/`; `next-steps/lit-gap-loop-20260910/audit/count_capability/` |
| **F6** | **The operator algebra, its wall, and its scope** | Left: composition additivity (cosine 0.982, decode gap 0.000). Centre: natural-text dose-response per operator with `rt_cos`. Right: the A5 re-judgement, target-achieved vs full-scope success vs collateral change, with Wilson intervals. | `079-compose/out/`; `../consolidation/H2-steering-buildup/out/`; `next-steps/fable-loop-20260910/audits/operator/` |
| **F7** | **Gates: fail-open, fail-closed, and blind** | Detector ROCs (density gate on gross vs near-manifold; entropy; confidence; NLI below the diagonal; `rt_cos`), and the degradation-audit acceptance-vs-correctness bars at cosine 0.94 / 0.8 / 0.6. | `087-fail-closed/out/`, `029-audit-v2/out/`; `next-steps/lit-gap-followup/audit/out_degraded/` |
| **F8** | **Norm is magnitude, not meaning** | Half-norm location-only errors and their repair by norm restoration, with the frozen reader recovering all 96. | `next-steps/lit-gap-loop-20260910/audit/count_norm/`, `count_reader/` |
| **F9** | **Three ontogenies** | Training fraction on x, normalized readiness on y: SAE atom inventory, surface-order code, operator causal usability. | `048-atom-ontogeny/out/`, `073-order-code-formation/out/`, `081-operator-install/out/` |
---
*End of version 0.3.*
