The short version
Two questions, one base model, five families of "reader," and a causal follow-up. Here is where they landed.
On Llama-3.2-3B, the model appears to plan a topic switch just-in-time — re-deriving the next topic from the prompt as it writes, holding only a weak choice-bias in its activations. A parascope reads the current generation state richly; the specific future is decodable only in the one setting the prompt can't reveal, and even there it isn't causally load-bearing. This reproduces "Where's the Plan?" (decodable ≠ causally used) on a new task design.
What a ParaScope is
At a paragraph break, a transformer's residual stream holds a compressed sense of what comes next. A ParaScope (Pochinkov, arXiv:2511.00180) turns that hidden state into the upcoming paragraph. Two shapes:
Continuation ParaScope — patch the break-token residual back into the model as context and let it generate. AutoEncoder-Map ParaScope — a trained map from residual → SONAR sentence-embedding space, then decoded to text. Both recover next-paragraph information worth roughly five tokens of "cheat" context at the 3B scale.
This project asks two things the original paper didn't. Q1 — move the read from the paragraph break to mid-paragraph, and from open text to controlled "talk about X, then Y" pairs: when does the future topic become readable? Q2 — stop training a probe and instead LoRA fine-tune the model itself into a reader of its own state: how good can decode get?
X-then-Y: is the plan there while it writes?
2,400 two-paragraph documents across four prompt families (explicit X→Y, derived-Y, secret-choice, model-chosen). Six layers captured at every paragraph-X token; positions binned 0–9 within the paragraph. Five independent reader families, one verdict.
Every reader agrees: the current topic X is strongly readable everywhere in the paragraph; the specific upcoming topic Y is linearly near-absent until the model actually gets there.
| Reader family | Current-X readout | Upcoming-Y readout |
|---|---|---|
| mini J-lens 64-topic pool | 0.66–0.72 acc (0.016) | 0.006–0.026 |
| Patchscopes patch L2 / L14 | 0.10–0.17 (≈10× chance) | ≈ chance |
| champion maghead probe per-token | cos 0.18–0.25 | cos 0.13–0.17 · r1 ≤0.006 |
| fresh ridge → SONAR(Y) | — | cos ~0.57 · r1 ~0.00 (collapse) |
| fresh logistic · secret-pick 4-way | — | 0.40–0.42 (0.25) |
Prominence = 1 − rank/63 (top of the 64-topic readout = 1.0) at each generated token. The current topic sits at the top of the J-lens readout across the whole paragraph; the upcoming topic stays low and only snaps upward at the ¶ paragraph boundary — often before its first content word.
Three things worth noticing
No prompt echo. Even when Y sits verbatim in the prompt (the explicit family), Y's rank-1 at the prompt's last tokens is 0.004. The parascope reads generation state, not the prompt text.
No boundary crystallization. At the exact final token of paragraph X — the probe's own training position — the champion still reads the just-finished X (cos 0.251) over the imminent Y (0.165, rank-1 0.005). A hard topic switch exposes that the boundary residual does not linearly encode the switch in SONAR space; the paper-era rank-1 on FineWeb is likely carried substantially by topic continuity between adjacent paragraphs.
One real plan signal. The exception is the secret pick — the family where the prompt lists all four options but only the model knows which it chose. There a fresh probe reads the choice at 0.40 vs 0.25 chance (best at L24/L27). That is the only place a genuine, prompt-independent forward plan shows up.
Is that plan causally load-bearing? No — at 3B.
The secret pick is decodable. So the sharp question: if you edit that representation, does the model change its mind? Intervention: add α·(μtarget−μorig) — slot-mean differences from mid-paragraph-1 captures — to all paragraph-1 positions during prefill, generate paragraph 2, parse the revealed pick. n=120 per arm.
| Config | base keep | steer→target | steer keep | ablate keep |
|---|---|---|---|---|
| L24,27 · α=4 | 0.650 | 0.100 | 0.583 | 0.692 |
| L24,27 · α=8 | 0.617 | 0.117 | 0.525 | 0.642 |
| L14,21,24,27 · α=4 | 0.558 | 0.092 | 0.567 | 0.625 |
Steering sits at chance; ablating the 3-dim pick subspace changes nothing. And base_keep is only ~0.6 at temperature 0.3 — the pick is substantially re-decided during paragraph 2. The mid-paragraph signal is a weak bias, not a locked plan.
| Config | steer→target | steer keep | steer derail |
|---|---|---|---|
| L24,27 · α=4 · all | 0.100 | 0.542 | 0.108 |
| L24,27 · α=8 · all | 0.083 | 0.483 | 0.133 |
| L14,21,24,27 · α=6 · all | 0.100 | 0.358 | 0.358 |
Higher α only derails the model (0.36 derail at 4-layer α=6) without ever flipping it to the target. The slot-mean direction isn't even write-capable — it's a pure read-only correlate of the pick. Decodable (0.40 vs 0.25), but neither causally load-bearing nor a usable control handle in diff-of-means form. This is the read≠write gap, and it directly reproduces "Where's the Plan?" on a new task.
LoRA continuation readers: how far can decode go?
Instead of a probe that maps a residual to a SONAR vector, LoRA-fine-tune the model into a reader of its own boundary state (LatentQA-style). A soft-projector turns donor residuals into a few soft tokens; the model generates the next paragraph from them. Reader = the model decoding itself. Scored on val shard 8, n=300, against the full-shard retrieval pool.
| Reader | Data | SBERT cos | SONAR cos | rank-1 |
|---|---|---|---|---|
| zero-shot continuation patch L14/21/27 | — | — | 0.000 | |
| LoRA patch L14/21/27 r32 | 8sh · 1ep | — | ||
| LoRA soft-projector L27 k8 r32 | 8sh · 1ep | 0.388 | ||
| LoRA soft L27 k16 r32 | 8sh · 2ep | 0.407 | ||
| LoRA soft ml3 L14/21/27 k16 r32 | 24sh · 1ep | 0.457 | ||
| LoRA soft ml3 L14/21/27 k16 r32 | 24sh · 2ep | 0.494 | ||
| LoRA soft ml6 L9–27 k16 r64 | 24sh · 2ep | 0.510 | ||
| LoRA soft ml3 L14/21/27 k16 r32 ★ decode SOTA | 24sh · 3ep | 0.549 | ||
| SONAR maghead-MLP champion probe → decode | 24sh | — | ||
| SONAR roundtrip ceiling decode true vector | — | — | — |
What the ladder says
Training the reader is everything. The zero-shot continuation parascope on Llama-3B is weak when patching only three layers (0.087) — the paper patched all of them. Tuning the reader takes decode from 0.087 to 0.670.
Epochs beat depth and rank. Holding the ml3/24sh recipe fixed and varying only epochs gives a clean, monotonic curve — 1ep 0.548 → 2ep 0.596 → 3ep 0.670 SBERT (rank-1 0.210 → 0.313 → 0.393), still rising on both axes and the SBERT gain even accelerating (+0.048 then +0.074). A third epoch (0.670) beat a deeper, twice-as-expensive six-layer rank-64 reader (0.623). Retrieval wasn't stalled — it was undertrained. A 5-epoch run is in flight to see whether 4–5 epochs keep climbing or saturate.
Multi-layer input was the unlock, same as for the probes: single-layer donors plateaued at val CE ~1.18; the three-layer donor stack dropped it to 0.59.
A detached auxiliary cosine head bolted onto the reader does not retrieve: rank1head lands at 0.10 vs the text path's 0.25 on the same checkpoint, despite a healthy SONAR cosine of 0.54. High cosine, no retrieval — cosine objectives mean-collapse, the repo's oldest lesson. Retrieval-grade vectors need InfoNCE, which is exactly what the maghead champion runs on raw residuals. Decode and retrieval want different losses.
The LoRA reader wins decode quality at one-third the data (readable English about the right topic); the probe wins retrieval (pin the exact next paragraph out of thousands). Decoder-side capacity was the binding constraint on decode — and it's cheap to scale.
Real decodes from the champion reader (ml3 3-epoch) on held-out validation paragraphs — the model reading its own boundary state and writing the next paragraph. Each dot is one example, placed by its SBERT similarity to the true continuation. Click along the range to see the spread behind the 0.67 mean.
Pattern across the range: the reader reliably recovers genre, structure, register and domain — often the exact opening words — while specific named entities, dates and numbers are where it diverges. Decodability is real but bounded; next-paragraph is one-to-many.
The probe leaderboard, and the magnitude wall
The AutoEncoder-Map side of the project: linear and MLP probes from residual → SONAR, scored by rank-1 retrieval and by decoded-text quality. This is the retrieval SOTA lineage the LoRA reader is measured against.
| Probe | Layers | rank-1 | dir-rank1 | decode | % of ceiling |
|---|---|---|---|---|---|
| llama-3b maghead linear | L27 | 0.594 | 0.628 | 0.363 | — |
| llama-3b maghead-MLP h2048×2 | L27 | 0.631 | 0.646 | 0.371 | — |
| llama-3b ml3 multi-layer | 14,21,27 | 0.648 | 0.656 | — | — |
| gemma-27b single | L58 | 0.658 | 0.676 | 0.343 | 43% |
| gemma-27b ml3 ★ overall SOTA | 50,54,58 | 0.676 | 0.693 | 0.368 | 46% |
The story in three moves
Data, then architecture, then base-model size. Doubling data lifted rank-1 ~0.06 at every layer, then saturated (24→32 shards = +0.005). An MLP direction head broke the linear rank-1 ceiling without losing decode coherence. A 9× bigger base model (gemma-3-27b) predicts the next-paragraph direction best of all. Multi-layer input, model size and data all stack: rank-1 0.658 → 0.661 → 0.676.
Decode doesn't improve in absolute terms with a bigger model — both llama and gemma sit at ~42–46% of their respective SONAR-roundtrip ceilings. The gap is magnitude: a linear d→1 magnitude head explains only ~30% of variance (llama) to ~70% error (gemma). The direction is already good; the size of the SONAR vector is the wall. That's why decoder-side levers — LoRA readers, better magnitude heads — are the path forward, and why next-paragraph decode is partly task-intrinsic (it's one-to-many).
For reference: the paper's 24-sublayer Llama-3B TAE ParaScope reports ~0.55 mean cosine; a single-layer L27 mse+zscore probe here reaches 0.514 val-cos — about 93% of the paper's signal from 1/24 of the input.
The reading toolkit: 2023–2026
Every technique that could, in principle, read an upcoming topic from activations while the model writes about something else. Filter by family; the rating is each tool's fit for the X-then-Y read.
Full 35-paper review with per-tool notes and the ranked shortlist: TECHNIQUES.md in the repo. Applicability rating = expected signal-to-effort for reading upcoming topic Y mid-paragraph.
What it all means
1. Llama-3B plans just-in-time. It re-derives the next topic from the prompt as it writes (Wu et al.'s "breadcrumbs" world), holding only a weak choice-bias in activations. The rich thing a parascope reads is the current state; the specific future is mostly reconstructed on the fly.
2. The decisive follow-up needs a bigger model. "Where's the Plan?" found Gemma-3-27B is the exception where boundary plans are causally used — and gemma-27b is also this repo's retrieval SOTA. Re-running the secret-choice causal protocol on gemma-3-27b (an A100-80G job) is the single highest-value next experiment.
3. Decode was capacity-bound, not information-bound. The LoRA reader's decode win at one-third the data says the binding constraint on decode quality was decoder-side capacity — cheap to scale, and it pushed well past the probe. Retrieval and decode want different losses (InfoNCE vs the reader's language-model objective); the best system likely uses both.
InfoNCE-style auxiliary loss to close the LoRA retrieval gap · all-layer donors · the gemma-27b causal port · sentence-unit probes · a future-supervised concept bottleneck (Predictive Concept Decoders) as a principled "SAE that reads ahead."
Methods & disclosure
Base model. Llama-3.2-3B-Instruct (28 layers, dres=3072). Sentence embeddings via SONAR (1024-d), decoded with the SONAR text decoder. Retrieval pools use the full validation shard (~10.3k paragraphs). Probe retrieval and LoRA-reader retrieval use the identical protocol (SONAR-encode the output, rank against true targets) so numbers are comparable — with the caveat that the LoRA path is scored text→SONAR-re-encode (lossy) while the probe is scored on its raw output vector.
X-then-Y dataset. 2,400 two-paragraph documents, four prompt families, ~78% compliant. Residuals captured at six layers across all paragraph-X tokens, binned 0–9 by within-paragraph position. Causal tests use slot-mean difference vectors added during prefill (and, as a control, at every generated token). Compute: 3× A4000 16GB.
This site and the underlying experiments were produced by an AI research agent. Results are from real training runs and evaluations on the hardware described; the writeup has been AI-compiled and AI-fact-checked against the project's result logs, not independently peer-reviewed. Numbers reflect single runs at the 3B scale and should be read as directional, not definitive. Treat conclusions as a research snapshot, not settled findings.