What to do
next.

Automated TAE interpretation: things to do

Consolidated 9 September 2026. This combines the four project pages you confirmed, the research/status audit, the SAE review, the 4l inventory, and your multilingual-embedding idea. It is a prioritized working plan, not a claim that the proposed experiments have run. Completed during this synthesis: a 24-item SONAR multilingual pilot, run with greedy decoding and beam size 5 on 4l.

The direction I recommend

Build a benchmark that rewards explanations for predicting unseen behavior, then use it to compare interpretation methods on SONAR and one accessible alternative. Reuse the existing archive. The most useful new exploratory branch is multilingual information retention: what survives direct translation, what an explicit English pivot removes, and whether a language-specific transformation is useful at all.

Keep SONAR as the multilingual baseline. Try QwenAR next for reconstruction and out-of-family interpretability, and LatentSeal for a different bottleneck size and noise-robustness objective. Neither is presently established here as a multilingual SONAR replacement. No omniSONAR migration or reproduction project is on the immediate list.

If choosing just three next work sessions: validate the benchmark's positive and negative controls; examine the multilingual pilot and design its held-out extension; fix SAE inference calibration before comparing existing dictionaries.

What the four pages contribute—and what needs updating

Supplementary evidence: status audit, SAE review, remote file inventory, H1–H3 consolidation, FABLE extension debrief, and TRACE/NIGHT8 report. Where reports disagree, trace the exact assay and result file; the newest narrative is not automatically the most accurate.

What is already done

• Campaign100 has 100 recorded experiment rows and H1–H3 consolidation. Some rows ended in instrument failure or a blocked scientific question; “ran” does not mean “answered.” The paper remains unfinished, with no completed publication-figure set and an empty related-work section in the inspected draft.
• TRACE's corrected strongest-baseline comparison has not established a robust automated-discovery advantage. One corrected full-baseline premium is 0.00 with a wide interval, approximately [−0.167, 0.222]. The earlier apparent advantage exploited unequal access to raw input and labels recoverable from surface text. This is a useful benchmark lesson, not evidence that interpretation can never help.
• H1's English cross-vocabulary/construction results are near chance: linear AUC 0.5092, MLP 0.4952. Its additive positive control supports the linear assay at the stated effect size; the MLP fails the same-amplitude gate. Japanese positives did not survive fresh-lexicon testing. These are reader- and split-specific results.
• Query-conditioned readers and decode-then-parse already have positive results in the earlier FABLE program. Recover their exact settings and reconcile them with H1 before describing conditional probing as new work or claiming role information is absent from z. This corrects the initial status note's presentation of that proposed next experiment.
• H2 already implements monitored latent rewriting. Natural-text results include negation success around 0.70 and tense around 0.85 in selected dose windows. Above/below is strongly context-dependent. A round-trip cosine detector was calibrated on its own sweep; independent calibration and evaluation are still needed.
• H3 finds highly readable order within fixed token multisets but weak transfer across pairs. This supports content-dependent readout structure; it does not uniquely identify the causal mechanism. The older binding_death localization attempts had instrument failures and should not be cited as successful causal localization.
• Matryoshka was already tried locally. Its saved full-dictionary FVU is approximately 0.3104 versus 0.311 for plain BatchTopK in the running report. That exploratory reconstruction tie does not settle feature quality. W40 already includes widths and seed pairs; matched-topic null outputs also exist despite older “owed” notes.

Priority 0: make the existing evidence usable

• [ ] T01 — Make a reproducible asset/claim manifest. Start with the existing 4l inventory. Record checkpoint type, exact path, model/config revision, normalization, dataset/split, script version and result provenance. Load one representative TRACE checkpoint, one c_pool SAE and one whole-z SAE; verify dimensions and a known reference output. Investigate missing smaller whole-z paths before assuming retraining is necessary. Do not mix c_pool with pooled z.

Deliverable: a manifest and a small reproducible load report. Done when: every asset selected for the first comparison loads and reproduces its reference behavior, or is explicitly excluded. CPU/cached work first; short GPU checks only where needed.

• [ ] T02 — Reconcile claims, paper and board. Map FABLE conditional-reader positives against Campaign100/H1 negatives by task, query information, construction, vocabulary and power gate. Correct stale claims about matched-topic controls, the 16% atom-energy “ceiling,” high-rank estimates and failed localization. Split fabrication into gross perturbation detection versus subtle meaning changes. Keep the approximately 460-bit capacity figure explicitly post-hoc and substrate-specific.

Deliverable: one claim table with supported scope, counterevidence, result paths and outstanding checks. Recompute the historical ledger only after correcting its inputs; label its values as heuristic evidence aggregation, not calibrated scientific probabilities. Done when: abstract, limitations and board agree on each claim's scope. Formal tier promotion remains an evidence decision, not an automatic bookkeeping task.

Priority 1: the automated-interpretation benchmark

• [ ] T03 — Freeze a small benchmark contract. Use TRACE for trained organisms with testable internal claims; GAUGE for synthetic instrument checks; SIEVE for encoder capabilities. Reuse TAE-Bench's stimulus and measurement code without conflating these three purposes.

Define three separate outcomes: prediction of unseen feature activation; prediction of a latent intervention's effect; and practical improvement from latent access over matched text access. Freeze what each method can see, its query/compute budget, test splits, primary metric and minimum useful effect before comparison.

Require a privileged positive control that can actually win, alongside shuffled/corrupted explanations, random directions, no-op interventions and strong surface baselines. Hold out propositions, templates and vocabulary; cluster uncertainty by independent family/model rather than treating paraphrases as independent. Use private organism variants for final testing.

Deliverable: a specification, a compact frozen dataset and a control report. Done when: the positive control clears the prespecified effect threshold and known-bad methods fail for the intended reason. If it cannot distinguish these, fix the benchmark before scaling methods. Depends on T01.

• [ ] T04 — Compare four explanation methods at matched budget. Start with activation-example explanations, intervention/output-based explanations, a combined method, and an adaptive agent that tests competing hypotheses. Give the adaptive method an equal-cost nonadaptive comparator. Reuse the flipbook code where appropriate.

A pilot of 100–200 stratified features is a planning suggestion, not a power calculation. Include frequent and rare features, lexical and paraphrase-stable ones, non-steerable features and corrupted controls. Require every explanation to specify its predicted behavior, applicability and uncertainty. Report held-out predictive loss, confident errors, coverage, intervention success, collateral changes and cost separately. A readable description or fluent decode alone is not success.

Deliverable: one comparison table with uncertainty, failure examples and a small blinded meaning/fluency audit. Done when: the experiment is powered for the chosen effect and replicated on a second trained substrate, or yields an appropriately scoped negative. Depends on T03 and, for SAE comparisons, T06. Do not expand dictionary width merely because the explanation detector score rises.

Priority 1: multilingual embeddings and your English-pivot idea

• [ ] T05 — Test shared meaning versus an English bottleneck. The initial pilot is complete using the existing SONAR encoder and decoder on 4l; see results and interpretation and pilot code. Direct French→Chinese preserves simple role and number contrasts without an explicit English text intermediate. Adding an English pivot changes outputs, but this small sample does not establish a better route or an internal English bottleneck. A greedy omission disappeared under beam search. The held-out extension below remains to do.

Compare D_zh(E_fr(x)) with D_zh(E_en(D_en(E_fr(x)))). Include an extra French round trip and an extra Chinese round trip, because any decode/re-encode step can lose information. Also compare embeddings of independently supplied equivalent French, English and Chinese texts, with same-topic wrong-meaning minimal pairs.

A successful direct French→Chinese decode shows that a separate explicit English-text step is unnecessary. It does not show the encoder's internal computation contains no English-associated structure. Conversely, matching an English translation's embedding does not make the vector literally an English encoding. Geometry alone cannot establish a unique internal representational language.

Next extension: collect independently checked translations and targeted information-loss contrasts: grammatical gender where English leaves it unspecified, formal/informal address with singular/plural context resolved, tense/aspect, negation and who-did-what-to-whom. Score meaning with bilingual annotation, allowing valid alternatives. Treat a backtranslation or SONAR cosine as supporting diagnostics, not independent ground truth.

Then test a vector translator: on a sufficiently large disjoint parallel training set, compare the identity map, mean language offset, orthogonal Procrustes and regularized linear map from French vectors to English-reference vectors. Tune only on validation; split by meaning/template, not paraphrase. Evaluate held-out Chinese decoding and retained distinctions as well as geometric fit. Compare French→Chinese-reference and French→English-reference alignment to test whether English is privileged. An improvement would establish utility of that fitted transformation, not an internal English pivot. Do not fit a 1024-dimensional map on this tiny pilot and call it generalization.

Deliverable: a route-by-phenomenon report, full generated texts, embeddings and a held-out mapping experiment. Done when: the routes' information loss is quantified independently of the encoder's own similarity metric. The pilot can finish now; the larger study depends on validated translations and T03-style controls.

Priority 1: SAE methods with a trustworthy comparison

• [ ] T06 — Repair inference calibration, then compare BatchTopK and Matryoshka. The local BatchTopK encoder applies batch selection at evaluation, making a sentence's activations depend on its batch companions. Implement a calibrated fixed inference threshold and verify held-out batch invariance, realized L0 distribution, FVU and dead features. Keep the original outputs/version for provenance.

Use plain BatchTopK as baseline and a faithful Matryoshka+BatchTopK recipe as challenger. Audit AuxK and the relative weighting of prefix losses; the local Matryoshka loop omits AuxK. Match data, normalization, training budget, sparsity and at least two seeds at one existing width. Evaluate feature prediction, paraphrase consistency, absorption/hierarchy controls and causal edits with content preservation, not reconstruction alone.

Deliverable: an audited inference implementation and one controlled comparison linked to T04. Done when: any claimed feature-quality gain survives held-out evaluation and seed uncertainty. If resources allow, test retained-seed ensembles at matched total width and active-feature budget before training bigger dictionaries. Matryoshka+JumpReLU, Group Bias Adaptation and Matching Pursuit are later challengers, not prerequisites. Detailed sources and local code audit.

Priority 2: the most informative scientific extensions

• [ ] T07 — Repair the binding assay, then test a cause. Reproduce the existing conditional readers and decoder parser on identical items. Add calibrated role/filler-rotation or tensor-product positive controls alongside additive shifts. Verify power per reader and per encoder, including the MLP. Rotate focal identities and hold out constructions and vocabulary. Only after this works, compare reconstruction and relational objectives at matched capacity/data. Row 009 needs a verified binding teacher; row 051 needs per-embedder calibration. A failed gate ends the inference, not the scientific question. Depends on T02.

• [ ] T08 — Test decoder dependence and selective auditing. Compare the base SONAR decoder with an accessible noise-robust decoder checkpoint if its actual availability/compatibility is verified. Reuse H2 edits; include near-manifold role/negation changes, natural text, random/no-op edits, and mean/shuffled latent baselines that expose decoder priors. Calibrate abstention on a separate set and report risk versus coverage on held-out domains. Check full proposition preservation. A cosine threshold is not a universal safety certificate. Deliverable: decoder × perturbation × monitor table, including failures.

• [ ] T09 — Try one accessible alternative TAE. QwenAR is the first reconstruction candidate; LatentSeal is the first robustness candidate. First reproduce each release's installation/reference checks in a separate environment. Then run identical fresh English reconstruction, binding, noise and editing tests, with length and domain stratification. Include model size, latency and inference memory; compare usable quality rather than assuming equal vector dimension means equal capacity. Evaluate the text bottleneck alone, ensuring no original-text bypass. Deliverable: a small encoder/decoder capability table and one chosen second substrate for T04. Multilingual support must pass a separate test.

• [ ] T10 — Finish the strongest remaining controls only if they affect a claim. For operator algebra: held-out constructions, composition order, inverse operations, random norm-matched directions and collateral changes. For capacity: fixed decoder/inverter, explicit distortion definition and fresh length curves; test the post-hoc ceiling on new data. For geometry: directly measure anisotropy and compare its predicted similarity floor with observed “unbinding,” rather than importing a universal VSA floor. For dictionaries: matched-frequency/topic comparisons and seed/subspace stability using the existing W40 outputs. Rank these by which paper claim survives or falls on their outcome.

Priority 2: write and package what is defensible

• [ ] T11 — Finish the paper from the reconciled claims. Add related work before novelty language: earlier sentence-role probing, ROLE, conditional probing, SONAR/LCM and current automated-interpretation evaluations. Make figures directly from provenance-linked data: binding plus power controls; capacity with censoring/scope; natural editing doses; auditing risk/coverage; dictionary seed stability; and benchmark comparison. Preserve negative and instrument-failure rows. A ledger visualization is supplementary evidence accounting, not the main result. Depends on T02; new-results sections depend on the relevant tests.

• [ ] T12 — Prepare a reproducible local release candidate. Document datasets, model requirements, licenses, pinned environments and CPU/small-GPU smoke commands. Validate in an isolated fresh checkout. Fix package-path assumptions and stale control claims. Include examples of both a valid result and an instrument failure. Public publishing is a separate step; this plan does not send messages, publish, or authorize a paid compute campaign.

Human calibration from row 060 remains real unfinished work: its existing panel was an LLM proxy. Use human annotation first where it resolves the multilingual and meaning-preservation labels, rather than adding a disconnected general similarity study.

What 4l already gives us

Paths below are relative to /workspace/HOME/guest/ on 4l; sizes are approximate audit snapshots. File existence was checked, not universal checkpoint loadability.

The audit saw four 16 GB A4000s available at the instant checked, not a standing reservation. Begin with cached CPU analysis and short decoding jobs; measure throughput before scheduling training. No inference from an idle snapshot to a promised training completion time is intended.

Model choice and the parked omniSONAR discussion

OmniSONAR stays a reading item. No public checkpoint/API was verified in the access audit; an author said on 31 July that release dates—and release itself—could not be promised. Author comment.

For reference, its paper reports a 1.5B text encoder and 1.8B decoder with a 1024-D bottleneck. Headline SONAR→OmniSONAR comparisons include FLORES xsim++ error 15.3→6.1, a selected MTEB average 63.338→74.114, and FLORES translation chrF 52.9→55.4. These establish useful benchmark gains, not improved SAE interpretability or exact English reconstruction. The recipe combines LLM initialization, translation, pooled/contrastive learning, hard negatives and language expansion. Paper.

The earlier $20k–100k successful-text-run estimate, centered roughly on $50k, was a speculative planning range, not Meta's reported bill or a reproduction quote. It assumed 128 GPUs for 125k core steps at 1–5 seconds/step (about 4.4k–22.2k GPU-hours), added the reported expansion stage and uncertain overhead, and used illustrative public A100 rental prices. It excludes original Llama pretraining, data production, speech, the distilled-model suite, research reruns and labor. Timing and hardware assumptions dominate the uncertainty. It is not a reason to rebuild an unavailable model for this project. Pricing examples used, Runpod cost guide.

Reading tied to decisions

Do not turn the whole bibliography into prerequisites. CHIVE, Pitfalls and the output-centric paper are enough to begin the interpretation benchmark; BatchTopK, Matryoshka and the reliability audit are enough to begin the SAE comparison.

Deferred until a smaller experiment justifies them

Another broad 100-row campaign; a larger SAE merely to improve description scores; a from-scratch omniSONAR reproduction; sweeping universal claims about pooled representations; and moving wholesale to latent-sequence/diffusion models. The latter are interesting, but change the experimental object. A deterministic embedding of a complete input cannot add unrestricted Shannon information to that input: practical uplift must be defined through constrained access, computational budget, or model-specific internal/causal predictions.