Experiments and writing plan
Companion to the manuscript. This plan is not a preregistration and starts no compute jobs. Main goal: a validated small benchmark, a common method comparison, and two explanatory case studies within nine pages.
Prioritized experiment queue
Start with N01, then N02 and N03. The main missing result is a fair comparison on shared data. Existing figures already support the two case studies; a new decomposition method or a complete account of residual meaning is not a prerequisite.
| Order | Work | Question / required output | Reuse and dependency | Priority |
|---|---|---|---|---|
| 1 | N01: benchmark contract | What can each method see, predict, and change? Produce the task/interface/baseline matrix and frozen scoring definitions. | Existing task generators and access audits; design work before final runs. | Core |
| 2 | N02: data and decoder checks | Can the substrate express each tested distinction? Produce shared train/calibration/test banks, native/target decoding results, and a semantic-scoring audit. | Retained SONAR pipeline and stimuli; after N01. | Core |
| 2 | N03: toy calibration | Does the evaluation recognize a correct interpretation? Produce results for rotated components, a known interaction, and bad/null interpretations. | GAUGE; after N01; can develop alongside N02. | Core |
| 3 | N04: common method pilot | Do the tasks have usable measurements and controls? Produce a pilot table for simple probes/operators, PCA/random controls, an existing SAE, and a structured composer on supported tasks. | Retained assets; after N02/N03. | Core |
| 4 | N05: final comparison | What capabilities do current methods provide on new cases? Produce the frozen six-task table, generalization results, per-family uncertainty, and Figure 2. | Pilot fixes first; final data remains separate. | Core |
| 4 | N06a/b: case-study extensions | Which patterns explain the table? Test algebra under shared semantic scoring and connect the readout/control gap to that protocol. | Existing Figures 3–4 and original outputs; after N02/N04, coordinated with N05. | Core |
| 5 | Second trained substrate | Does the central comparison generalize beyond SONAR? Reproduce the selected main contrast with a decoder-capability check. | N05 protocol; choose a compatible retained system first. | Desirable; otherwise scope claims to SONAR |
| Separate | N07: residual pilot | What useful information survives the fitted lexical account? Produce one controlled readout/transfer result with lexical and decoder controls. | N02 scorer and E01/E03 assets; does not block N05. | Optional |
All new-run entries above remain [needs to be run]. N01 is [needs to be improved] because the draft contract exists but is not frozen. N08 below covers writing, reproducibility, and the final nine-page manuscript rather than another experiment.
First work session: concrete deliverables
- Write
protocol/contract_v1.md: six tasks, exact allowed inputs/outputs, label access, query budgets, primary metrics, and which methods enter each task. Explicitly identify supervised, decoder/text-reference, and privileged-oracle methods. - Write
protocol/assets_v1.json: chosen checkpoint paths/revisions, vector normalization, SAE substrate/inference rule, and decoding settings. Verify retained assets before scheduling new training. - Write
protocol/data_plan_v1.md: content/relation/polarity contrasts, two-sentence composition, independent family splits, natural-text extension, and the semantic annotation rubric. Specify a pilot cohort; use pilot variance to size the final study. - Write
protocol/controls_v1.md: exact toy codes, privileged positives, shuffled/no-op/random controls, and sensitivity checks required for negative conclusions. Known targets do not imply known mechanisms on real TAEs. - Begin N02/N03 development against that contract. Save their outputs and failures; keep the final evaluation unopened. These paths are proposed deliverables, not files already created.
The first session is successful when the experiment table has an executable interpretation and the control/data pilot can start. It does not need a new scientific discovery.
What to save from every run
Use separate directories such as runs/N03/pilot_v1/ and runs/N05/final_v1/, retaining the original archive. Before execution, record the hypothesis, input/split hashes, checkpoint/configuration, frozen metrics, stopping conditions, and expected resource budget. After execution, retain per-item predictions or edited vectors/decodes, aggregate results, uncertainty method, source/code hashes, actual resource use, and a short scoped result report. Sensitive raw inputs should not automatically enter the public publishing manifest.
Early work is mainly protocol/data preparation and cached CPU analysis. Native decoding, SAE inference checks, and final vector interventions require short measured GPU pilots before estimating the full budget. No GPU-day estimate or external judging budget is committed by this plan.
Decisions that control scope
- If a privileged control fails, repair the instrument before interpreting real-model negatives.
- If native decoding cannot express a contrast, report capability/coverage explicitly; do not attribute the failure to interpretation alone.
- If a pilot metric saturates or measures a shortcut, revise it on development data and freeze a fresh final protocol. Do not remove final tasks because their results are inconvenient.
- If methods tie under valid controls, report the tie and uncertainty. A positive advantage is not a required outcome.
- If residual structure is inconclusive, keep it as an open question. Avoid expanding into a new SAE training campaign to rescue the narrative.
- Finish with one common comparison and two case studies. Full capacity, multilingual, and training-dynamics sweeps remain outside this paper's core plan.
Page and figure allocation
| Manuscript section | Pages, including figures | Main artifact |
|---|---|---|
| 1. Introduction + abstract | 1 | Figure 1: testbed/contract |
| 2. TAE-Bench | 2 | Six-task contract table |
| 3. Validation/setup | 1 | Control summary |
| 4. Method comparison | 2 | Figure 2: capability profile |
| 5. Two case studies | 2 | Figures 3–4: algebra; readout/control gap |
| 6. Discussion/related work/limitations | 1 | Scoped interpretation and open questions |
| Total | 9 | References and appendix additional |
The HTML is a working reading version, not a verified nine-page PDF. Freeze a venue template and measure pages at N08. Residual exploration belongs in a short open-question paragraph unless it earns a case study through clear evidence. Do not expand to a third case study.
N01 — Freeze the benchmark contract [needs to be improved]
Supports: §§2–3. Reuse: SIEVE task ideas, GAUGE authored controls, TRACE access lessons, Campaign100 stimulus/checkpoint records.
Work: Choose exact checkpoints and six task definitions. Specify training/test inputs, outputs, semantic supervision, encoder/decoder budgets, task-specific baselines, unsupported method cells, primary estimands, minimum useful effects, and complexity accounting. Keep fixed-model method evaluation separate from encoder comparison. Resolve the current source-text versus latent-only access question per track. Supervised/privileged methods are labeled.
Done when: one versioned contract and method-interface table determine every planned table cell without consulting final labels. No automatic score exclusions based on poor method performance. No claim of information beyond complete text without a computational/access qualification.
N03 — Known-code calibration [needs to be run]
Supports: §3. Depends on: N01. Can develop alongside N02.
Work: Instantiate a rotated slot-component code and a code with a known interaction. Include interference/noise. Privileged controls can access the generating mechanism; ordinary competitors cannot. Exercise the exact final scorer and method adapters. Test shuffled explanations/labels, irrelevant components, random directions, and no-op edits. Accept behaviorally equivalent factorizations.
Done when: correct controls succeed where information is recoverable; known-bad methods fail for the intended reasons; relevant sensitivity limits are measured. A failed control invalidates the inference, not the biological/linguistic hypothesis. Existing GAUGE scores do not substitute for this run.
N04 — Common pilot on retained assets [needs to be run]
Supports: §§3–4. Depends on: N01–N03.
Work: Verify checkpoint loading and inference, then run probes/operators, PCA/random dictionaries, retained SAEs, and a structured composer on their supported tasks. Keep whole-z and c_pool dictionary substrates separate. Check inference batch dependence, realized sparsity, and training/inference budgets. Do not train a new SAE family merely because an old score looks low.
Done when: the pipeline is executable, task controls pass, expected ceilings have headroom where appropriate, uncertainties can support sample-size decisions, and unsupported cells are declared. If every cell saturates or every privileged control fails, revise the benchmark using pilot data only. Archive revisions before freezing a new final set. Honest method ties remain results.
N05 — Frozen comparison and generalization [needs to be run]
Supports: §4 / Figure 2. Depends on: N04 and a fresh final split.
Work: Execute the common comparison under fixed budgets. Save predictions before revealing counterfactual outputs. Report per-task baselines, confidence/coverage where applicable, semantic preservation, and paired differences. Bootstrap at independent family level and separate fitted-seed uncertainty from conditional sampling uncertainty. Run the chosen natural-text extension and the central contrast on a second trained substrate if feasible; otherwise scope the main empirical claim to SONAR.
Done when: the table is generated from retained per-item outputs, metric sanity checks pass, uncertainty and exclusions are explicit, and the result survives a fresh final cohort. No requirement that a proposed method wins; no promotion of exploratory dose selection into confirmatory evidence.
N06 — Two case studies [needs to be improved]
Supports: §5 / Figures 3–4. Reuse: S01/S04/S05 for algebra; S02 for reading versus editing.
N06a — Algebra: [existing result] Figure 3 now reads original per-pair-seed numeric aggregates. Keep the small organism's installation experiment distinct from SONAR fits; full per-item/interval reproduction remains [needs to be improved]. [needs to be run] Evaluate reused composition maps and selected editing operators on N02 data, with fixed calibration doses, target-latent controls, and constituent/content preservation. Report sentence position separately from thematic role. Do not infer the encoder's implementation from a fitted surrogate's decoder utility.
N06b — Readout/control: [existing result] Figure 4 now verifies the E04b swap/retention means and paired point contrast against retained predictions. The primary baseline chooses the best control per proposition; the manuscript explains its difference from a single best overall arm. Saved intervals were imported, not recomputed. [needs to be improved] Bootstrap reproduction and broader content-preservation checks. [needs to be run] Connect the case to N05's common prediction/control tasks, with structured energy-matched controls and target/collateral scoring.
Done when: each case explains a specific pattern in the common comparison, with provenance-linked figures and bounded claims. Case choice follows evidential clarity, not whether a direction appears positive.
N07 — Residual sentence-information pilot [needs to be run]
Optional; supports: §5.3 or discussion/appendix. Reuse: S03/S07 lexical anatomy, S06 dictionary findings. Depends on: working N02 scoring; need not block N05.
Work: Fit an additive lexical/slot account on training data; compute held-out residuals. Measure remaining word identity, relation/polarity information, and transfer. Compare with the fitted component and native vector. Use learning curves and sensitive planted interactions. If coherent structure remains, test a predicted intervention on a matched carrier against shuffled and norm-matched controls.
Interpretation: subtracting identical bag predictions preserves same-word sentence differences by construction. Restoring an exact residual is tautological. Decoding the remainder alone is exploratory. A useful result requires generalizable predictions or specific consequences beyond those identities. Do not call an unexplained component pure meaning.
Done when: one bounded result identifies either a reproducible target or why the proposed test cannot identify one. A new residual SAE or full manifold theory is outside the core finish line.
N08 — Manuscript and release [needs to be improved]
Supports: whole paper. Depends on: N05/N06 for final empirical claims.
Work now: maintain the editable draft, provenance, accessible status colors, and links to the original campaign. The schematic and two existing-result figures are rendered with numeric provenance. Remaining: complete related work, render the new comparison figure after N05, write the outcome-specific abstract/conclusion, pin benchmark dependencies and asset licenses, create example adapters, smoke-test a clean environment, and typeset to nine pages. The public draft page is not the completed benchmark release.
Done when: a new reader can reproduce one valid score and one failed control; all main claims have traceable evidence and limitations; the venue PDF meets the page budget. Sources/data protocols and extended results stay in the appendix.
Execution order and scope boundaries
N01 → N02/N03 → N04 → freeze → N05/N06 → N08. N07 is optional. Reconcile only evidence selected for this paper rather than reopening the whole archive. Capacity laws, large multilingual sweeps, training dynamics, monitoring deployment claims, and a new SAE architecture are not core dependencies. No old experiment is silently recast as a result under the new protocol.