# Experiments and writing plan

Companion to the [manuscript](index.html). This plan is not a preregistration and starts no compute jobs. Main goal: a validated small benchmark, a common method comparison, and two explanatory case studies within nine pages.

## Prioritized experiment queue

**Start with N01, then N02 and N03.** The main missing result is a fair comparison on shared data. Existing figures already support the two case studies; a new decomposition method or a complete account of residual meaning is not a prerequisite.

| Order | Work | Question / required output | Reuse and dependency | Priority |
|---|---|---|---|---|
| 1 | **N01: benchmark contract** | What can each method see, predict, and change? Produce the task/interface/baseline matrix and frozen scoring definitions. | Existing task generators and access audits; design work before final runs. | Core |
| 2 | **N02: data and decoder checks** | Can the substrate express each tested distinction? Produce shared train/calibration/test banks, native/target decoding results, and a semantic-scoring audit. | Retained SONAR pipeline and stimuli; after N01. | Core |
| 2 | **N03: toy calibration** | Does the evaluation recognize a correct interpretation? Produce results for rotated components, a known interaction, and bad/null interpretations. | GAUGE; after N01; can develop alongside N02. | Core |
| 3 | **N04: common method pilot** | Do the tasks have usable measurements and controls? Produce a pilot table for simple probes/operators, PCA/random controls, an existing SAE, and a structured composer on supported tasks. | Retained assets; after N02/N03. | Core |
| 4 | **N05: final comparison** | What capabilities do current methods provide on new cases? Produce the frozen six-task table, generalization results, per-family uncertainty, and Figure 2. | Pilot fixes first; final data remains separate. | Core |
| 4 | **N06a/b: case-study extensions** | Which patterns explain the table? Test algebra under shared semantic scoring and connect the readout/control gap to that protocol. | Existing Figures 3–4 and original outputs; after N02/N04, coordinated with N05. | Core |
| 5 | **Second trained substrate** | Does the central comparison generalize beyond SONAR? Reproduce the selected main contrast with a decoder-capability check. | N05 protocol; choose a compatible retained system first. | Desirable; otherwise scope claims to SONAR |
| Separate | **N07: residual pilot** | What useful information survives the fitted lexical account? Produce one controlled readout/transfer result with lexical and decoder controls. | N02 scorer and E01/E03 assets; does not block N05. | Optional |

All new-run entries above remain **[needs to be run]**. N01 is **[needs to be improved]** because the draft contract exists but is not frozen. N08 below covers writing, reproducibility, and the final nine-page manuscript rather than another experiment.

## First work session: concrete deliverables

1. Write `protocol/contract_v1.md`: six tasks, exact allowed inputs/outputs, label access, query budgets, primary metrics, and which methods enter each task. Explicitly identify supervised, decoder/text-reference, and privileged-oracle methods.
2. Write `protocol/assets_v1.json`: chosen checkpoint paths/revisions, vector normalization, SAE substrate/inference rule, and decoding settings. Verify retained assets before scheduling new training.
3. Write `protocol/data_plan_v1.md`: content/relation/polarity contrasts, two-sentence composition, independent family splits, natural-text extension, and the semantic annotation rubric. Specify a pilot cohort; use pilot variance to size the final study.
4. Write `protocol/controls_v1.md`: exact toy codes, privileged positives, shuffled/no-op/random controls, and sensitivity checks required for negative conclusions. Known targets do not imply known mechanisms on real TAEs.
5. Begin N02/N03 development against that contract. Save their outputs and failures; keep the final evaluation unopened. These paths are proposed deliverables, not files already created.

The first session is successful when the experiment table has an executable interpretation and the control/data pilot can start. It does not need a new scientific discovery.

## What to save from every run

Use separate directories such as `runs/N03/pilot_v1/` and `runs/N05/final_v1/`, retaining the original archive. Before execution, record the hypothesis, input/split hashes, checkpoint/configuration, frozen metrics, stopping conditions, and expected resource budget. After execution, retain per-item predictions or edited vectors/decodes, aggregate results, uncertainty method, source/code hashes, actual resource use, and a short scoped result report. Sensitive raw inputs should not automatically enter the public publishing manifest.

Early work is mainly protocol/data preparation and cached CPU analysis. Native decoding, SAE inference checks, and final vector interventions require short measured GPU pilots before estimating the full budget. No GPU-day estimate or external judging budget is committed by this plan.

## Decisions that control scope

- If a privileged control fails, repair the instrument before interpreting real-model negatives.
- If native decoding cannot express a contrast, report capability/coverage explicitly; do not attribute the failure to interpretation alone.
- If a pilot metric saturates or measures a shortcut, revise it on development data and freeze a fresh final protocol. Do not remove final tasks because their results are inconvenient.
- If methods tie under valid controls, report the tie and uncertainty. A positive advantage is not a required outcome.
- If residual structure is inconclusive, keep it as an open question. Avoid expanding into a new SAE training campaign to rescue the narrative.
- Finish with one common comparison and two case studies. Full capacity, multilingual, and training-dynamics sweeps remain outside this paper's core plan.

## Page and figure allocation

| Manuscript section | Pages, including figures | Main artifact |
|---|---:|---|
| 1. Introduction + abstract | 1 | Figure 1: testbed/contract |
| 2. TAE-Bench | 2 | Six-task contract table |
| 3. Validation/setup | 1 | Control summary |
| 4. Method comparison | 2 | Figure 2: capability profile |
| 5. Two case studies | 2 | Figures 3–4: algebra; readout/control gap |
| 6. Discussion/related work/limitations | 1 | Scoped interpretation and open questions |
| Total | 9 | References and appendix additional |

The HTML is a working reading version, not a verified nine-page PDF. Freeze a venue template and measure pages at N08. Residual exploration belongs in a short open-question paragraph unless it earns a case study through clear evidence. Do not expand to a third case study.

## N01 — Freeze the benchmark contract [needs to be improved]

**Supports:** §§2–3. **Reuse:** SIEVE task ideas, GAUGE authored controls, TRACE access lessons, Campaign100 stimulus/checkpoint records.

**Work:** Choose exact checkpoints and six task definitions. Specify training/test inputs, outputs, semantic supervision, encoder/decoder budgets, task-specific baselines, unsupported method cells, primary estimands, minimum useful effects, and complexity accounting. Keep fixed-model method evaluation separate from encoder comparison. Resolve the current source-text versus latent-only access question per track. Supervised/privileged methods are labeled.

**Done when:** one versioned contract and method-interface table determine every planned table cell without consulting final labels. No automatic score exclusions based on poor method performance. No claim of information beyond complete text without a computational/access qualification.

## N02 — Shared corpus and decoder capabilities [needs to be run]

**Supports:** §§2–4. **Depends on:** N01.

**Work:** Build a controlled proposition bank with content, a relational contrast, and polarity, plus a two-sentence composition bank and natural-text transfer set. Separate new combinations, construction/rewording shifts, and unseen vocabulary. Keep independent proposition families intact across splits. Fix calibration and final cohorts. Check native source and counterfactual decoding; retain failures. Use an independent semantic audit that samples both parsed and unparsed outputs.

**Metrics:** target success, content preservation, fluency, parser coverage, and applicability as separate quantities. Semantic scoring is primary; own-encoder similarity is supporting evidence. Report full-cohort and capability-qualified outcomes. Determine confirmatory sample size from pilot variation and the chosen useful effect rather than an arbitrary round number.

**Done when:** data/split hashes, decoder settings, audit rubric, annotation record, and scoring code are frozen; coverage failures are quantified and task ceilings are measurable.

## N03 — Known-code calibration [needs to be run]

**Supports:** §3. **Depends on:** N01. Can develop alongside N02.

**Work:** Instantiate a rotated slot-component code and a code with a known interaction. Include interference/noise. Privileged controls can access the generating mechanism; ordinary competitors cannot. Exercise the exact final scorer and method adapters. Test shuffled explanations/labels, irrelevant components, random directions, and no-op edits. Accept behaviorally equivalent factorizations.

**Done when:** correct controls succeed where information is recoverable; known-bad methods fail for the intended reasons; relevant sensitivity limits are measured. A failed control invalidates the inference, not the biological/linguistic hypothesis. Existing GAUGE scores do not substitute for this run.

## N04 — Common pilot on retained assets [needs to be run]

**Supports:** §§3–4. **Depends on:** N01–N03.

**Work:** Verify checkpoint loading and inference, then run probes/operators, PCA/random dictionaries, retained SAEs, and a structured composer on their supported tasks. Keep whole-z and c_pool dictionary substrates separate. Check inference batch dependence, realized sparsity, and training/inference budgets. Do not train a new SAE family merely because an old score looks low.

**Done when:** the pipeline is executable, task controls pass, expected ceilings have headroom where appropriate, uncertainties can support sample-size decisions, and unsupported cells are declared. If every cell saturates or every privileged control fails, revise the benchmark using pilot data only. Archive revisions before freezing a new final set. Honest method ties remain results.

## N05 — Frozen comparison and generalization [needs to be run]

**Supports:** §4 / Figure 2. **Depends on:** N04 and a fresh final split.

**Work:** Execute the common comparison under fixed budgets. Save predictions before revealing counterfactual outputs. Report per-task baselines, confidence/coverage where applicable, semantic preservation, and paired differences. Bootstrap at independent family level and separate fitted-seed uncertainty from conditional sampling uncertainty. Run the chosen natural-text extension and the central contrast on a second trained substrate if feasible; otherwise scope the main empirical claim to SONAR.

**Done when:** the table is generated from retained per-item outputs, metric sanity checks pass, uncertainty and exclusions are explicit, and the result survives a fresh final cohort. No requirement that a proposed method wins; no promotion of exploratory dose selection into confirmatory evidence.

## N06 — Two case studies [needs to be improved]

**Supports:** §5 / Figures 3–4. **Reuse:** S01/S04/S05 for algebra; S02 for reading versus editing.

**N06a — Algebra:** **[existing result]** Figure 3 now reads original per-pair-seed numeric aggregates. Keep the small organism's installation experiment distinct from SONAR fits; full per-item/interval reproduction remains **[needs to be improved]**. **[needs to be run]** Evaluate reused composition maps and selected editing operators on N02 data, with fixed calibration doses, target-latent controls, and constituent/content preservation. Report sentence position separately from thematic role. Do not infer the encoder's implementation from a fitted surrogate's decoder utility.

**N06b — Readout/control:** **[existing result]** Figure 4 now verifies the E04b swap/retention means and paired point contrast against retained predictions. The primary baseline chooses the best control per proposition; the manuscript explains its difference from a single best overall arm. Saved intervals were imported, not recomputed. **[needs to be improved]** Bootstrap reproduction and broader content-preservation checks. **[needs to be run]** Connect the case to N05's common prediction/control tasks, with structured energy-matched controls and target/collateral scoring.

**Done when:** each case explains a specific pattern in the common comparison, with provenance-linked figures and bounded claims. Case choice follows evidential clarity, not whether a direction appears positive.

## N07 — Residual sentence-information pilot [needs to be run]

**Optional; supports:** §5.3 or discussion/appendix. **Reuse:** S03/S07 lexical anatomy, S06 dictionary findings. **Depends on:** working N02 scoring; need not block N05.

**Work:** Fit an additive lexical/slot account on training data; compute held-out residuals. Measure remaining word identity, relation/polarity information, and transfer. Compare with the fitted component and native vector. Use learning curves and sensitive planted interactions. If coherent structure remains, test a predicted intervention on a matched carrier against shuffled and norm-matched controls.

**Interpretation:** subtracting identical bag predictions preserves same-word sentence differences by construction. Restoring an exact residual is tautological. Decoding the remainder alone is exploratory. A useful result requires generalizable predictions or specific consequences beyond those identities. Do not call an unexplained component pure meaning.

**Done when:** one bounded result identifies either a reproducible target or why the proposed test cannot identify one. A new residual SAE or full manifold theory is outside the core finish line.

## N08 — Manuscript and release [needs to be improved]

**Supports:** whole paper. **Depends on:** N05/N06 for final empirical claims.

**Work now:** maintain the editable draft, provenance, accessible status colors, and links to the original campaign. The schematic and two existing-result figures are rendered with numeric provenance. **Remaining:** complete related work, render the new comparison figure after N05, write the outcome-specific abstract/conclusion, pin benchmark dependencies and asset licenses, create example adapters, smoke-test a clean environment, and typeset to nine pages. The public draft page is not the completed benchmark release.

**Done when:** a new reader can reproduce one valid score and one failed control; all main claims have traceable evidence and limitations; the venue PDF meets the page budget. Sources/data protocols and extended results stay in the appendix.

## Execution order and scope boundaries

N01 → N02/N03 → N04 → freeze → N05/N06 → N08. N07 is optional. Reconcile only evidence selected for this paper rather than reopening the whole archive. Capacity laws, large multilingual sweeps, training dynamics, monitoring deployment claims, and a new SAE architecture are not core dependencies. No old experiment is silently recast as a result under the new protocol.