# TAE-Bench paper project

Active writing project, started 11 September 2026. Working title: **TAE-Bench: Text Autoencoders as a Testbed for Understanding Neuralese**.

The contribution is a benchmark plus an empirical investigation of what interpretation methods enable. Keep the broad neuralese framing. Word residuals, linearity, and better dictionaries are research questions within this framework, not prerequisites for the paper.

- `PAPER.md` → `index.html`: manuscript draft, with the benchmark introduced early.
- `EXPERIMENTS.md` → `experiments.html`: bounded execution plan and completion criteria.
- `EVIDENCE.md` → `evidence.html`: scoped existing results and provenance.
- `sources/`: dated, verbatim source-report snapshots; these are historical evidence, not new runs.
- `evidence_sources.json`: original paths, SHA-256 hashes, and snapshot metadata.
- `figures/`: the schematic and two existing-result figures in SVG, PDF and PNG, plus their frozen numeric data/provenance.
- `extract_figure_data.py`: extracts original aggregates and checks the order-edit means/paired point difference against retained rows.
- `make_figures.py`, `figure_requirements.txt`: reproduce all three figures from `figures/data.json` without models or the rest of the archive.
- `build.py`, `style.css`, `ui.js`, `requirements.txt`: reproducible static website.

## Status vocabulary

- **[existing result]** (green): a result is reported in the linked archive, within the stated scope. Does not mean reproduced in this project or submission-ready.
- **[needs to be improved]** (amber): existing prose, experiment, measurement, or evidence needs the named improvement.
- **[needs to be run]** (purple): the stated evaluation is outstanding; no numerical result is implied.

Colors always accompany text. Section color indicates its principal remaining dependency; mixed sections retain inline labels. There is no misleading percentage-complete score.

## Length and scope

Target: 9 manuscript pages including figures, excluding references and appendix: introduction 1; benchmark 2; validation/setup 1; comparison 2; case studies 2; discussion/related work/limitations 1. This is a page allocation, not a claim that the HTML is a typeset nine-page submission. Abstract shares the introductory allocation. Choose a venue template and typeset after the central table is populated.

The full experiment plan and evidence register are companion/appendix material. Do not move them into the main nine pages. Keep the original Campaign100 manuscript and board available at their existing URLs. This project does not modify their scientific content.

## Build

From this directory, use an environment with `pip install -r requirements.txt`, then `python build.py`. The existing environment also works:

```sh
/data/research/tae-status-2026-09-09/.site-venv/bin/python build.py
```

The Markdown files are the source of truth. `build.py` regenerates the three HTML pages and does not execute experiments or publish. The top-level overview is generated by `../build_site.py`; its source now links this project first. Published project files are explicitly listed in `../PUBLISH_MANIFEST.txt`.

To rebuild the figures, install `figure_requirements.txt` in a separate environment, then run `python make_figures.py`. Run `python extract_figure_data.py` only when intentionally refreshing the frozen source measurements from the full archive. The extraction verifies point aggregates; it does not rerun models or recompute confidence intervals. Figure 2 remains a clearly labeled unrun comparison; no placeholder scores are plotted.

## Editorial commitments

1. Distinguish a known linguistic target from a known internal mechanism.
2. Evaluate interpretation methods on a fixed substrate; encoder comparisons are separate.
3. Do not pool old results into a common leaderboard: datasets, budgets, metrics, and substrates differ.
4. Match information access and identify supervised or privileged methods.
5. Report target success, collateral changes, and coverage separately. Failed probes do not prove absent information.
6. Treat fitted operators as predictive/behavioral models unless encoder-level mechanism evidence is supplied.
7. Do not claim residuals are pure meaning or that an SAE's components are uniquely correct atoms.
8. Use one main comparison and two case studies. New experiments require their own frozen protocol; creating this draft launches none.