Appendix G — The L2 "organism" text autoencoders: training and generalisation

G.1 Lineage and naming [existing result]

The seed replicates reuse train_v1.py, corpus_v1.py, model.py byte-for-byte (diff against 4l:/workspace/HOME/guest/night8/organism_v1/ checked 12 September 2026: identical). Checkpoints live only on the GPU box: 4l:/workspace/HOME/guest/night8/organism_v1/run_dz{256,64}/{ckpt_best,ckpt_final}.pt and 4l:/workspace/HOME/guest/night8/organism_v1_seeds/run_seed{1,2,3}/ckpt_final.pt (30 MB each, no optimiser state).

G.2 Architecture and objective [existing result]

Source: /data/research/benchmarks/night8/organism_v1/model.py, train_v1.py.

Objective. Paraphrase-canonicalisation: encode a surface sentence, decode the fixed-order canonical rendering of the same meaning tuple — AGENT [not] VERB[tense] [COUNT as digits] PATIENT ADJS [in the LOC] . — with canonical synonyms and explicit not. Loss is token cross-entropy on the canonical stream (pad ignored, no label smoothing), teacher-forced. Surface ≠ canonical for 0.99368 of training pairs (V1_REPORT §1; recomputed here 0.993963, G.5). The canonical form has an exact inverse parser (corpus_v1.parse), which is what every "round-trip" number below uses.

Optimiser and schedule (train_v1.py defaults, unchanged in every run):

Hardware: one RTX A4000 per run on the 4l box (NVML broken; device liveness verified by a trial matmul; memory read from torch.cuda.max_memory_allocated).

G.3 Data-generating process [existing result]

Source: /data/research/benchmarks/night8/organism_v1/corpus_v1.py (constants and sample_factors), ORGANISM_V1_SPEC.md, V1_REPORT §1.

Meaning factors (all logged; all required by the canonical target):

String-inert factors (logged, never rendered; theorem-grade absent): mood (3), narrator (20).

Surface latents (change the surface, not the target): paraphrase family (9: active .18, passive .15, cleft .10, fronted-locative .10 [needs a location], nominalisation .10, there-construction .10, question .04, topicalisation .13, pseudo-cleft .10), verb synonym (2–3 per concept), object synonym (2 per concept), number style (digit / word), filler adverb (8, p=.3), locative preposition (3).

Corpus: 2,000,000 (surface, canonical, factors) triples, sample_pairs(2_000_000, seed=0, split="train"), generated in-process (18.2 s on 4l, 19.5 s here). Unique surfaces 1,964,397 (metrics.json:train_unique_surface; recomputed identically in G.5). Surface length mean 9.80 / max 18; canonical mean 6.79 / max 11.

Held-out split rule. An ordered two-person combination (verb, agent, patient) is held out iff the first byte of md5("verb|agent|patient") is divisible by 20 (≈ 5 % of ordered combos). split="train" never emits a held-out combination; split="heldout" emits only those. Leakage in both directions was asserted 0 on 20,000 / 2,000 samples at build time (V1_REPORT §1). Because the rule hashes the ordered triple, the reverse role assignment of a held-out combination may occur in training.

G.4 Compute accounting per checkpoint [existing result]

Sources: run_dz256/metrics.json, run_dz64/metrics.json under /data/research/benchmarks/night8/organism_v1/; figures/organisms/seed{1,2,3}_metrics.json (verbatim copies of 4l:/workspace/HOME/guest/night8/organism_v1_seeds/run_seed{1,2,3}/metrics.json, SHA-256 in figures/organisms/SHA256SUMS); /data/research/benchmarks/night8/organism_v2/run_{sem_dz64,sem_dz256,recon_dz64}/metrics.json. Derived columns: tokens/param = tokens_seen / params; epochs = steps × 256 / 2,000,000.

"Tokens seen" is train_v1.py's counter (line 177): non-pad source tokens plus non-pad decoder-input tokens per pair. The decoder input is <bos> + canonical + <eos> with its last column dropped, so <eos> is counted for every item except the batch-longest ones. Per pair this is 18.57 tokens (544,050,648 / (114,440 × 256)); content tokens without <bos>/<eos> are 16.59 per pair (G.5).

Time to ceiling (in-distribution exact match, evaluations every 2,000 steps; from eval_log): first ≥ 0.99 at step 10,000 / 8,000 / 10,000 / 10,000 / 10,000 and first 1.000 at step 26,000 / 26,000 / 28,000 / 26,000 / 30,000 for dz256, dz64, seed1, seed2, seed3. Small dips (≥ 0.9993) occur between the first 1.000 and step 40,000; after step 40,000 every evaluation on all three splits is 1.000 for all five runs (checked from the same files). The seed 0 runs therefore spent roughly 65 of their 90 minutes on a flat plateau; the 35-min replicates stop after about 10 min of plateau (SEEDS_REPORT §Setup). v2's rows are included because they are the only runs of this family with a rule-based stop; their tokens/param (48–50) is where the plateau rule cuts the same schedule.

G.5 Non-repeated tokens per parameter [existing result]

Computed 12 September 2026 on this box by /data/research/tae-interp/neuralese-paper/corpus_token_stats.py (CPU only; no torch; the Vocab class is executed from the archived model.py), output figures/organisms/corpus_token_stats.json, log figures/organisms/corpus_token_stats.log. The corpus is regenerated with the archived corpus_v1.sample_pairs(2_000_000, seed, split="train"), the exact call in train_v1.py. Check: the recomputed unique-surface count for seed 0 equals the logged 1,964,397; seeds 1–3 reproduce their logged 1,964,165 / 1,964,356 / 1,963,945. Generation took about 20 s per seed; no sub-sampling was needed.

(a) Tokens in the de-duplicated corpus, seed 0. Every surface string maps to exactly one canonical string, so unique (surface, canonical) pairs = unique surfaces = 1,964,397 (unique canonicals: 1,702,567 — different paraphrases of the same meaning tuple share a target).

Consistency check: one pass under the train_v1.py accounting (37,175,564) × 14.648 passes = 544.6 M, against the logged 544,050,648 tokens seen (difference 0.1 %, the dropped <eos> of the batch-longest items).

Tokens seen vs. non-repeated tokens.

The 70 tokens/parameter headline is therefore about 4.7 distinct tokens per parameter seen roughly 15 times each; the replicates saw the same 4.7 distinct tokens about 6 times each and reach the same 1.000 finals (G.6).

(c) Distinct token n-grams, seed 0 corpus (a diversity measure; counted within sequences after the 20/13 caps):

For scale: the source side has 0.50 distinct 4-grams per parameter (d_z=256), and the whole canonical side uses 432 of the 953 content types (names, canonical lemmas in two tenses, digits, canonical nouns, adjectives, locations, not, in, the, .).

Seeds 1–3 (same generator, seed=1,2,3; the seed also sets weight init and batch order, SEEDS_REPORT §Setup):

G.6 Generalisation: what was measured and what it means [existing result]

Definition used on this page. For these models "generalisation" means correct canonicalisation of sentences whose factor combination was not in the 2,000,000 training pairs, inside a closed synthetic vocabulary of 953 types and nine paraphrase families. It is not transfer to open-domain text, to unseen words, or to unseen constructions. Every split below is drawn from the same generator; only the combination rule differs.

G.6.1 organism_v1: the three training-time splits

Sources: run_dz{256,64}/metrics.json:final, summary_tables.md, V1_REPORT §2; seeds from figures/organisms/seed{1,2,3}_metrics.json:final.

Per-factor round-trip (parse of the greedy canonical output vs. logged factors) is 1.000 for all eight meaning factors — verb, agent, patient, count, adjs, location, tense, negation — on all three splits, for all five checkpoints (final.*.per_factor in each metrics.json; summary_tables.md "Per-meaning-factor round-trip"). Token accuracy is 1.000 on every final split.

The held-out-combo split is the compositional certificate of the v1 report: those exact role assignments were never seen, yet agent and patient are placed correctly in the canonical order. Its limits: the reverse assignment may have been seen; every name, verb and construction was seen many times; and the split is 5 % of two-person triples, not a held-out name or verb.

A weaker out-of-DGP datum exists as a by-product of the M2b battery (V1_REPORT §3, M2b note): verb toggles applied to unconstrained backgrounds create verb-class × patient-type combinations that the generator never produces (e.g. a social verb with a counted object patient); base round-trip on those sentences is 0.961 (dz256) and 0.977 (dz64), versus 1.000 on in-DGP backgrounds. This is the only measurement of behaviour outside the generator's support, and it was not designed as one.

G.6.2 Seed replication (d_z=64, seeds 1–3, 35 min)

Source: /data/research/benchmarks/night8/organism_v1_seeds/SEEDS_REPORT.md and the seed metrics.json files. All three replicates reach 1.000 on every split, factor and binding measure with 5.8 epochs (G.4). What varies across seeds is not the behavioural ceiling but the certification tier of weak surface latents (loc_prep, num_style, patient_syn, verb_syn flip between P_NONLINEAR_ONLY / P_LINEAR / NO_GT; SEEDS_REPORT §1); the eight meaning factors and binding are P_LINEAR in all seeds.

G.6.3 B2: agent-disjoint TEST items (the paper's substrate check)

Source: /data/research/tae-interp/neuralese-paper/experiments/B2-organism-shortfall/results/TABLE.md (coverage lines and READ tables per tag; "Seed spread" section), results/read.json, NOTES.md (stimulus design). Stimuli: the 100 names are shuffled with seed 20260912; TRAIN uses the first 60, TEST the last 40; two-person items draw both roles from the split's names; 300 TEST items per factor; backgrounds sampled with split="any", so 46 of the 300 ROLE TEST items fall on training-held-out (verb, agent, patient) combos. "Certified" = the decoder parses both the base and the counterfactual item to their logged factors (parse(D(z_src)) and parse(D(z_tgt)) exact). "Lens" = accuracy of reading the factor off the decoder's own output; it is the READ ceiling.

Reading: on names the models never saw in the TRAIN half of the stimulus set (but did see in pre-training) the decoder certifies 297–300 of 300 items per factor, and reading each factor off the decoder output is at ceiling (0.995–1.000). The untrained same-architecture copies certify 0 items and have a lens of 0.000 (their decodes never parse), which is the intended null: coverage is a property of training, not of the parser. Note the asymmetry inside the untrained rows: linear probes on the untrained z still read POLARITY at 0.998 / 0.990 and COUNT at 0.768 / 0.403 (TABLE.md, untrained READ), so "readable from z" and "certified through the decoder" are different endpoints, as the manuscript's editorial commitment 5 requires.

Seed spread of the READ cells (fraction of lens; four d_z=64 models: seed 0 at 90 min and seeds 1–3 at 35 min; TABLE.md "Seed spread"):

The 98-way COUNT read is the one cell with visible seed variance (0.91–0.97 of ceiling) and the one where a 35-min replicate is not systematically better or worse than the 90-min reference. ROLE-XC (train on active renderings, test on passives) stays at 1.000 for role_id across all four models; the binary role_bin label reads at chance (0.47–0.53) in every model — the TRACE caveat that a designated-A binary label confounds binding with a name-comparison nonlinearity (NOTES.md, READ cells).

G.6.4 organism_v2 as the contrast: where generalisation fails

Source: /data/research/benchmarks/night8/organism_v2/summary_tables.md "Training" and "TOTAL accuracy by magnitude"; V2_REPORT §2–3. Same architecture, corpus size and schedule; the canonical target is computed (TOTAL = count1 + count2 as digit tokens, agents sorted alphabetically, together iff the two agents are equal).

The same substrate family therefore shows both faces: v1 generalises across every combination it was tested on; v2 generalises alphabetical ordering to never-seen pairs (0.99) and the carry rule of addition (ones digit 0.91 at d_z=256) but memorises the magnitude support (tens digit clamps at the training maximum, 0.993 of held-out totals decode as 7x) and does not generalise the equality test to held-out self-pairs (0.000). The recon twin, trained on the same data to copy the surface, has no such failures (0.995 / 1.000) because its target requires no computation. Generalisation here is a property of the target function, not of the encoder or the corpus.

G.7 Figure A1 — training curves [existing result]

Rendered by make_organism_figure.py from the five metrics.json files (loss logged every 100 steps to four decimals, shown as a 2,000-step trailing mean on a log axis; exact match at the 2,000-step evaluations, first 40,000 steps shown; after step 40,000 every evaluation on all three splits is 1.000 for all five runs). Plotted values are frozen in figures/organisms/figA1_data.json. Loss values below 1e-4 are floor-rounded in the logs. No per-step validation loss was logged, and no curves exist for the M1/M2 battery quantities.