# RESULT — 074 grokking-check: does 5×-budget over-training induce delayed emergence?

**Tier: T3-exploratory. DONE + HARVESTED** (train tmux `c100_074` on box, GPU0/CVD=0,
launched v2 10:29:59Z, **DONE 16:05:58Z** = final step 177557 = 100% of the 2.5e9-tgt-tok
budget = **15,151 epoch-equivalents** over the 6,000-sentence memorized set, ~5.6 GPU-h;
battery harvest tmux `c100_074b`, CPU CVD="", 15 ckpt batteries in ~14 min, DONE 16:37:07Z).
Reuses 073's identical `stimuli_agent_patient.json` (2000 items) so numbers are directly
comparable to the standard-budget organism. **Mean Brier 0.1195** (P1 T, P2 T, P3 T, P4 T,
P5 T — all five frozen predictions correct).

## Verdict (headline — the 5×-budget answer)

**NO grokking. Extended over-training (5× the standard budget, 15,151 epochs deep in the
flat-train-loss memorization regime) induces ZERO delayed emergence of role abstraction and
NO late generalization or geometry improvement. Every watched metric is flat across the whole
over-training plateau.** The 6k-sentence set is memorized to train_f1 0.998 / train_exact
0.979 by ~4% of budget; from there val reconstruction, role-binding AUC, and the surface-order
code are all frozen for the remaining ~96% of training. The frozen "grokking observed"
criterion is met at **NO checkpoint** for **any** of {role cross AUC, val_exact, surface AUC}.

Extra epochs on a memorized set buy nothing but a deeper overfit (val_dae_ce climbs 6.2→18.7
and stays there). Over-training time is **not** a substitute for the lexical diversity that
006/073 tie role-code formation to.

## Reconstruction curve vs step (P1 memorization positive-control — from progress/metrics.jsonl)

| step | %bud | epochs | val_f1 | val_exact | val_ce | **train_f1** | **train_exact** | train_ce |
|---|---|---|---|---|---|---|---|---|
| 2000 | 1.1% | 171 | **0.255** | 0.00 | 6.25 | 0.532 | 0.113 | 2.424 |
| 4000 | 2.2% | 341 | 0.162 | 0.00 | 9.07 | 0.900 | **0.633** | 0.161 |
| 6000 | 3.4% | 512 | 0.139 | 0.00 | 11.12 | 0.982 | 0.887 | 0.011 |
| 8000 | 4.5% | 683 | 0.132 | 0.00 | 12.64 | 0.997 | 0.961 | 0.003 |
| 16000 | 9.0% | 1365 | 0.124 | 0.00 | 15.43 | 0.998 | 0.975 | 0.001 |
| 40000 | 22.5% | 3413 | 0.116 | 0.00 | 17.58 | 0.998 | 0.977 | 0.000 |
| 72000 | 40.6% | 6144 | 0.108 | 0.00 | 18.46 | 0.998 | 0.979 | 0.000 |
| 120000 | 67.6% | 10240 | 0.112 | 0.00 | 18.69 | 0.998 | 0.979 | 0.000 |
| 160000 | 90.1% | 13653 | 0.112 | 0.00 | 18.68 | 0.998 | 0.979 | 0.000 |
| 177557 | 100% | 15152 | 0.112 | 0.00 | 18.66 | 0.998 | 0.979 | 0.000 |

**The signature is overfit, not grokking.** val_f1 PEAKS at **0.255 at step 2000** (before the
set is memorized), then **monotonically declines** to a flat ~0.11 and stays there for the
entire remaining 99% of training; **val_exact = 0.000 at every eval**; val_ce rises 6.2→18.7.
This is the exact opposite of the grokking shape (flat-low-then-late-jump). **P1 PASS**:
train_exact 0.633 by step 4000 (2.2% budget ≪ 10%), final train_dae_ce 0.0001 ≪ 0.60×val_dae_ce
(=11.2) — the overfit/grokking regime was fully entered, so grokking WAS testable, and did not occur.

## ROLE / SURFACE trajectory vs step (battery sweep, agent_patient, primary = linear pooled)

| step | %bud | val_f1 | z_bag | **roleXall(lin)** [95% CI] | roleXflip | roleXlex | **surfX** | winLin | winMlp |
|---|---|---|---|---|---|---|---|---|---|
| 0 | — | — | 0.500 | 0.495 [.493,.498] | 0.478 | 0.501 | 0.521 | 0.513 | 0.599 |
| 250 | — | — | 0.500 | 0.496 [.492,.501] | 0.499 | 0.507 | 0.497 | 0.506 | 0.630 |
| 1000 | — | — | 0.500 | 0.498 [.496,.500] | 0.487 | 0.504 | 0.513 | 0.502 | 0.600 |
| 2000 | 1.1% | 0.255 | 0.500 | 0.489 [.484,.495] | 0.470 | 0.496 | 0.526 | 0.521 | 0.648 |
| 4000 | 2.2% | 0.162 | 0.500 | 0.491 [.485,.497] | 0.473 | 0.493 | 0.523 | 0.520 | 0.650 |
| 8000 | 4.5% | 0.132 | 0.500 | 0.489 [.483,.495] | 0.466 | 0.489 | 0.530 | 0.524 | 0.638 |
| 16000 | 9.0% | 0.124 | 0.500 | 0.489 [.483,.495] | 0.464 | 0.493 | 0.532 | 0.525 | 0.645 |
| 32000 | 18.0% | 0.120 | 0.500 | 0.490 [.483,.496] | 0.460 | 0.492 | 0.538 | 0.528 | 0.638 |
| 48000 | 27.0% | 0.116 | 0.500 | 0.489 [.484,.495] | 0.460 | 0.493 | 0.537 | 0.530 | 0.644 |
| 64000 | 36.0% | 0.110 | 0.500 | 0.489 [.483,.495] | 0.458 | 0.498 | 0.539 | 0.533 | 0.641 |
| 96000 | 54.1% | 0.108 | 0.500 | 0.489 [.483,.495] | 0.462 | 0.494 | 0.534 | 0.531 | 0.643 |
| 128000 | 72.1% | 0.109 | 0.500 | 0.488 [.483,.494] | 0.461 | 0.495 | 0.535 | 0.533 | 0.647 |
| 160000 | 90.1% | 0.112 | 0.500 | 0.489 [.483,.494] | 0.464 | 0.495 | 0.532 | 0.531 | 0.641 |
| **177557** | **100%** | **0.112** | **0.500** | **0.488 [.483,.494]** | 0.464 | 0.496 | **0.532** | 0.531 | 0.640 |

surfX = surface_cross_all_pooled (linear). roleXall = role cross_all_pooled (P2 primary cell).
roleXflip = role cross_all_flipped (073's anti-transfer metric). roleXlex = role
cross_lexical_pooled (lexical holdout). winLin/winMlp = role within-construction readout ceiling.

- **ROLE (roleXall, the P2 cell): dead flat at/below chance for all 15 checkpoints** — range
  **0.488–0.499**, every 95% CI straddling 0.49; roleXlex (lexical holdout) 0.489–0.509 ≈ chance
  throughout. Role abstraction **never leaves chance** — over-training manufactures no binding
  capability. No late jump; the untrained ckpt is already at chance and it stays there.
- **SURFACE (surfX): forms EARLY, weak, then flat.** Rises from ~0.50 (untrained) to ~0.53 by
  step ~4000 (in lockstep with the transient val_f1 peak), then **frozen at 0.53–0.54** for the
  rest of the run. Critically this is **far weaker than 073's standard-budget A_D_s0 (surfX
  0.706 at final)** — the memorized 6k set produces only a vestigial surface-order code, and
  5× more epochs on it do **not** strengthen it (Δ from 40%→100% budget = −0.001). roleXflip
  stays ~0.46 (073 reached 0.30) — **no anti-transfer signature emerges** either, consistent
  with the weak/vestigial surface code.

## Frozen "grokking observed" criterion — applied per metric

| metric | plateau [min,max] (flat ≤0.03?) | late-window mean | jump (need ≥0.10) | **grokks?** |
|---|---|---|---|---|
| role cross AUC (roleXall lin) | [0.489, 0.491] flat=0.002 ✓ | 0.489 | **−0.001** | **NO** |
| surface AUC (surfX) | [0.523, 0.539] flat=0.016 ✓ | 0.532 | **−0.001** | **NO** |
| val_exact | max 0.000 | 0.000 | 0.000 | **NO** |

All three metrics are flat-across-plateau AND have late-minus-plateau jump < 0.05 (monotone-flat
= NO grokking by the frozen rule). **Grokking is NOT observed at ANY checkpoint.**

## Operator-geometry proxy

`eval_grok.sh` reused 073's binding battery, which computes role/surface AUCs, the z_bag control,
and readout ceilings but **does not** compute an explicit operator-geometry (knee-angle / SVD)
proxy — so none was produced here. The available geometry-adjacent readouts (within-construction
readout ceiling winLin 0.51–0.53, winMlp 0.60–0.65; z_bag exactly 0.500) are **flat across the
whole run** — no evidence of a late geometry cleanup. (A dedicated operator-geometry sweep over
the kept milestones would be a cheap follow-up; see below.)

## Gates

| Gate | Outcome |
|---|---|
| G1 GPU foreign-free | **PASS** — at battery-claim GPU0=35, phys2/3=15 MiB (all ≪1000); battery ran CPU-only (CUDA_VISIBLE_DEVICES=""), GPU untouched throughout. Train phase held GPU0 only. |
| G2 no divergence (v2) | **PASS** — v2 reached 100% budget cleanly (final.pt + progress written), **nan_skips=0** the entire run, final grad_norm 0.0075. (The v1 false-positive watchdog was removed — see provenance.) |
| G3 memorization control (=P1) | **PASS** — train_exact 0.633 by 2.2% budget, 0.979 final; the overfit regime was entered, so a null is a real null, not a regime failure. |
| G4 stimuli valid | **PASS** — z_bag within-role AUC = **0.500** at every one of 15 checkpoints; norms finite. |
| G5 encoder identity | **PASS** — each ckpt loaded with its correct step (0…177557); untrained ck_0 AUCs all ≈ chance (roleXall 0.495, surfX 0.521, z_bag 0.500) — loader sanity confirmed. |

## Predictions → Brier (frozen in PREREG_LITE.md, verbatim through amendment v2)

| Pred | P | Outcome | Brier |
|---|---|---|---|
| P1 memorization control PASSES (train_exact ≥0.60 by 10% budget AND final train_ce < 0.6×val_ce) | 0.85 | **TRUE** (0.633 @ 2.2%; 0.0001 ≪ 11.2) | 0.0225 |
| P2 role abstraction NEVER leaves chance (cross linear pooled AUC ≤0.60 every ckpt) | 0.70 | **TRUE** (max roleXall = 0.499, all 15 ckpts) | 0.0900 |
| P3 NO grokking of generalization (val_exact monotone/plateau, no late jump) | 0.65 | **TRUE** (val_exact = 0 throughout) | 0.1225 |
| P4 over-training does NOT sharpen recon late (val_f1 late window not > 40%-budget value +0.05) | 0.60 | **TRUE** (val_f1 ~0.11 late vs ~0.108 @40%; Δ≈0) | 0.1600 |
| P5 IF anything moves late it's recon not role (role stays flat) | 0.55 | **TRUE** (nothing moves late; role dead flat at chance) | 0.2025 |

**Mean Brier = 0.1195.** All five frozen predictions correct — a well-calibrated null. The
"nothing emerges from over-training alone" prior (rooted in 006's diversity→role-code link) was
right on every axis: memorization control fired, role stayed at chance, no val jump, no late
recon sharpening, and the (absent) motion was on the recon side, not role.

## Provenance — v1 watchdog false-positive & v2 fix

v1 (tmux c100_074, 09:08–09:23Z) was **healthy through step 8000** (train_loss 0.215→0.017,
train_f1 0.998, NO NaN) but the **inherited rising-val-CE watchdog G2** (`val_ce > 1.5× best ×2`)
fired at step 8000 (4.5% budget) and wrote `out/DIVERGED` (rc=3) — that rising val CE **is** the
intended memorization/overfit signature this experiment is designed to enter (exactly what P1
gates on), not an instability. **Stability-only amendment v2** (PREREG §AMENDMENT v2): removed the
val-CE watchdog (true divergence caught only by non-finite guards), retained grad-clip max_norm 1.0,
NaN-skip-then-abort-after-20, peak lr 5e-4→2e-4 warmup+cosine→1e-5. Dataset / model / budget /
ckpt-schedule / eval-cadence / memorization gate / grokking criterion / predictions P1–P5 all
UNCHANGED. v1 ckpts superseded (new lr schedule regenerates the trajectory) and deleted; v1 log
retained as `out/run_v1_diverged.log`. v2 ran to 100% clean.

## What did NOT run / limitations

- **Single small rung / single seed.** Rung-A arm-D d_z=256 (~11.6M-param ladder TAE), seed 0
  only. No cross-seed or cross-rung replication of the over-training null.
- **Low lexical diversity by design.** The 6k-sentence memorized set has low lexical diversity
  (006 caveat). This experiment therefore tests **over-training TIME**, holding diversity low —
  **NOT** diversity. It cannot distinguish "role abstraction needs diversity" from "role
  abstraction needs both diversity and time"; it only rules out "time alone suffices". The
  vestigial surfX 0.53 (vs 073's 0.706 on diverse standard-budget data) is direct evidence that
  the low-diversity memorized regime is a **weaker** organism, not merely a slower one.
- **Only agent_patient** was run (per recipe, for direct 073 comparability); genitive/causal/
  temporal skipped. z_bag=0.500 and untrained-chance controls carry the validity load.
- **No operator-geometry proxy computed** — the reused battery does not emit one; only AUC/ceiling
  geometry-adjacent readouts (all flat) are available.
- Battery JSONs (15) + BATT_*.md + trajectory_074.json + progress.json + metrics.jsonl +
  run.log + run_v1_diverged.log are in repo `out/`; the .pt checkpoints stayed on box (milestones
  0/16000/final kept for re-eval, others deleted — see box-cleanup note in the harvest log).

## Placement in Block I (training dynamics)

Sits alongside **073** (order-code-archaeology). **073** answered *when* the surface-order /
anti-transfer code forms during **normal, standard-budget** training on diverse data: mid-training
(~11–74% budget), gradually, locked to reconstruction, reaching surfX 0.706. **074** answers the
complementary question: does **5× more training past budget on a memorized low-diversity set** add
anything? **No** — it produces only a vestigial surfX 0.53, no role code, no late jump. Together:
the order/role code is a **byproduct of diverse-data reconstruction learning** (073), and **cannot
be manufactured by over-training time in the absence of that learning** (074). Over-training is not
a lever for representational abstraction in the TAE.

## Follow-up worth funding? **N (for over-training); Y-modest (for the diversity axis)**

Over-training time is now a **closed** lever — a clean, calibrated null (mean Brier 0.1195); no
value in more epochs or bigger budgets on memorized sets. The scientifically live question this
isolates is **diversity**: 074 (low-diversity, long-time → no role code) vs 073 (high-diversity,
standard-time → role code) implicate lexical diversity, not compute, as the driver. A **cheap
diversity-ladder** (fixed standard budget, vary train-set size 6k→60k→600k sentences, same battery
sweep, ~1 GPU-h each) would directly test whether role/surface-code strength scales with diversity
— the actual mechanism 006/073/074 jointly point at. Also cheap: an **operator-geometry sweep**
over the three kept 074 milestones to confirm no geometry cleanup hides behind the flat AUCs.