How the toy autoencoders store a token sequence in one vector

Five mechanisms, one scene each, drawn in an honest computed 3-D projection of a 64- or 128-dimensional latent. The projection basis is chosen from the real code geometry and every scene prints the fraction of each vector's squared length that the three axes actually capture — the rest lives in dimensions you cannot see. Numbers come from the trained checkpoints of arch_sweep unless a caption says synthetic.

0.0 / 0.0 s

What you are looking at

A latent here is a single 64-d (runs500/*_dz0, runs_causal/*) or 128-d (runs/mlp1_n4_wdp1) vector that must carry a whole sequence of uniformly random tokens. Three dimensions cannot hold that, so each scene picks a basis that makes one specific claim visible — a slot plane, a rotation plane, the direction of one token's code — and reports the captured energy. Arrows are per-position code vectors C_t[x_t]; the thick vector is their sum, which is the latent (pre-tanh; |u| is small enough that tanh is linear in every solved run).

Sources

Mechanism: mech_deep/DEEP_REPORT.md (mlp1_n4), mech_deep/DEEP_REPORT_ATTN.md (attn1_n4 and the hashing plateau), mech_deep/DEEP_REPORT_CAUSAL.md (causal rank collapse, graded amplitudes, DFT bridge). Geometry extracted by viz3d/extract.py (run on 4l, CPU) into viz3d/data/*.json; this page renders those JSONs and nothing else. Palette per site/CONTRACT.md.