ai gen
Literature review · model cards

The text autoencoders, one diagram each

For every encode-a-sentence-to-one-vector model in the literature review, plus the program's own in-house ladder: what the encoder and decoder were initialised from, how big they are, how wide the bottleneck is, what loss trained them, and on how much data. Every line carries the section of the paper it was read from; where a paper does not say, the card says not stated rather than guessing.

data encoder / decoder (initialisation and size) the vector loss and training
01

Bowman sentence VAE2016

Generating Sentences from a Continuous Space
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio — Stanford / UMass Amherst / Google Brain · arXiv:1511.06349

A single-layer LSTM sequence autoencoder with a Gaussian prior on the code, trained on the ELBO with KL annealing and word dropout to stop the decoder ignoring the latent. The origin of the posterior-collapse problem and of the two standard fixes.

in the review ↗
Input model
  • Initialisationno pretrained model; the encoder and decoder are single-layer LSTMs described as trained from scratch§3
  • Word embeddingsa learned dictionary, not pretrained vectorsFigure 1
  • Explicit "random init" statementnot stated in paper
  • Teacher / distillationnone§3
  • Relation to prior work"essentially forming a sequence autoencoder with the Gaussian prior acting as a regularizer on the hidden code"; same high-level architecture as the VRAE of Fabius and van Amersfoort (2014)§3
Model size
  • Architecturesingle-layer LSTM RNN encoder and single-layer LSTM RNN decoder§3
  • Books Corpus model512 hidden units§5
  • Classification-appendix model1200 hidden unitsAppendix, Text classification
  • Penn Treebank VAE (tuned)embedding dim 353, LSTM state dim 191, z dim 13; inputless-decoder variant 499 / 350 / 111Table 11
  • Penn Treebank RNNLM baseline (tuned)embedding dim 464, LSTM state dim 337; inputless variant 305 / 68Table 11
  • Parameter countnot stated in paper
  • Vocabulary size and tokenizernot stated in paper; an unk token and an eos token are used§3.1, §5
  • FFN width, attention headsnot applicable (LSTM)
Vector
  • Formationthe encoder LSTM's final state passes through two linear maps to the mean and variance of a diagonal Gaussian q(z|x); z is a single sample from itFigure 1, §2.2
  • Priorstandard diagonal Gaussian, μ = 0, σ = 1; reparameterization trick of Kingma and Welling (2015)§2.2
  • Dimensionality13 (PTB, standard decoder) and 111 (PTB, inputless decoder), both chosen by the hyperparameter tuner — Table 11; the Books Corpus models' z dim is not stated in paper
  • How the decoder reads itz is the initial state of the decoding LSTMFigure 1
  • Variant triedconcatenating z to the decoder input at every time step — "little difference"§3
  • Variants triedsoftplus variance parametrization, and feedforward stacks between encoder and latent and between latent and decoder; the latter only trains with highway layers§3
  • Richer posteriors triedDRAW-style multistep recognition and normalizing flows; "unable to reap significant gains over our plain vae"§3
  • Normalisation of znot stated in paper
  • Downstream usethe posterior mean is taken as the sentence vectorAppendix, Text classification
Loss / training
  • Objectivethe variational lower bound L(θ;x) = −KL(qθ(z|x) ‖ p(z)) + Eqθ(z|x)[log pθ(x|z)] ≤ log p(x)§2.2, Eq. 1
  • Estimatorreconstruction from one sample of q(z|x) per step, KL in closed form§2.2
  • Failure mode addressedwithout intervention q(z|x) collapses to p(z), KL goes to zero and the model is an RNNLM§3.1
  • KL cost annealinga weight on the KL term starting at 0 and raised gradually to 1, at which point the weighted cost is the true lower bound; "annealing from a vanilla autoencoder to a vae"§3.1
  • Annealing schedulesigmoid; rate tuned as a hyperparameter, value not stated§3.1, Figure 2
  • Word dropouta fraction of the ground-truth previous-word tokens the decoder conditions on are replaced with unk, parameterised by a keep rate k ∈ [0,1] and tuned for both VAE and RNNLM§3.1
  • Historyless / inputless decodingk = 0, so the decoder conditions only on the number of words emitted so far; its generation process is fully differentiable§3.1, §4
  • Keep rates used0.62 VAE / 0.66 RNNLM on Penn Treebank; a fixed 75% keep rate for the Books Corpus model and everything after itTable 11, §5
  • Standard embedding dropouttried in the decoder, "did not help the model learn to use the latent variable"§3.1
  • Both tricks requiredwithout word dropout and cost annealing together, training "reliably results in models with equivalent performance to the baseline rnnlm, and zero kl divergence"§4
  • Decoding directionthe Books Corpus models are trained to decode right-to-left, to shorten dependencies for the VAE§5
  • Optimizerstochastic gradient descent — §2.2; no variant named
  • Learning rate, batch size, steps/epochsnot stated in paper
  • Hyperparameter searchBayesian optimisation after Snoek et al. (2012), 10 hours per configuration, 12 in parallel, best of 200 runs on devAppendix, Hyperparameter tuning
  • Hardware and total wall-clocknot stated in paper
Data
  • Penn Treebank(Marcus et al., 1993), standard train–test split, for the §4 language-modelling experiments — §4; sentence and token counts not stated in paper
  • Books Corpus(Kiros et al., 2015) for imputation, analysis and classification: "text from 12k e-books, mostly fiction", "after pruning, contains approximately 80m sentences"§5
  • Held-out set10k Books Corpus sentences; train and test performance "very similar"§6.1
  • Languagenot stated in paper; all data and samples are English§5
  • Max sentence length, vocabulary cut-off, pruning criterianot stated in paper
  • Evaluation data320k sentences split 80/10/10 for the adversarial classifiers, half real endings and half imputed, with the final 20% of each sentence imputed; MSRP and TREC for the classification appendix§5, Appendix
Notes

On Penn Treebank language modelling with a standard decoder the VAE loses to the plain RNNLM: test NLL 101 with a KL term of 2 and test perplexity 119, against the RNNLM's 100 and 116 (Table 2) — the paper bills this as a negative result in the abstract, and notes the comparison is already conservative because the VAE figure is a lower bound while the RNNLM's is exact. The VAE only wins once the decoder is crippled: with an inputless decoder it reaches test NLL 125 (KL 15) and perplexity 380 against 135 and > 600. The KL term never dominates the reconstruction term in any run. On imputing the final 20% of Books Corpus sentences, evaluated by how well a discriminator separates completions from real endings, the VAE reaches 22.39% adversarial error against a unigram classifier and 35.59% against an LSTM classifier, versus 28.32% and 38.92% for a compute-matched RNNLM beam search (Table 4); the two models' sentence NLLs are near-identical at 46.14 and 46.01. Sweeping the word-dropout keep rate on Books Corpus moves information into z monotonically — KL 0.01 at 100% keep, 8.75 at 75%, 20.91 at 0% — while total likelihood degrades somewhat (Figure 3). As sentence features the posterior means are middling: 72.9 accuracy / 81.4 F1 on MSRP against skip-thought's 73.0 / 81.9, and 87.0 on TREC against skip-thought's 91.4, though they beat a plain sequence autoencoder's 84.2 and are complementary to skip-thought when concatenated (Tables 9, 10). Qualitatively, linear interpolations between codes stay grammatical (Tables 8, 12) where a plain sequence autoencoder's do not (Table 1).

Release: no code or weights release is mentioned anywhere in the paper.

This card was read from arXiv:1511.06349v4 over all 12 pages including both appendices. Parameter count, vocabulary size, tokenizer, learning rate, batch size, training steps, hardware, wall-clock, the numeric KL annealing rate, the Books Corpus latent dimensionality and the language are all left off because the paper does not state them; the only dimensionalities given are the tuner-selected Penn Treebank values in Table 11 and the 512 / 1200 hidden-unit counts for the Books Corpus models.

source: models/bowman-vae.md
02

SDAE (Hill, Cho, Korhonen)2016

Learning Distributed Representations of Sentences from Unlabelled Data
Felix Hill, Kyunghyun Cho, Anna Korhonen — Cambridge / NYU · arXiv:1602.03483

A recurrent encoder-decoder trained to reconstruct a sentence from a copy corrupted by word deletion and bigram swaps, with the corruption rate tuned on held-out data. The first sentence-level denoising autoencoder benchmarked head-to-head against SkipThought and bag-of-words baselines.

in the review ↗
Input model
  • Initialisationtrained from scratch; no pretrained encoder, decoder or teacher§2.3
  • +embs variantwords are represented by fixed pre-trained embeddings§2.3
  • Which pre-trained embeddingsnot stated in paper — the paper never names word2vec or any other source for the S(D)AE +embs runs, and gives no dimension
  • How the fixed embeddings enternot stated in paper; the released code maps them into the RNN through a learned projection and does not update themREADME
  • SAE siblingthe same model with the noise switched off, po = px = 0, N ≡ id§2.3
Model size
  • Architecture"the same LSTM-based encoder-decoder architecture as NMT", itself "a standard architecture (Cho et al., 2014)"§2.3, §2.2
  • Released code is GRU, not LSTMdesent.py defaults to encoder='gru', decoder='gru_cond', with no LSTM pathREADME
  • Word-embedding dimension100Table 1
  • Word-order sensitiveyes; requires neither ordered sentences nor a structured resourceTable 1, §2.3
  • Layers, parameter count, vocabulary, tokenizernot stated in paper (code defaults: one layer each side, 30k vocabulary, lower-cased text)README
Vector
  • Dimensionality2400 — Table 1 — the same as their NMT baseline; SkipThought is 4800, the log-linear models 100–500
  • Poolingnot stated in paper; the released code takes the encoder's final hidden stateREADME
  • How the decoder reads itnot stated in paper; in the released code it initialises the decoder through a tanh feed-forward layer and is added into the gate and candidate pre-activations at every stepREADME
  • Normalisation, noise on z, priornone; the model is a plain autoencoder, not a VAE§2.3
  • Encoding cost≈640 s for 0.5m sentences, and the dimension was large enough to make an online nearest-neighbour demo impracticalTable 1, §5
Loss / training
  • ObjectiveLrec alone — predict the original S from the corrupted source N(S|po, px); no second term, no weighting§2.3
  • Exact loss formnot stated in paper for SDAE; the summed per-token negative log-likelihood is spelled out only for SkipThought§2.1, §2.3
  • Noise, step 1each word w in S is deleted independently with probability po§2.3
  • Noise, step 2then each non-overlapping bigram wiwi+1 has wi, wi+1 swapped with probability px§2.3
  • Noise valuessearched po, px ∈ {0.1, 0.2, 0.3}, best at po = px = 0.1footnote 8
  • Tuning setthe 500 SICK 'trial' pairs, held out from both evaluation suites§2.4, §3.2
  • Epochs and hardwareone epoch on one GPU; GPU model not stated§2.4
  • Training time§2.4 says one epoch took ≈8 days; Table 1 gives 72 h in the S(D)AE row. The paper states both§2.4, Table 1
  • Optimizer, learning rate, batch size, update countnot stated in paper (code defaults: Adam, lr 0.01, batch 16, clip 2.0)README
Data
  • Corpusthe Toronto Books Corpus, used because it has the inter-sentential coherence SkipThought and FastSent need — SDAE itself does not§2.4, §2.3
  • Size70m ordered sentences from over 7,000 books; 0.9B words§2.4, Table 3
  • Languagenot stated in paper
  • Max sentence length, preprocessing, filteringnot stated in paper (code: max length 100, lower-casing only)README
  • Comparison scaleC-PHRASE, the strongest baseline on STS, is trained on 2.8B words, 3× more textfootnote 5, Table 3
Notes

Headline results (Table 3, supervised; Table 4, unsupervised Spearman/Pearson). SDAE is the best model in the paper on MSRP paraphrase identification at 76.4 / 83.4, three points above SkipThought, and beats the no-noise SAE on five of the six supervised tasks — the paper's main evidence that the corruption is doing work. Elsewhere it trails: MR 67.6, CR 74.0, SUBJ 89.3, MPQA 81.3, TREC 77.6 against SkipThought's 76.5 / 80.1 / 93.6 / 87.1 / 92.2. On the unsupervised suites it collapses — STS 2014 overall .17/.15 and SICK .46/.46, against CBOW's .64/.65 and FastSent's .63/.64 — which §5 attributes to the cost being computed through a non-linear decoding, leaving the informative geometry invisible to cosine distance. The +embs variants trade MSRP for everything else (SDAE+embs. 73.7 / 80.7 on MSRP, but .37/.38 on STS). No reconstruction BLEU and no perplexity are reported. Siblings in the same paper: SAE (the p = 0 ablation), FastSent and FastSent+AE (log-linear bag-of-words predicting adjacent sentences, 2 h to train), plus comparisons against SkipThought, ParagraphVec DBOW/DM, SkipGram, CBOW, TFIDF, DictRep, CaptionRep, NMT and C-PHRASE.

Release: footnote 13 promises code and pre-trained models but gives no URL; the code is at github.com/fh295/SentenceRepresentation (Theano, Python 2, no licence file, no SDAE weights) — README.

Everything on this card was read from arXiv:1602.03483v1 by an extraction pass over the full 11-page PDF (9 pages of content, no appendix); items marked README come from github.com/fh295/SentenceRepresentation, which the paper itself does not link. The parameter count, layer count, vocabulary size, optimizer, learning rate, batch size, sentence-vector pooling, decoder conditioning, corpus language, and the identity of the +embs embeddings are deliberately left off the sourced rows because the paper states none of them. The paper's two irreconcilable training times (≈8 days in §2.4, 72 h in Table 1) and its LSTM/GRU discrepancy with the released code are reported rather than resolved.

source: models/sdae.md
03

ARAE2018

Adversarially Regularized Autoencoders
Junbo (Jake) Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, Yann LeCun — NYU / Harvard SEAS / Facebook AI Research · arXiv:1706.04223

A 300-unit LSTM autoencoder whose code is normalised onto the unit sphere and matched to a learned MLP prior by a WGAN critic, trained by three-step alternation instead of a KL term. The adversarial-prior answer to VAE posterior collapse on text.

in the review ↗
Input model
  • Initialisationno pretrained component anywhere; encoder, decoder, generator and critic are all freshly parameterised§5, Appendix F
  • Word embeddingslearned as part of the model, size 300Appendix F
  • Explicit "random init" statementnot stated in paper; it follows only from the absence of any pretraining discussion
  • Distillation teachernone — not stated in paper
  • Baselinesstandard AE, RNN LM, recurrent VAE and AAE all trained from scratch at matched size, "All models are of the same size to allow for fair comparison"§6.1
Model size
  • Encoderone-layer LSTM, 300 hidden unitsAppendix F
  • Decoderone-layer LSTM, 300 hidden unitsAppendix F
  • Generator gθMLP 100-300-300, so noise s is 100-dAppendix F
  • Critic fwMLP 300-300-1, weight clipping ε = 0.01, 5 iterations per loopAppendix F
  • Attribute classifier (transfer only)MLP 300-200-100, SGD lr 0.1Appendix F
  • Parameter countnot stated in paper, for any component
  • Vocabulary / tokenizernot stated in paper; <unk> and _num_ appear in the sample sheets, so some UNK-ing and number normalisation was doneFigure 6, Figure 8
  • Other variantsMNIST ARAE is MLP 784-800-400-100 / 100-400-800-1000-784; the semi-supervised SNLI run only enlarges the GAN to generator 100-150-300-500 and critic 500-500-150-80-20-1, ε = 0.02Appendix F
Vector
  • Dimensionality300, not stated as such; forced by the 300-unit encoder LSTM, the 100-300-300 generator and the 300-300-1 criticAppendix F
  • Poolinglast hidden state of the encoder RNN, "encφ(x) = z to be the last hidden state of an encoder RNN"§5
  • Unit-norm constraintyes, "the encoder output is normalized to lie on the unit sphere", in both ARAE and the AE baseline§5, Appendix B
  • Generator outputbounded to (−1, 1)n by a tanh output layer, so prior samples are bounded but not on the sphere§5, Appendix B
  • Why boundedboth constraints are what Proposition 1's moment-convergence argument requiresAppendix B
  • Into the decoder (i)"we feed z as an additional input to the decoder RNN at each time step"§5
  • Into the decoder (ii)"The LSTM state vector is augmented by the hidden code c at every decoding time step, before forwarding into the output softmax layer"Appendix F
  • Noise on the codeadditive Gaussian noise injected into the code and annealed to zero; initial σ and decay rate not stated in paperAppendix F
  • Priorlearned, z̃ = gθ(s) with s ∼ N(0, I); a fixed N(0, I) prior is "too constrained and suffers from mode-collapse"§3
  • Decodinggreedy search, because the exact argmax required by Corollary 1 is intractable§5
  • Transfer modeldecoder conditioned as pψ(x | z, y); for Yelp this is two separate decoder RNNs, one per sentiment§3, §6.2
Loss / training
  • Objectiveminφ,ψ Lrec(φ,ψ) + λ(1)·W(PQ, Pz)§3
  • Lrecsentence-level cross-entropy, −log pψ(x | encφ(x)); §4 shows it upper-bounds the total variation distance between data and model distributions§2, §4
  • W termWasserstein-1 between the encoder's code distribution and the prior, estimated by an embedded critic§2, §3
  • Lipschitz constraintnaive weight clipping to [−ε, ε]d, ε = 0.01 for text; gradient penalty explicitly not tried§2, footnote 1, Appendix F
  • Alternation, one pass per iteration (Algorithm 1)(1) encoder+decoder on Lrec; (2) critic on E[fw(z)] − E[fw(z̃)], then clip; (3) encoder and generator adversarially on the same term with sign flipped§3
  • Generator updateθ is trained only in step 3, never in step 2§3
  • Critic inner loop5 iterations per loop for text, 10 for MNIST and for the semi-supervised variantAppendix F
  • λ(1) for the main SNLI text modelnot stated in paper; only λ(1) = 0.2 for MNIST and λ(1)a = 1 / λ(1)b = 10 for Yelp/Yahoo are given§6.2, Appendix F
  • Transfer extensionadds −λ(2)·Lclass(φ,u) with λ(2) = 1 always, plus two more steps per loop, (2b) train the latent attribute classifier and (3b) train the encoder to fool it§3, §6.2
  • Autoencoder optimizervanilla SGD, learning rate 1, gradient clipping max_grad_norm = 1Appendix F
  • GAN optimizerAdam, lr 5e-5 on the generator and 1e-5 on the criticAppendix F
  • Corruption / denoisingnone; "the ARAE is not explicitly trained to denoise an input"§7
  • KL weight, KL annealing, word dropoutn/a, there is no KL term — not stated in paper
  • Batch size, epochs, wall-clocknot stated in paper
  • Early stoppingreverse perplexity "was helpful for early-stopping"§6.1 footnote 7
  • Hardwarea donated NVIDIA Titan X Pascal GPU "was used for this research"; no count or durationAcknowledgements
Data
  • Text generation and semi-supervisedSNLI (Bowman et al. 2015)§5, §6.1, §6.3
  • SNLI size"The full SNLI training set contains 543k sentence pairs"§6.3
  • Semi-supervised splits120k Medium, 59k Small, 28k Tiny labelled pairs, the remainder used unlabelled; 22.2% / 10.8% / 5.25% of the training set§6.3, Table 5
  • Length filterstated only for the semi-supervised run, "the subset of unsupervised data of length < 15", which means ARAE sees less unlabelled data than the AE baseline§6.3
  • Max sentence length for the main SNLI modelnot stated in paper
  • Sentiment transferYelp, split into unaligned positive and negative reviews following Shen et al. 2017; size not stated in paper§6.2
  • Topic transferYahoo QA (Zhang et al. 2015), three classes — Science & Math, Entertainment & Music, Politics & Government — run separately on titles and on replies; size not stated in paper§6.2, Table 4
  • Discretised imagesbinarized MNIST, the non-text arm of the paper§5
  • Languagenot stated in paper; every corpus and sample shown is English
  • Vocabulary cut-off, preprocessing, filteringnot stated in paper
Notes

Generation on SNLI (Table 1) is scored by reverse perplexity, an LM trained on 100k generated samples and evaluated on real held-out data: ARAE 82.2 against LM samples 90.6 and AE samples 97.3, with real data at 27.4; forward PPL runs the other way, ARAE 44.3 against LM 18.8 and AE 87.8. A fixed-prior AAE-style text model reaches reverse PPL 980 through mode collapse (§6.1), which is the paper's case for a learned prior. Yelp sentiment transfer (Table 3) trades transfer rate against content: ARAE at λ(1)b = 10 hits 81.8% automatic transfer and 20.18 BLEU where the AE baseline gets 59.3% and 37.28; human raters score it 74% transfer and 3.8 naturalness against the cross-aligned AE's 57% and 2.7. Semi-supervised SNLI accuracy (Table 5) is 70.9 / 66.8 / 62.5 for ARAE against 68.5 / 64.6 / 59.9 for a plain AE. Under k-word swaps the plain AE reconstructs better at k = 0 (NLL 1.06 vs 2.19) but ARAE wins from k = 1 onward, reaching 7.47 vs 9.97 at k = 4 (Figure 4). No reconstruction BLEU or autoencoding accuracy is reported anywhere, which is the gap most relevant to a TAE reader.

Release: code at github.com/jakezhaojb/ARAE (§1), BSD 3-Clause per the repository metadata; no pretrained weights, and the repo's own disclaimer records that WGAN-GP later replaced WGAN in the code (README).

This card was read from arXiv:1706.04223v3, the ICML 2018 version, over all 16 pages including Appendices A–G. Deliberately left off because the paper does not state them: parameter counts, vocabulary size and tokenizer, max sentence length for the main SNLI model, Yelp and Yahoo corpus sizes, batch size, epoch or step counts, and λ(1) for the SNLI text model. Note that the paper uses WGAN with weight clipping, not WGAN-GP — footnote 1 explicitly disclaims having tried the gradient penalty; the code-level values σ₀ = 0.2 for the code noise, the GAN-loop schedule, and the 500-unit widening for Yelp/Yahoo appear in v2's appendix but not in v3, and are recorded only in the fact sheet.

source: models/arae.md
04

CP-VAE2020

On Variational Learning of Controllable Representations for Text without Supervision
Peng Xu, Jackie Chi Kit Cheung, Yanshuai Cao — Borealis AI / McGill / Mila · arXiv:1905.11975

Diagnoses why manipulating a sequence-VAE latent code breaks decoding — the edited code lands in vacant regions of the aggregated posterior — and fixes it by pinning half the latent onto a learned orthogonal probability simplex and adding a margin loss that forces the simplex to fill.

in the review ↗
Input model
  • Initialisationencoder and decoder networks trained from scratch; "all the other models including ours train the decoding network from scratch"§5.2
  • Only pretrained partthe input feature for z(1), averaged GloVe embeddings over the input tokens, fed to an MLP§4.1, §5.1
  • CP-B variantthe same slot filled with BERT sentence representations instead of GloVe; BERT size, layer and pooling not stated in paper§5.2
  • GloVe variant usedglove.840B.300d; the paper gives no dimensionREADME
  • Teachernone. GPT-2 345M and CNN classifiers appear only as evaluation metrics§5.2
Model size
  • ArchitectureLSTM sequence VAE with a split encoder — MLP for z(1), LSTM for z(2)§4.1
  • LSTM encoderinput 256, hidden 1024 (Yelp, Amazon); input 512, hidden 1024 (AG-News)Table 6
  • LSTM decoderinput 128, hidden 1024 (Yelp, Amazon); input 512, hidden 1024 (AG-News)Table 6
  • Layer count, bidirectionality, MLP depth/widthnot stated in paper
  • Parameter countnot stated in paper, for any variant
  • Vocabulary / tokenizernot stated in paper; preprocessing deferred to the He et al. 2019 codebaseAppendix A
  • Dropout0.5 after the embedding layer and the LSTM layer, decoder onlyAppendix C.1
  • Diagnostic β-VAE of §3LSTM encoder and decoder, input 256 / hidden 1024, latent 80, β = 0.35Appendix B.1
Vector
  • Split latentz = (z(1), z(2)), qφ(z|x) = qφ1(z(1)|x)·qφ2(z(2)|x), so the KL splits in two; after Bao et al. 2019§4.1
  • Dimensionsz(1) = 16, z(2) = 64, total 80 (Yelp, Amazon); z(1) = 32, z(2) = 96, total 128 (AG-News)Table 6, §5.1
  • Constraint on z(1)the posterior mean is forced into a K-dim probability simplex in RN, µ = Σ piei with Σ pi = 1, ⟨ei, ej⟩ = 0 for i ≠ j§4.2, Eq. 2
  • Mappingµ = π(h) = E·softmax(Wh + b), with E the learnable basis matrix; π counts as part of the encoder§4.2, Eq. 3
  • K3 for Yelp and Amazon, 10 for AG-NewsTable 6, §5.1, §5.3
  • z(2)ordinary Gaussian posterior against an isotropic prior, unconstrained§4.1
  • How the decoder reads znot stated in paper — initial state vs. per-step concatenation is never specified
  • Normalisationnone; scale is set only by α, which fixes ‖ei‖² = α once LREG ≈ 0§4.2
  • Manipulationz(1) is replaced outright by a chosen basis vector vp or vn, picked from the average pi of 10 positive and 10 negative dev sentences§5.1
Loss / training
  • ObjectiveL = LVAE + LREG + LS-REC, a plain sum with no scalar weights written on the three terms§4.3, Eq. 8
  • LVAEthe ELBO of Eq. 1, with the KL term split into a z(1) and a z(2) part§2, §4.1
  • LREG‖EᵀE − αI‖, enforcing orthogonal bases; with orthogonality µᵀµ = α·Σpi², which bottoms out at α/K, so the z(1) KL cannot fully collapse§4.2, Eq. 4–6, Appendix D
  • LS-RECmargin loss (1/m)·Σ max(0, 1 − h·µ + h·µ⁻i) over m random training sentences as negatives, pushing p away from uniform so the simplex fills§4.3, Eq. 7
  • mnot stated in paper
  • α100 for Yelp and Amazon, 10 for AG-NewsTable 6
  • KL weightsβ = 0.2 on z(1) for every dataset; on z(2), β = 0.35 for Yelp and Amazon, KL annealing 0.1 → 1.0 over 10 epochs for AG-NewsAppendix C.2
  • OptimizerAdam lr 0.001 for the encoding network, SGD lr 1.0 for the decoding network, batch size 32Appendix C.1
  • Scheduletrained "until the reconstruction loss stops decreasing"; epochs, steps, hardware and wall-clock not stated in paperAppendix C.1
  • Corruption / word dropoutnot stated in paper
  • Decodingbeam search, beam size 5Appendix C.4
  • Tuninggrid over K, α and the four width hyperparameters, selected for low reconstruction loss and high KL; Amazon reuses the Yelp setting untunedAppendix C.3
Data
  • Yelprestaurant reviews, the Li et al. 2018 split; sentence count not stated in paper§3.1, §5.2
  • Amazonproduct reviews, the same Li et al. 2018 / Sudhakar et al. 2019 splits; size not stated in paper§5.2
  • Bothhuman gold-standard references exist for the test sets§5.2
  • AG NewsZhang et al. 2015, four topics, 30,000 train and 1,900 test samples per category, description field only, title dropped§5.3, Appendix E
  • Languagenot stated in paper
  • Max sentence length, filtering, token countsnot stated in paper; preprocessing deferred to external codebasesAppendix A
  • Analysis subsets1000 test codes for the NLL histograms, 100,000 training points for the topological analysis§3.2, Appendix B.5
Notes

The diagnosis comes first. A β-VAE with aggressive encoder training (He et al. 2019) on Yelp, 80-d latent, yields one dimension z[s] that classifies sentiment at ~75% against ~50% for the rest (§3.1); moving it by σ, 2σ or to the training extremum either fails to flip sentiment or destroys the content. Two probes explain why: NLL of the manipulated codes under a mixture-of-Gaussians estimate of the aggregated posterior rises with the manipulation magnitude (§3.2, Fig. 2), and the mapper algorithm on the highest-density region gives one connected component for β-VAE on OMNIGLOT but many disconnected components at every scale for β-VAE on text (§3.3, Fig. 3). CP-VAE closes both gaps — flat NLL discrepancy, one connected component for z(1) (§3.4). On Yelp against unsupervised baselines it gets 60.22 accuracy / 33.69 BLEU where β-VAE trades one against the other (26.76/27.91 at ±σ, 88.08/4.57 at the extremum); the ablations are decisive, dropping LREG collapses accuracy to 10.82 and dropping LS-REC to 12.28 (Table 2). Against supervised style transfer (Table 3) CP-G reaches 66.7 AC / 35.5 BL / 7.5 GLEU / 67.8 PPL on Yelp and 60.1 / 35.4 / 11.5 / 109.1 on Amazon, below the GPT-2-based B-GST on GLEU but above the adversarial baselines, which score 0.0 GLEU on Amazon. On AG News with K = 10 the discovered bases match the four gold topics at F1 74.59 / 79.49 / 63.97 / 66.77, comparable to LDA and well above k-means on pooled embeddings (Table 7), and generation can switch basis vector mid-sentence and still finish fluently (§5.3, Table 5).

Release: code at github.com/BorealisAI/CP-VAE under CC BY-NC-SA 4.0; no checkpoints published, training is from scratch (README).

This card was read from arXiv:1905.11975v4 over all 16 pages including Appendices A–G. Deliberately left off because the paper does not state them: any parameter count, the vocabulary and tokenizer, the LSTM layer count, the number of negative samples m, the GloVe and BERT dimensions (300 and 768 are repo/common knowledge, not paper facts), the Yelp and Amazon corpus sizes, the maximum sentence length, and the mechanism by which z reaches the decoder.

source: models/cp-vae.md
05

DAAE2020

Educating Text Autoencoders: Latent Representation Guidance via Denoising
Tianxiao Shen, Jonas Mueller, Regina Barzilay, Tommi Jaakkola — MIT CSAIL / AWS · arXiv:1905.12777

A single-vector LSTM adversarial autoencoder trained to reconstruct a sentence from a word-deleted copy of itself. The paper proves a high-capacity autoencoder can pair sentences with latent points arbitrarily, and that input perturbation is what forces the pairing to respect similarity.

in the review ↗
Input model
  • Initialisationrandom; no pretrained encoder, decoder or embedding table is used anywhere§3, Appendix F
  • One architecture, many objectivesAE, DAE, VAE, β-VAE, AAE, LAAE and DAAE are all implemented with the same network, differing only in the lossAppendix F
  • Teachernone; E, G and D are trained jointly from scratch under a min-max objective§3
  • Recurrent, not Transformer, by measurement"We also tried Transformer models … but they did not outperform LSTMs on our moderate-size datasets"§3, footnote 2
  • Random-init baselinean untrained RNN encoder already has good neighborhood recall, and reconstruction-only training does not overturn that biasAppendix H
Model size
  • Encoderone-layer LSTM, hidden dimension 1024, word embedding dimension 512Appendix F
  • Decoderone-layer LSTM, hidden dimension 1024, autoregressive over emitted symbols§3, Appendix F
  • Discriminatorfeed-forward MLP, one hidden layer of size 512, outputs P(z from prior)§3, Appendix F
  • Parameter countnot stated in paper
  • Vocabulary10K (Yelp), 20K (Yahoo); tokenizer not stated in paper, corpora are pre-tokenised with _num_ and _UNK§5 Datasets, Table 1
  • Directionalitynot stated in paper; the released encoder is a bidirectional LSTM whose two final states are concatenated to 2048 before projectionREADME
Vector
  • Dimensionality128 for Yelp, 256 for YahooAppendix F
  • Poolingthe encoder's final LSTM hidden state, linearly projected down to the latent dimension§3, Appendix F
  • Encoder is deterministicstochastic encodings arise only from the input perturbation, q(z|x) = Σx̃ pC(x̃|x)·pE(z|x̃), "without any reparameterization-style tricks"§3, Eq. 6
  • How the decoder reads itz is projected and added to the input word embedding at every decoder step; not an initial state, not cross-attentionAppendix F
  • Priora given p(z), described as "typically Gaussian"; the experimental prior is not further specified§1, §3
  • Noise on znone — latent Gaussian perturbation is the LAAE baseline's mechanism, and the paper's argument is that input-space noise works where latent-space noise does not§5 Baselines, §5.2
  • Normalisation of znot stated in paper
Loss / training
  • ObjectiveminE,G maxD Lrec(θE,θG) − λ·Ladv(θE,θD)§3, Eq. 1
  • LrecEp(x,x̃)[−log pG(x | E(x̃))] — the clean sentence is the target, the perturbed one the input§3, Eq. 4
  • LadvEp(z)[−log D(z)] + Ep(x̃)[−log(1 − D(E(x̃)))], evaluated on encodings of the perturbed sentences§3, Eq. 5
  • λ10, "to ensure the latent codes are indistinguishable from the prior"Appendix F
  • Perturbation recipedelete each word independently with probability p; final setting p = 0.3§5 Perturbation Process, §5.2
  • p is the trade-off knobswept across the full generation–reconstruction curve, as β is for β-VAE and λ1 for LAAE (finalised at β = 0.15, λ1 = 0.05)§5.2, Figure 4
  • Perturbation variants<mask> replacement and random-word replacement give similar trade-off curves; word deletion has the highest neighborhood recall§5 Perturbation Process
  • GAN practicalitiesE is trained to maximise log D(E(x)) rather than −log(1−D(E(x))); WGAN gave no gains§3, footnote 3
  • OptimizerAdam, learning rate 0.0005, β1 = 0.5, β2 = 0.999Appendix F
  • Test timeperturbations disabled, greedy decodingAppendix F
  • Batch size, epochs, annealing schedule, hardware, wall-clocknot stated in paper; the repo defaults are batch 256, 50 epochs, dropout 0.5README
Data
  • Yelp reviews(Shen et al., 2017): 444K / 63K / 127K sentences train/dev/test, all under 16 words, vocabulary 10K; the sentiment labels are discarded for training§5 Datasets
  • Yahoo answers(Yang et al., 2017): document-level source, sentence-segmented and filtered to 2–50 words, giving 495K / 49K / 50K sentences, vocabulary 20K§5 Datasets
  • Languagenot stated in paper; English by implication of the corpora§5 Datasets
  • Token countnot stated in paper
  • Synthetic corpus for the geometry experiment500 binary sequences of length 50, five clusters of 100, generated by flipping cluster-centre bits with probability 0.2; latent dimension 2, repeated at 5 with t-SNE§4.1, Appendix B
  • Probe sets for vector arithmetic100 past and 100 present dev sentences for the tense vector, 100 negative and 100 positive for the sentiment vector, 200 test sentences for human evaluation§5.3, Appendix G
Notes

The theoretical core is Theorem 1 (§4.2, proof in Appendix C): under an unconstrained encoder, an L-Lipschitz decoder and a discriminator that matches the prior, every one-to-one mapping from {x1,…,xn} to {z1,…,zn} yields the same optimal reconstruction objective, so an AAE can learn an arbitrary bijection between sentences and latent points while reconstructing perfectly. Theorem 2 (Appendix D) shows the four-point denoising objective is maximised when close x pairs go to close z pairs, and Theorem 3 (Appendix E) bounds the clustered-data denoising objective by (1/n²)ΣSi≠Sj log σ(L‖E(xi)−E(xj)‖) − log K, which is looser when clusters are separated in latent space. Empirically: DAAE has 1.5–2× AAE's neighborhood recall on Yelp (Figure 3; ARAE below 1%), and a generation–reconstruction curve strictly below the others (Figure 4, Yahoo in Figure I.3) — between a VAE at β = 1 whose reconstruction BLEU is near 0 and an AAE that reconstructs near-perfectly with poor Forward PPL. Zero-shot tense inversion by adding a fixed latent offset: ACC 50.3 / BLEU 54.3 / PPL 32.0, against β-VAE 49.0 / 43.5 / 44.4 and AAE 9.7 / 82.2 / 37.4 (Table 2); human raters prefer DAAE 48 to 25 over β-VAE, with 44.6% of tenses correctly inverted. Sentiment transfer scales with the offset: 36.2 / 40.9 / 40.0 at ±v, 73.6 / 18.2 / 54.1 at ±1.5v, 91.8 / 7.3 / 61.8 at ±2v, against the supervised Shen et al. (2017) system at 81.7 / 12.4 / 38.4 (Table 3).

The paper's family: AAE and DAAE are the objects of study; β-VAE, LAAE (latent Gaussian noise plus an L1 penalty on the log-variance) and ARAE are the main-text baselines; plain AE and DAE appear only in the Appendix H recall analysis.

Release: code and processed data at github.com/shentianxiao/text-autoencoders, Apache 2.0, PyTorch 1.1 — no pretrained checkpoints (README).

Everything on this card was read from arXiv:1905.12777v3 over 18 pages, covering §1–§6 and Appendices A–J; items marked README come from github.com/shentianxiao/text-autoencoders. Parameter count, batch size, epoch count, hardware, wall-clock time, tokenizer, latent normalisation and the explicit prior distribution are deliberately left off because the paper does not state them, and no reconstruction-BLEU number is quoted because those values appear only as points in Figure 4 and are never tabulated.

source: models/daae.md
06

Optimus2020

"Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space"
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, Jianfeng Gao — Microsoft Research · arXiv:2004.04092

A pretrained BERT encoder bolted to a pretrained GPT-2 decoder through a single Gaussian latent, trained as a β-VAE on 2M Wikipedia sentences with cyclical β annealing and per-dimension KL thresholding. The first sentence VAE built at PLM scale.

in the review ↗
Input model
  • Initialisationencoder and decoder both initialised from pretrained PLM weights, not from scratch§4.2
  • EncoderBERTBASE, "L=12, H=768, A=12, Total Parameters=110M"§4.2
  • DecoderGPT-2, "L=12, H=768, A=12, Total Parameters=117M", i.e. the smallest GPT-2§4.2
  • Exact checkpointsbert-base-cased and gpt2README
  • Frozen or fine-tunedfine-tuned end-to-end during pre-training; φ = {φBERT, WE}, θ = {θGPT-2, WM, WD}§4.2
  • Freezing downstream onlybackbone frozen for label-conditional generation and for the feature-based GLUE scheme; first 11 encoder layers frozen for dialog§5.2, §5.3, §C.2
  • Teachernone; the recipe as specified contains no distillation term§4.2, §4.3
  • Segment embeddingsretained though unnecessary, "due to BERT initialization"§4.2 footnote 3
Model size
  • Encoder12 layers, hidden 768, 12 heads, 110M parameters§4.2
  • Decoder12 layers, hidden 768, 12 heads, 117M parameters§4.2
  • Total parameter countnot stated in paper; the two component counts are given separately and never summed, and WE/WM/WD are given only as shapes
  • FFN / intermediate widthnot stated in paper
  • Encoder tokenizerWordPiece, vocabulary 28996 (cased)§4.2
  • Decoder tokenizerGPT-2 modified BPE, vocabulary 50260§4.2
  • Asymmetric tokenisationencoder input is WordPiece, decoder output is BPE, and the reconstruction loss is computed in BPE§4.2
  • Variantsone size only; scaling to larger encoders/decoders is left to future work§4.2
Vector
  • Formationz = WEh[CLS], where h[CLS] ∈ RH is the last-layer hidden state of the [CLS] token and WE ∈ RP×H§4.2
  • Dimensionalitynot stated in paper; AU = 32 in every Optimus row of Tables 1 and 8–11 is the only hint. The repo uses 32 for language modeling and 768 for latent manipulationREADME
  • Distributionqφ(z|x) Gaussian, reparametrisation trick; Gaussian sampling is kept even in the AE variant at β = 0§4.1
  • Priormultivariate Gaussian§4.1
  • Memory injectionhMem = WMz with WM ∈ RLH×P; hMem is split into L vectors of length H, one attended by GPT-2 in each layer alongside x<t§4.2, Figure 2a
  • Embedding injectionh′Emb = hEmb + WDz with WD ∈ RH×P, added at the embedding layer and used at every decoding step§4.2, Figure 2b
  • Defaultthe integration of both schemes§4.2, §B.1
  • Ablationmemory beats embedding by a clear margin on reconstruction error per word, and the combination is slightly better still; attributed to memory exposing z at every layer rather than only at input and output§4.2, §B.1, Figure 5
  • Normalisation of znot stated in paper
  • Downstream classifiers read h[CLS], not z,to match BERT's trainable-parameter count; the paper argues the KL on z reshapes h[CLS] anyway§5.3
Loss / training
  • ObjectiveLβ = LE + β·LR, with LE = −Eqφ(z|x)[log pθ(x|z)] and LR = KL(qφ(z|x)‖p(z))§4.1, eqs (4)–(6)
  • Two regimesAE (β = 0, regulariser removed) and VAE (β > 0)§4.1
  • Reconstruction is sentence-level,not the token-level masked-LM loss used by BERT and XLNet§4.1
  • β schedulecyclical over 10 periods; within a period, β = 0 for the first 0.5 of the period, annealed 0→1 over the next 0.25, held at 1 for the final 0.25§4.3
  • KL thresholding (free bits)for β > 0, LR is replaced by L′R = Σi max[λ, KL(qφ(zi|x)‖p(zi))], per latent dimension i§4.3, eq (9)
  • λ values0.05, 0.10, 0.25, 0.50, 1.00 (0.01 in place of 0.10 on Yelp); λ = 0.50 called the good trade-offTable 1, Tables 8–11
  • Information-bottleneck readingFβ ≥ −Iq(z, x̃) + β·Iq(z, x), so the objective upper-bounds the IB Lagrangian§4.1 eq (8), §A
  • Noise / corruptionnone; no word dropout, masking or denoising appears in the objective, and the posterior sample is the only stochasticity§4.1, §4.3
  • Optimizer, learning rate, batch size, pre-training epochs/stepsnot stated in paper; the released pre-training checkpoints are all at step 508523README
  • Hardware and time"around 50 hours for one epoch on eight V100 DGX2 GPU's", on Microsoft's Philly cluster§C.1, Acknowledgments
  • LM fine-tuningload the pretrained model and run one further β cycle for one epoch§5.1
  • GLUE fine-tuninglearning rate [2, 3, 4, 5] × 10−5, 3 epochs, best run selected§5.3
  • Yelp low-resource protocol1 to 10K labelled samples per class, 10 trials at small sizes, 100 epochs each§5.3
  • Label-conditional generationfine-tune the VAE, freeze the latent space, then train a conditional GAN over it and decode from z = G(ε, y)§C.2.1, eqs (16)(17)
  • Dialog fine-tuningSpaceFusion-style, zτ = τzS2S + (1−τ)zAE with τ ∼ Uniform(0,1), first 11 encoder layers frozen§C.2
  • Self-assessed under-training"still under-trained due to limited compute resource, as the training reconstruction loss can still decrease"§6
Data
  • CorpusEnglish Wikipedia§4.3
  • Size1990K sentences, 96.45% of the Wikipedia sentences used in BERT§4.3, §B.2
  • Max sentence length64§4.3, §B.2, Figure 6
  • Unitnatural sentences, explicitly not the fixed-length text sequences GPT-2 trains on§4.3
  • LanguagesEnglish only§4.3
  • Token countnot stated in paper
  • Other filteringnot stated in paper beyond the length cut
  • LM fine-tuning setsPTB, SNLI, Yahoo, Yelp; per-dataset sizes not stated§6.1
  • DialogDailyDialog, 13,118 conversations, each utterance conditioned on the previous 10 context utterances§5.2
  • Label-conditionalshort Yelp of Shen et al. (2017), 444K train, 10K validation, 10K test§5.2
  • UnderstandingYelp sentiment at 1–10K labels per class, and the nine GLUE datasets§5.3, Table 7
Notes

Best-λ importance-weighted perplexity is 23.58 on PTB, 21.99 on Yelp, 22.34 on Yahoo and 13.47 on SNLI against GPT-2's 24.23 / 23.40 / 22.00 / 19.68, so Optimus is ahead on three of four datasets, and far ahead of the two-layer-LSTM VAEs it replaces (PTB 96.86–108.81); all 32 latent units stay active at every λ, where annealing and aggressive-training baselines collapse to 0–12 (Tables 1, 8–11). Downstream: DailyDialog BLEU F1 0.336 against iVAEMI 0.285, stylized-response F1 0.316 against StyleFusion 0.294, label-conditional Yelp G-score 0.630 against Control-Gen 0.584, and a feature-based GLUE validation average of 0.607 against BERT's 0.531 — though under full fine-tuning the gap closes to 0.798 against 0.793 (Tables 4–7). Latent arithmetic zD = zB − zA + zC and linear interpolation are shown qualitatively in Tables 2, 3 and 13–19; no reconstruction BLEU or round-trip accuracy is reported anywhere, only reconstruction NLL in nats.

Release: code on GitHub with no LICENSE file (GitHub API reports license: null), five Wikipedia pre-training checkpoints at checkpoint-508523 (β ∈ {0.0, 0.5} at latent 32; β ∈ {0.0, 0.5, 1.0} at latent 768) and three SNLI-tuned latent-768 checkpoints at checkpoint-31250, hosted on Azure Blob rather than the HuggingFace Hub (README).

This card was read from arXiv:2004.04092v4 over all 22 pages, body and appendices A–C and Tables 1–21; items tagged README come from github.com/ChunyuanLI/Optimus. Deliberately left off the sourced rows: the latent dimension (the paper never writes P, and 32/768 are repo facts), the total parameter count, the FFN width, and the pre-training optimizer, learning rate, batch size and epoch count — none of which the paper states. Note also that the v4 PDF carries a duplicated draft layer on pages 6–8, that Table 1's Yelp λ labels disagree with Table 9's, and that Table 1 and Table 11 report different GPT-2 perplexities on SNLI (19.68 vs 20.24).

source: models/optimus.md
07

Autobot (sentence bottleneck AE)2021

Sentence Bottleneck Autoencoders from Transformer Language Models
Ivan Montero, Nikolaos Pappas, Noah A. Smith — University of Washington / Allen Institute for AI · arXiv:2109.00055

A frozen RoBERTa-base with an attention-pooling bottleneck and a single-layer decoder bolted on, trained for 100k steps on BERT's corpus with a masked reconstruction loss. Adds 1.6% parameters to the pretrained model and changes nothing inside it.

in the review ↗
Input model
  • Initialisationthe encoder is a pretrained transformer LM; the paper works with RoBERTa, base for the main configuration§2, §2.2
  • Frozenthe encoder is fixed during autoencoder training; only the bottleneck and the decoder are learned§2.2, Table 4
  • Trained from scratchthe attention-pooling bottleneck β and the single-layer decoder§2.1, §2.2, Figure 1
  • Decoder initialisationnot stated in paper; no checkpoint is named for it and no weight sharing with the encoder is described
  • Teachernone described; no distillation step appears anywhere in the paper (not stated in paper)
  • Larger variantAUTOBOT-large over RoBERTa-large, reported on GLUE only — Table 3, Table 8; withheld from STS for resource reasons§3.2
  • The freeze is autoencoding-onlyduring downstream NLI and GLUE finetuning all parameters, encoder included, are unfrozenAppendix A.2, A.4
  • Unfreezing during autoencoding hurtsSTS falls from 78.59 to 77.24 / 76.17 / 76.20 as 1 / 2 / 3 top RoBERTa layers are made trainableTable 6
Model size
  • Total trained parameter size127M for AUTOBOT-base, against 125M for RoBERTa-base and 355M for RoBERTa-largeTable 1, Table 4
  • Overhead1.6% additional parameters relative to RoBERTa-base§1, §3.2
  • Bottleneckquery, key and value transforms, given as 3d² with "d the dimensionality of each head (d = 64)" — §2.1; see Notes, this does not reconcile with the 1.6%
  • Depthdecoder is 1 layer — Table 4, §2.2; encoder layer count not stated in paper, only the name RoBERTa-base
  • WidthsHidden Size 768, Bottleneck Heads 12 — Table 4; FFN width and decoder head count not stated in paper
  • Architecturetransformer throughout, framed as extending recurrent text autoencoders to transformers§2
  • TokenizerRoBERTa-base's pretrained tokenizer — Appendix A.1; vocabulary size not stated in paper
  • Max sequence length128Table 4
Vector
  • Dimensionalitynot stated in paper as a number; z is "of size d" where H is T×d, and the only width given is Hidden Size 768Figure 1, Table 4
  • Poolingβ(H; θ) = MultiHead(q, K, V) over the frozen model's final hidden states, chosen because CLS and basic pooling were shown sub-optimal by prior work§2.1, Eq. (2)
  • Querythe pretrained model's CLS vector, used as the single context vector u§2.1
  • Heads12 bottleneck heads, per-head dimensionality 64Table 4, §2.1
  • How the decoder reads itz is the entire cross-attention memory; with one slot, Attention(Q, zTWK, zTWV) = zTWV and the queries drop out§2.2, Eq. (3)
  • Gategt = σ(GQt + G′z), ot = gt ⊙ zTWV, an LSTM-style fix that restores per-step dependence§2.2, Eq. (4)
  • Other conditioning pathsnone; z is not an initial state, not concatenated to inputs, not added to embeddings§2.2
  • Noise, KL, prior on znone; the model is a deterministic denoising AE, not a VAE§2, §4
  • Normalisation of znot stated in paper
  • Pooling ablationβ 81.88 vs MEAN 80.78, CLS 79.67, MAX 78.76 on STS-B devTable 2
Loss / training
  • ObjectiveLrec only — negative log-likelihood over the reconstructed inputs, a single term with no weighting or annealing§2.2
  • Denoisingthe masked LM objective is adapted into a generative denoising one; the input is perturbed and the clean input reconstructed§1, §2
  • Corruption"the same masking procedure as Devlin et al. (2019)"; mask rate and replacement split not restated§2
  • OptimizerAdam, LR 1e-3 (searched over {1e-3, 1e-4, 1e-5}), linear decay, 4000 warmup, dropout 0.1Table 4
  • Steps100k, searched over {1k, 10k, 100k}Table 4, §2.2
  • BatchMax Tokens 24576; no sentence-count batch size, no epoch countTable 4
  • Hardware5× RTX 2080 Ti (11GB, 128GB RAM) for base models, 4× TITAN RTX (24GB, 256GB RAM) for largeAppendix A.1
  • Wall-clocknot stated in paper
  • Step-count ablationSTS 74.38 / 75.45 / 78.01 / 78.59 at 1 / 1k / 10k / 100k steps, against an SBERT-style baseline of 77.03Table 5
  • Optional in-domain stagea further 10k steps on Yelp, same recipe, for the generation experiments§3.4, Appendix A.3
  • SoftwarePython 3.7, PyTorch 1.6.0, Sentence-Transformers 0.3.7, modified Fairseq 0.9.0 and Transformers 3.3.1Appendix A.1
Data
  • CorpusBooksCorpus and English Wikipedia, used because "the RoBERTa dataset is not publicly available"§3.1, Appendix A.1
  • Sizenumber of sentences and tokens not stated in paper
  • LanguageEnglish only§3.1
  • Preprocessingsentences extracted from both corpora, tokenized with the RoBERTa-base tokenizer, capped at 128 tokensAppendix A.1, Table 4
  • Filteringnot stated in paper
  • Downstream corporaNLI (Bowman et al. 2015) for similarity finetuning, STS (Cer et al. 2017) for evaluation, GLUE, and Yelp reviews for style transfer§3.1
Notes

On STS the supervised AUTOBOT-base reaches 78.59 Spearman against SRoBERTa-base 76.89 and RoBERTa-base 75.37, close to RoBERTa-large's 80.16 with 127M trained parameters instead of 355M (Table 1); unsupervised it reaches 58.49 against 46.35 for averaged BERT embeddings. On GLUE dev it moves CoLA from 63.6 to 66.0 and SST from 94.8 to 95.0 while the multi-sentence average is unchanged at 88.7, with MRPC the one regression (88.4 vs 90.2, Table 8); the same pattern holds at large scale. On Yelp sentiment transfer by vector arithmetic it reports 95.6 accuracy at 11.90 self-BLEU, at 0.5× the inference cost of Emb2Emb and 1/140th that of FGIM (Table 7), and matches an in-domain-trained DAE without ever seeing Yelp (Figure 2). No reconstruction metric — BLEU, accuracy or perplexity — is reported anywhere, despite the conclusion's claim that the model "is able to reconstruct its pretraining data"; and SentEval is not used, the similarity protocol being SBERT's NLI→STS pipeline.

Release: code at github.com/ivanmontero/autobot, no trained checkpoints and no licence file (README).

This card was read from arXiv:2109.00055v2 over all 10 pages, main text, references and both appendices. Deliberately left off the sourced rows: the sentence-vector dimension as a bare number (the paper gives only "z of size d" and a Hidden Size 768 hyperparameter row), the decoder's initialisation and any embedding tying, RoBERTa's own layer/FFN/head/vocabulary configuration, BERT's mask rate, the corpus size in sentences or tokens, and the training wall-clock — none of which this paper states. One flagged inconsistency: §2.1 puts the bottleneck at 3d² parameters "with d the dimensionality of each head (d = 64)", i.e. 12,288 parameters, which cannot produce the 1.6% overhead of Table 1; 3·768² would. The card records the paper's wording rather than resolving it.

source: models/autobot.md
08

Bag-of-Vectors AE2022

Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation
Florian Mai, James Henderson — Idiap Research Institute / EPFL · arXiv:2110.07002

A small from-scratch Transformer autoencoder whose latent is not one vector but a variable-size bag of them — one per input token, then pruned by an L0Drop gate to a target open-gate ratio r. Built so that Emb2Emb-style latent-space editing survives texts too long for a single-vector bottleneck.

in the review ↗
Input model
  • Initialisationrandom; "we instead pretrain all autoencoders from scratch directly on the data of the target task"§5 "Autoencoder Pretraining"
  • No pretrained encoder or decoder in the main experiments;BART is rejected off-the-shelf because it "wasn't trained to have a smooth embedding space, for example using L0Drop"§5 "Autoencoder Pretraining"
  • Teachernone; no distillation anywhere in the paper§3, §5
  • Side studyan L0Drop layer inserted into BART and fine-tuned on Yelp-Sentences at target rate 0.4, Adam lr 0.00005 — Appendix C.2.2; BART variant and size not stated in paper
  • Downstream stageencoder and decoder frozen, only the mapping Φ trained§2, Figure 2, Appendix B.3
Model size
  • Depth3 Transformer encoder layers and 3 decoder layersAppendix B.2
  • Heads2 in the autoencoder; 4 in the downstream mapping ΦAppendix B.2, Appendix B.3
  • Widthd_model is set equal to the latent dimension — 512 Yelp-Reviews, 32 Yelp-Sentences, 128 GigawordAppendix B.2
  • Parameter countBoV-AE 15.1M / 0.960M / 2.725M against fixed-size 14.578M / 0.958M / 2.758M on the three datasetsTable 4
  • FFN inner widthnot stated in paper
  • Vocabulary / tokenizera fresh BPE tokenizer per dataset, HuggingFace tokenizers, capped at the 30k most frequent tokensAppendix B.1
  • Mapping Φone-layer Transformer decoder at the latent dimension; parameter count not stated in paperAppendix B.3
Vector
  • Not a single vectorthe latent is a bag, Z = (R^d)+, X = {z_1 … z_n} := enc(x), "a (multi)-set of vectors that (i) can grow arbitrarily large, and (ii) where the elements are not ordered"§3
  • Per-vector dimensionality512 (Yelp-Reviews), 32 (Yelp-Sentences), 128 (Gigaword)§5.1, §5.2, Appendix A.1
  • How many vectorsthe encoder emits one per input token, which the paper calls "likely too many, as it makes copying from the input to the output trivial"§3.1
  • PruningL0Drop (Zhang et al. 2021) computes a scalar gate g_i ∈ [0,1] per output, multiplies it in, and deletes vectors whose gate falls below ε — §3.1; ε not stated in paper
  • Sparsity knobtarget open-gate ratio r, models named L0-r; ladders at r = 0.05/0.1/0.2/0.4 (Yelp-Reviews) and 0.2/0.4/0.6/0.8 (Yelp-Sentences, Gigaword)§5.2, Appendix A.1, Appendix C.1
  • Realised bag sizenot stated in paper; only the target ratio is reported
  • How the decoder reads itordinary cross-attention over the bag, order irrelevant — §3; O(n·|s|) dot-productsAppendix C.2.1
  • Single-vector contrast ("fixed")identical model with the encoder's last-layer vectors averaged into one; this is the Emb2Emb setting of Mai et al. (2020), where Φ can be an MLP and the loss a vector distance§2, §5 "Autoencoder Pretraining"
  • Normalisation, Gaussian noise, prior / KLnone; regularisation is bag-size sparsity plus input token dropout§3.1, Appendix B.2
Loss / training
  • ObjectiveLrec + LL0, a plain sum; no annealing and no alternating steps stated§2, §3.1
  • Lrecreconstruction NLL, L_rec = NLL(A(x), x) for A = dec ∘ enc§2
  • Corruptiondenoising objective, each token dropped from the input with probability 10%Appendix B.2
  • LL0λ_L0 · max(r, (1/n) Σ_i g_i) — a free-bits-style hinge that hits an explicit target ratio instead of the plain λ Σ_i g_i, adopted because the latter's λ "is very sensitive with respect to other hyperparameters"§3.1, Eq. 1
  • λ_L0 = 10 for every BoV model; only r is varied§3.1, Appendix B.2
  • Optimizer / LR / batchAdam, lr 0.00005 (Yelp-Reviews, Gigaword) or 0.0001 (Yelp-Sentences), batch 64, dropout 0.1Appendix B.2
  • Steps2M on Yelp-Reviews and Gigaword, 1.5M on Yelp-Sentences; validated every 20k steps, best checkpoint keptAppendix B.2
  • Hardware / computesingle GPUs from a shared grid, "vary between GeForce GTX Titan X and RTX 3090"; 7,530 GPU hours reported (6,640 pretraining, of which 5,760 Yelp-Reviews), ≈25,000 over development — Appendix B.1; "more than a month on a single 24GB GPU" for the largest§5 footnote 2
  • Downstream lossL(ẑ_y) = L_sim(z_x, ẑ_y) + λ_sty·L_sty(ẑ_y), with the adversarial term of Mai et al. (2020) dropped§2, §2 footnote 1
  • Lsima novel fully differentiable Hausdorff distance — the average Hausdorff min replaced by a softmin-weighted sum over Euclidean distances§4.3, Eqs. 6–9
  • Bag-size handlingloss summed over prefix bags with a window w_l = 1 for n−k ≤ l ≤ n+k, default k = 0; at inference the bag size minimising the loss is selected from up to N candidates, N = 250 or 30§4.2, Eq. 5, Appendix B.3
  • Mapping architecturesa plain Transformer decoder, or Transformer++, a pointer-generator variant ẑ_t = (1−p_gen)(z_copy + z_offset) + p_gen z'_t that chooses between copying an input vector plus an offset and generating a new one§4.1, Eqs. 2–4
Data
  • Yelp-Sentencesthe sentiment-transfer data of Shen et al. (2017) used as-is; 9.7 / 8.5 average words, 177k inputs, 267k outputsTable 3, Appendix B.5.1
  • Yelp-Reviews (new here)Yelp Open Dataset, May 2021, restaurants only, 1–2 stars negative and 5 stars positive, capped at 100 words; 400k positive plus 400k negative for training and 50k each for validation and test — Appendix B.4.1; Table 3 lists 500k / 500k at 56.1 / 48.7 words, and §5.1 gives the average as 52 words
  • GigawordGraff et al. (2003) preprocessed after Rush et al. (2015) but lower-cased; 27.2 / 8.2 average words, a 500k subset of the >8.5M training samples — Table 3, Appendix A.1, Appendix B.6.1; Appendix B.6.1 says "1 million examples" instead
  • LanguagesEnglish onlyAppendix B.4.1, Appendix B.6.1
  • Preprocessingper-dataset BPE at 30k tokens; Yelp-Reviews uses review text only, with metadata such as reviewer names excludedAppendix B.1, Ethics Statement
  • Max sentence length100 words for Yelp-Reviews; not stated in paper for the other twoAppendix B.4.1
Notes

Reconstruction: at d=512 on Yelp-Reviews the single-vector model plateaus at validation NLL ≈3.9 while L0-0.1 reaches ≈2.1 (§5.1, Figure 8); at d=32 on Yelp-Sentences fixed ≈3.0 against ≈1.0 for L0-0.6 (§5.2.1); at d=128 on Gigaword fixed 3.01 against 0.9 for most BoV rates (Appendix A.2). Style transfer on Yelp-Sentences is better for BoV up to r=0.6 and collapses at r=0.8, read as the bag learning to copy rather than compress (§5.2.2). The two component ablations: differentiable Hausdorff scores 24.2 against 14.6 for the step-wise average Hausdorff (§5.2.3), and the backprop window k=5 scores 35.8 against 28.2 at k=0 and 21.2 at k=20 (§5.2.4). Gigaword summarization ROUGE-L (Table 2): fixed 13.1, best BoV L0-0.2 19.8, with Transformer++ mainly helping at r>0.2 by shortening outputs. An appendix study inserts L0Drop into BART and fine-tunes, reaching 35 self-BLEU at 61% transfer accuracy — comparable to the from-scratch L0-0.4 (38 / 55%) but more fluent (Appendix C.2.2).

Release: code shipped as supplementary material, promised publicly under MIT on publication; no repository URL in the paper, no checkpoints mentioned, and the Yelp-Reviews dataset can only be redistributed as a reconstruction script (Appendix B.1, Ethics Statement).

This card was read from arXiv:2110.07002v2 over all 22 pages, including Appendices A–C. Deliberately left off because the paper does not state them: FFN inner width, the L0Drop cutoff ε, the realised average bag size in vectors (only the target ratio r is given), Φ's parameter count, BART's size in the appendix study, and the number of seeds — no variance or error bars are reported anywhere. Four internal inconsistencies are recorded in the fact sheet rather than silently reconciled here.

source: models/bov-ae.md
09

SONAR2023

SONAR: Sentence-Level Multimodal and Language-Agnostic Representations
Paul-Ambroise Duquenne, Holger Schwenk, Benoît Sagot — Meta AI · arXiv:2308.11466

A 200-language NLLB translation model with its token-level cross-attention replaced by a single mean-pooled vector, trained on translation plus an embedding-space MSE tie plus a lightly weighted denoising term. The object of study for the whole TAE program.

in the review ↗
Input model
  • Initialisationwhole encoder–decoder initialised from the pretrained NLLB 1B dense MT model, not from scratch§3.1, §5.1
  • Both halvesencoder from the NLLB 1B encoder, decoder from the NLLB 1B decoderFigure 1
  • NLLB's own data, as the paper describes ithuman-labelled bitext, back-translated and mined data, trained with full token-level cross-attention§5.1
  • Teachernone for the text model; the teacher–student step exists only for the speech encoders, where the SONAR text encoder is the teacher§3.2, §6
  • Optional second stagedecoder-only fine-tuning with the encoder frozen ("random interpolation decoding")§3.1, §5.5
Model size
  • Depth24-layer Transformer encoder and 24-layer Transformer decoder, inherited from NLLB 1B dense§5.1
  • Parameter countnot stated in paper; only the name "NLLB 1B dense model" is given (the NLLB paper's 1B-class dense model is 1.3B)§5.1
  • d_model, FFN width, attention headsnot stated in paper
  • Vocabulary / tokenizernot stated in paper; the released tokenizer asset is a 256k SentencePiece model, consistent with NLLBREADME
Vector
  • Dimensionalitynot stated in paper; the released encoder emits 1024-d vectorsREADME
  • Poolingmean-pooling over the token-level outputs of the encoder§3.1, §5.1
  • Alternatives triedmax-pooling (worse, value range differs from NLLB training), EOS-token pooling (unstable), BOS-token output listed as an option§3.1, §5.1
  • How the decoder reads itthe decoder cross-attends to this single vector at every decoding step; no token-level cross-attention§3.1
  • Normalisation / scaling of znot stated in paper
Loss / training
  • ObjectiveL = LMT + α·LMSE + β·LAE/DAE, a weighted sum, not alternating batches§5.1, Figure 1
  • LMTtranslation cross-entropy, decoding the target sentence from the source sentence's embedding§3.1
  • LMSEmean-squared error between the embedding of a sentence and that of its translation; no frozen teacher, collapse is prevented by the other terms§3.1, §5.4
  • LDAEdenoising auto-encoding after mBART (Liu et al. 2020), applied to the target sentences of the same bitext; the noise recipe is not stated§3.1, §5.1
  • Final weightsα = 0.1, β = 0.01 (DAE); ablations at β = 1.0 (AE), β = 0.1 (DAE), and MSE aloneTable 1, §5.5
  • Updates100k for the main runs; the AE-only ablation 50k; decoder fine-tuning a further 50k§5.1, §5.3, §5.5
  • LR / batch"same learning rate and batch size as NLLB training"; no numbers§5.1
  • Optimizer, hardware, wall-clocknot stated in paper
  • Decoder fine-tuning recipefreeze the encoder; for a bitext pair (x, y) draw z as a random interpolation of the two embeddings and train the decoder to emit y§3.1, §5.5
  • Target-language controlnot described in the paper (the released API takes a target-language token, NLLB style)§5.1
Data
  • Corpusall bitext used for NLLB training: human-labelled, back-translated and mined§5.1
  • Languages200 target languages, versus LASER's English and Spanish only§5.1
  • Sizenumber of sentence pairs and tokens not stated in paper
  • Monolingual datanone; "left to future work"§5.1
  • Filteringnot stated in paper
Notes

Headline results on FLORES-200 devtest averaged over 200 languages (Table 1): xsim error 1.4 (LaBSE 10.7, LASER3 5.1), xsim++ 15.2 (LaBSE 36.1, LASER3 36.4), translation spBLEU 32.9 X→eng and 20.7 eng→X against the NLLB 1B topline of 35.2 / 24.9, auto-encoding spBLEU 32.4 (41.7 with the fine-tuned decoder). The MT+AE ablation with β = 1.0 reaches 94.6 auto-encoding spBLEU, which is why the DAE weight is set so low.

Speech side: student encoders initialised from w2v-bert 2.0 (600M) are trained by MSE regression onto the frozen text embedding of the transcript, with a 3-layer attention-pooling head (§3.2, §6.1). 37 languages, 43,628 hours of ASR data (Table 10).

Release: code MIT, weights mixed MIT and non-commercial; checkpoints text_sonar_basic_encoder, text_sonar_basic_decoder, finetuned_decoder (README).

Everything on this card was read from arXiv:2308.11466v2 by an extraction pass over the full 14-page PDF; items marked README come from github.com/facebookresearch/SONAR. Values the paper does not state but that follow from the NLLB 1B config (d_model 1024, FFN 8192, 16 heads, 256k vocab, ~1.3B params) are deliberately left off the sourced rows.

source: models/sonar.md
10

Large Concept Model2024

"Large Concept Models: Language Modeling in a Sentence Representation Space"
LCM team — Loïc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov et al. — FAIR at Meta · arXiv:2412.08821

Not an autoencoder. A next-sentence-embedding predictor trained inside the frozen SONAR space, after a robust median/IQR rescaling of the 1024-d vectors. MSE regression collapses to the mean; two diffusion variants and a residual-quantized variant do not. Scaled to 7B.

in the review ↗
Input model
  • Encoder and decoder are SONAR, frozen"the concept encoder and decoder are frozen"; "The encoder and decoder are fixed and are not trained"§1, Figure 1
  • Why SONARbest reported xsim / xsim++ scores and prior use in large-scale bitext mining§2.1
  • SONAR as restated hereencoder–decoder with a fixed-size bottleneck instead of cross-attention, trained on 200-language MT plus denoising auto-encoding plus an MSE loss at the bottleneck§2.1, Figure 2
  • Initialisation of the LCM itselfnot stated in paper
  • The one init detail givenTwo-Tower residual blocks start as the identity, by zero-initialising W and b of the AdaLN modulator§2.3.4, Eq. 21
  • Distillation teachernone; the teacher–student step mentioned in §2.1 belongs to SONAR's speech extension§2.1
  • Decoders fine-tuned in this paperone on quantized representations, 1.2M English sentences; one for noise robustness, per Eq. 27§2.3.5, §2.5.2
Model size
  • Ablation scaleall four variants "configured to have approximately 1.6B trainable parameters"§2.4.1
  • Base-LCM32 layers, d_model 2048, 16 heads, RoPE, pre-norm RMSNorm, SwiGLU, dropout 0.1; FFN width not stated§2.4.1
  • One-Tower32 blocks, 32 heads, FFN inner 8192, d_model 2048, learned position embeddings§2.4.1
  • Two-Tower (1.6B)5 contextualizer layers, 13 denoiser layers, 16 heads, shared d_model 2048; RoPE in the contextualizer, no positional embeddings in the denoiser§2.3.4, §2.4.1
  • Two-Tower (7B)5 contextualizer layers, 14 denoiser layers, d_model 4096, 32 heads; exact parameter count not stated§3
  • Denoiser conditioningAdaLN regressing scale γ, shift β and residual gates α from a 256-d frequency timestep embedding through a 2-layer SiLU MLP§2.3.4, Eq. 21–22
  • Quant-LCMsame architecture as One-Tower, except Quant-LCM-d's output width of 8192 for the softmax§2.4.1
  • Vocabulary / tokenizernone — the model reads and writes continuous vectors; Quant-LCM-d's target space is 8192 units per codebook§2.3.5
  • Contextdocuments wrapped at 128 sentences in the ablations, extended to 2048 concepts at 7B§2.4.1, §3
Vector
  • Dimensionality: 1024, stated once, in passing: SONAR embeddings are "fixed size tensors of 1024 floats"Appendix A
  • Formationby the frozen SONAR encoder; SONAR's pooling is not described in this paper§2.1
  • Normalisationa robust scaler fitted to randomly sampled SONAR vectors from several corpora and domains, which "removes the median statistics and scales the data according to the interquartile range (IQR)"§2.3.1
  • Formulanormalize(x) = (x − µ)/σ, denormalize(x) = µ + σx, with µ the median and σ the IQR§2.3.1, Eq. 4
  • Scaler constants, fitting-sample size, per-coordinate or scalarnot stated in paper
  • Into the modelPreNet(x) = normalize(x)·WTpre + bpre, mapping R1024 → Rd_model§2.3.1, Eq. 1
  • Out of the modelPostNet(x) = denormalize(x·WTpost + bpost), mapping back to R1024§2.3.1, Eq. 2
  • How the context entersBase-LCM, a causal sequence; One-Tower, noisy and clean embeddings interleaved with a mask attending only to clean ones; Two-Tower, denoiser cross-attention over a causally encoded contextualizer output§2.3.1, §2.3.3, §2.3.4
  • Noise / priorvariance-preserving diffusion, xt = αtx0 + σtε with ε ∼ N(0, I); reverse process starts at N(0, I)§2.3.2, Eq. 7–9
  • Quantizationresidual vector quantization of the SONAR space, 64 codebooks × 8192 units, fitted on 15M English sentences; all 64 codebooks recover ≈70% of continuous auto-encoding BLEU§2.3.5, Figure 9
  • Storageembeddings precomputed and held at FP16 in parquet; 1 TB of text becomes 15–20 TB of vectorsAppendix A
Loss / training
  • Base-LCML = Ex∼q Σn ‖f(x<n; θ) − xn‖², plain MSE regression of the next concept§2.3.1, Eq. 5–6
  • Stoppingdocuments suffixed with SONAR-encoded "End of text."; halt when cosine to that vector, or to the previous prediction, exceeds 0.9§2.3.1
  • Diffusion LCMsx0-prediction with L(θ) = Et[ω(t)·Ex0,ε‖x0 − µθ(αtx0 + σtε, t)‖²], defaulting to ω(t) = 1§2.3.2, Eq. 16
  • Weightings tried and rejectedclamped-SNR ω(t) = max(min(eλt, λmax), λmin) at (0, 10) and (0.001, 5); fragility weighting ω(x0) = sigmoid(a·F(x) + b), a = −4, b = 3.5, F a 3-layer MLP fitted on 50M sentences§2.3.2 Eq. 17–19, §2.4.4, Table 6
  • Noise schedulecosine by default, αt² = f(t)/f(0), f(t) = cos²(((t+s)/(1+s))·π/2), s = 0.008, T = 100, rescaled to zero terminal SNR§2.3.2, Eq. 12
  • Schedules comparedquadratic at (β₀, β_T) = (0.001, 0.0012) and (0.02, 0.022), and a sigmoid schedule introduced here, f(t) = sigmoid(δ − γ·logit(t)), at (1.5, −1), (1.5, −2), (0.8, −1), (3.5, 0)§2.3.2 Eq. 13–14, §2.4.3
  • Classifier-free guidance∇xlogγp(x|y) = (1−γ)∇xlog p(x) + γ∇xlog p(x|y); trained unconditionally by dropping self-attention (One-Tower) or cross-attention-mask rows (Two-Tower), both at rate 0.15§2.3.2 Eq. 20, §2.3.3, §2.3.4
  • Inference defaultsS = 40 of T = 100 steps by the trailing schedule, gscale = 3, grescale = 0.7, σinit = 0.6, epsilon-scaling λeps = 1.00045§2.3.2, §2.4.1
  • Quant-LCMQuant-LCM-d, cross-entropy on the next codebook's unit with the codebook index fed in as input; Quant-LCM-c, MSE on the continuous target given the intermediate quantized vector§2.3.5, Eq. 23
  • Ablation runs250k steps on 32 A100s on Meta's RSC, total batch 229k concepts; optimizer and learning rate not stated§2.4.1
  • 7B pre-training124k steps on 256 A100s on Meta's RSC, batch 1M concepts, AdamW (0.9, 0.95), ε = 1e-5, weight decay 0.1, cosine schedule with 10k warm-up to LR 3e-4, gradient clip 10§3
  • 7B instruction finetuningcosine schedule from LR 3e-5, 7 epochs, batch 262K sentences, loss on answer sentences only, each suffixed "End of response."§3
  • Wall-clock training timenot stated in paper
Data
  • AblationsFineweb-edu, chosen for reproducibility; "training data in the order of 1.3T tokens" for the 1.6B models§2.4.1, Abstract
  • 7B pre-training2.3B documents, 2.7T tokens, 142.4B concepts; the corpus is not named§3
  • Whole programme~4B documents, 310B sentences, averaging 27 tokens and 88 characters per sentence, ~889 TB of raw textAppendix A
  • LanguageEnglish only throughout training; no multilingual or aligned data§4.2, §8
  • SegmentationSaT Capped at a 200-character maximum, selected over SpaCy Capped on Auto-BLEU over 10k documents / ~500k sentences§2.2, Figure 3
  • Instruction tuningCosmopedia stories at ablation scale; "publicly available instruction tuning datasets following Chung et al. (2024)" at 7B, 389M sentences of which 53M are answers§2.4.1, §3
  • Filteringfragility is proposed as a filter; the 5% most fragile embeddings are hyperlinks, references, unique ids, code-switched or numeric text§2.5.2
Notes

The paper's central negative result is that plain MSE regression in SONAR space fails: Base-LCM reaches the lowest ℓ2 but not the lowest round-trip ℓ2-r, which the authors read as averaging plausible continuations into a point that is not a valid sentence (Table 3), and it scores 23.69 ROUGE-L / 0.482 coherence after Cosmopedia tuning against 33.40 / 0.968 for One-Tower and 33.64 / 0.938 for Two-Tower; a 1.4B token-level Llama trained on the same data ("smaLlama") scores 34.88 / 0.984 (Table 4). Quant-LCM-c 30.87 and Quant-LCM-d 28.01 fall between. Two-Tower was scaled to 7B for its smaller memory footprint. Two-Tower-7B-IT reaches 36.47 ROUGE-L on CNN DailyMail — above Llama-3.1-8B-IT's 34.97 and Gemma-7B-IT's 31.14, below a dataset-tuned T5-3B's 37.56 — and 23.71 on XSum, the best of every model compared (Table 10); it is more abstractive (lowest OVL-3) and less repetitive, but scores lowest on the CoLA fluency classifier, as does the human ground truth. On long-context LCFO it trails Llama-3.1-8B-IT substantially, and on the reverse task of summary expansion the LLMs lead on ROUGE-L (Tables 11–12). Trained on English alone, it scores 23.5 against Llama-3.1-8B-IT's 20.7 on English XLSum and 20.2 against 19.7 averaged over the six languages both models officially cover, with Vietnamese at 30.4 (§4.2).

Two side results bear on the space itself: a SONAR decoder fine-tuned on noised embeddings lifts auto-encoding BLEU from 79.5 to 88.0 on Flores and from 70.5 to 85.6 on Gutenberg (Table 7), and the "fragility" measure of §2.5.2 shows semantic loss under small perturbation rising sharply with sentence length, which is the stated reason for the 200-character cap.

Release: training and evaluation code MIT at github.com/facebookresearch/large_concept_model, with recipes for the 1.6B MSE and Two-Tower diffusion LCMs and Quant-LCM listed as pending; no trained LCM checkpoints are released (README).

Everything on this card was read from arXiv:2412.08821v2 over the full 49-page PDF including Appendices A–D. The LCM's own initialisation, the FFN widths of Base-LCM and Two-Tower, the exact counts behind "approximately 1.6B" and "7B", the ablation runs' optimizer and learning rate, the identity of the 7B corpus, and above all the numeric median and IQR constants of the robust scaler are deliberately left off: the paper states none of them.

source: models/lcm.md
11

Cramming 1568 tokens2025

"Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity"
Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, Mikhail Burtsev — AIRI / MIPT / LIMS · arXiv:2502.13063

No encoder at all. A frozen pretrained decoder LLM is handed trainable input vectors that are optimised by gradient descent separately for each text, and the question asked is how many tokens one such vector can hold. Llama-3.1-8B holds 1568.

in the review ↗
Input model
  • Frozen pretrained LLM as decoder; only the [mem] input embeddings are trained§3
  • No encoder: the encoder is replaced by a per-sample optimisation procedure§1, §3
  • [mem] vectors randomly initialised before optimisationAppendix A
  • Transformer LLMs: Pythia-160M/410M/1.4B/2.8B, OPT-1.3B, OLMo-1B-0724, Sheared-LLaMA-1.3B, Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8BTable 2
  • State-space LLMs: Mamba-130M/370M/790M/1.4B, same procedure§4.6, Appendix C, Table 2
  • Main configuration: one [mem] vector, Llama-3.1-8B, PG-19§4, §4.1
  • No LLM weights are updated and no distillation teacher is used§3
Model size
  • Parameter counts: 0.13B–8.0B across the 14 checkpointsTable 2
  • Input hidden size: 768 (Pythia-160M, Mamba-130M) to 4096 (Llama-3.1-8B)Table 2
  • Vocabulary: 32,000 (Sheared-LLaMA) / 50,272 (OPT) / 50,280 (Mamba) / 50,304 (Pythia, OLMo) / 128,256 (Llama-3)Table 2
  • Layers, FFN width, attention headsnot stated in paper; only HF checkpoint names are given
  • Tokenizernot stated in paper beyond vocabulary size; models loaded from HuggingFace TransformersTable 2, Appendix A
  • Trainable parameters of the method: K × d_model floats, no count given§3
  • Pythia-160M's 2048-token context is the binding limit in the multi-vector sweep§4.4
Vector
  • Dimensionality equals the host LLM's input embedding size§4, Table 2
  • Headline case: a single 4096-d bfloat16 vector in Llama-3.1-8BAppendix F
  • Number of vectors: one in most experiments§4
  • Sweeps: K = 1…16 for Llama-3.2-1B, K = 1…32 for Pythia-160M§4.4
  • Not produced by any network; found by gradient descent on that one text§3
  • Enters the decoder as a prefix: the LM sees [m_1,…,m_K, t_1,…,t_i] and predicts t_{i+1}§3
  • Generation at inference starts from the learned [mem] tokens§3
  • No normalisation, Gaussian noise, prior or KL term— none appears in the paper (none described; not stated in paper)
  • Theoretical bound L ≤ (d_model × b) / log₂|V|; a 2048-d 16-bit vector is 32,768 bits ≈ 1,931 tokens at |V|=128,256Eq. (1), §1, §3
  • Solutions are non-unique: intra-sample cosine similarity between [mem] vectors for the same text rarely exceeds 0.8 and overlaps the inter-sample distributionAppendix E, Fig. 7
  • Interpolating between two solutions for the same text always hits reconstruction errors; no basinAppendix E, Fig. 8
Loss / training
  • Objective: Lrec alone — standard next-token cross-entropy on the target text, LM frozen§3
  • No second term: no KL, MSE tie, denoising, corruption or word dropout— none appears in the paper (none described; not stated in paper)
  • Optimizer: AdamW, LR 0.01, β₁ = β₂ = 0.9, weight decay 0.01Appendix A
  • Up to 5,000 steps per text, early stopping when token-level accuracy reaches 1.0Appendix A
  • Batch size and LR schedulenot stated in paper; one text per optimisation run
  • Hardware: one A100 80GB per run, up to 4 GPUs for parallel experimentsAppendix A
  • Wall clock: seconds for small models and short texts, 10–20 minutes at the large endAppendix A
  • PyTorch + HuggingFace TransformersAppendix A
  • Measurement metrics, not losses: Decoding Capacity Lmax at accuracy threshold 0.99, Token Gain Ctokens, Information Gain CH = HLM − HLM+[mem] in bitsEq. (2)–(4), §3, §4.1
Data
  • PG-19 books (Rae et al. 2020); noted as part of the Pile and so likely in pretraining§4 Data
  • Fanfics: 21 AO3 works over 20,000 words each, published after October 2024, HTML stripped§4 Data, Appendix B
  • Random text: words sampled from the top 100,000 GloVe vocabulary entries§4 Data
  • GovReport 64-token sequences, used only for the embedding-structure analysisAppendix E
  • 50 texts per target length per source§4.1, Appendix B
  • Length grid [64, 80, 96, 128, 160, 192, 256, 384, 512, 768, 1024, 1280, 1568, 2048, 2560, 3072]; the cross-entropy study spans 8 to 1568 tokens, to 3072 for larger models§4.1, §4.2, §4.3
  • Texts cut to exact token counts and forced to start at a sentence boundaryAppendix B
  • Language and total corpus sizenot stated in paper
Notes

Decoding Capacity from a single vector on PG-19 (accuracy threshold 0.99): Pythia-160M 80, Pythia-410M 96, Pythia-1.4B 160, Pythia-2.8B 128, OPT-1.3B 128, OLMo-1B 384, Sheared-LLaMA-1.3B 512, Llama-3.2-1B 512, Llama-3.2-3B 1024, Llama-3.1-8B 1568 tokens (Fig. 1, Table 3). Fanfics give the same numbers, so the effect is not memorisation of pretraining data; random word sequences give 65/72/139/316/460/792 for the six main models. Information Gain, the cross-entropy reduction in bits, is roughly constant per model across all three sources — 396 bits for Pythia-160M up to 4866±547 bits for Llama-3.1-8B on PG-19, 4541±759 on random text — which is the paper's central claim: capacity is set by the entropy to be removed, not by text length or domain. Texts whose unconditioned cross-entropy falls below that per-model figure are reconstructed exactly; above it, compression shaves off a constant offset (§4.3, Fig. 3). Capacity scales close to linearly in the number of vectors: Pythia-160M reaches 2016 tokens with 32 vectors (its context limit), Llama-3.2-1B 7,168 tokens with 16, though Llama-3.2-1B falls below the ideal linear trend (§4.4). Mamba-1.4B reaches 512 tokens from one vector, matching the Transformer 1B models, so the effect is architecture-agnostic (§4.6). Measured against the Eq. (1) bound, utilisation peaks near 0.3 and is higher for newer models; within Pythia it falls with size, within Llama and Mamba it rises (§4.5, no per-model numbers given). Appendix F supplies the honest counter-accounting: as bytes on disk the single vector is about 0.8x the raw text, worse than zlib at 2.28x or arithmetic coding with pythia-160m at 6.77x; the x1568 figure is a reduction in the number of embeddings, not in bits.

Release: code at github.com/yurakuratov/hidden_capacity (train.py, model.py wrapper built on the RMT implementation, per-model run scripts, plotting notebooks, preprocessed PG-19 and fanfics chunks); no LICENSE file, licence not stated; there are no weights to release since the [mem] vectors are per-text artefacts (README).

This card was read from arXiv:2502.13063v3 over all 17 pages. Layer counts, head counts, FFN widths and tokenizer identities for the host checkpoints are deliberately left off because the paper gives only names, parameter counts, input hidden sizes and vocabulary sizes; per-model capacity-utilisation values are left off because Fig. 5 is a scatter plot with no table behind it; and no reconstruction BLEU or downstream evaluation appears because the paper reports none.

source: models/cramming-1568.md
12

Omnilingual SONAR2026

"Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech"
Omnilingual SONAR Team — João Maria Janeiro, Pere-Lluís Huguet Cabot, Ioannis Tsiamas, Yen Meng, Vivek Iyer et al. — Meta · arXiv:2603.16606

SONAR rebuilt on a Llama-3.2-1B encoder-decoder. The 2023 MSE tie and denoising term are dropped for a split-softmax contrastive loss with synthetic hard negatives; token-level cross-attention exists only as a pretraining stage and is then replaced by the same single-vector bottleneck. 1024-d, CLS-pooled, 4,200+ language varieties.

in the review ↗
Input model
  • Initialisationencoder and decoder both initialised from the pretrained Llama-3.2-1B, not from scratch§4.2, §5
  • Encoder adaptationLlama's causal attention replaced by bidirectional attention; a [CLS] token prepended as the pooling token§4.2
  • Decoder adaptationcross-attention blocks added to the Llama stack, their weights initialised from Llama's pretrained self-attention weights; grouped-query attention matching Llama's self-attention config§4.2
  • New vocabulary tokensembedding initialised as the mean of the token's Llama-3 sub-token embeddings§4.2
  • Stage 2 initialises from the Stage-1 seq2seq model, not from Llama— §4.4.1; ablation gives 0.65 xsim from seq2seq vs 1.02 from Llama vs 13.35 from randomTable 13b
  • Teachernone for the foundational OmniSONAR-200. A frozen SONAR 2023 model is used, but only to produce guide embeddings for false-negative filtering§4.4.1
  • Later teachersfrozen OmniSONAR-200 for the omnilingual student, frozen OmniSONAR for the small encoders and the speech encoder§4.5, §4.6, §4.8
  • Decoder finetuning stageencoder frozen at the omnilingual weights, decoder resumed from OmniSONAR-200 on the Stage-1 translation loss§4.7
Model size
  • Encoder1.5B parameters, 16 layers, d_model 2048, FFN 8192, SwiGLU, 32 attention heads, 8 KV heads, head_dim 64§5, Table 36
  • Decodersame architecture, 1.8B parameters total; the encoder/decoder parameter split and whether embeddings are tied are not stated in paper§5
  • Vocabulary256K (exactly 256,232) for both the 200-language and the omnilingual tokenizer§4.1, §5, Table 36
  • TokenizerBPE. The 200-language one extends Llama-3's 128k vocabulary to 256k by continued merge training; the omnilingual one is trained from scratch§4.1
  • Fertility44 tokens per FLORES sentence across 200 languages, against 79 for the stock Llama-3 tokenizer§4.1
  • Smaller distilled variantsMedium 884M, Small 511M, Tiny 233M, xTiny 39M, by structured pruning plus MSE distillation; all keep the 1024-d output§4.8, Table 10, Table 36
  • Max sentence lengthnot stated in paper; only per-GPU token budgets and length bucketing are given§5
Vector
  • Dimensionality1024§5, Table 36
  • PoolingCLS pooling on the prepended [CLS] token, with a linear down-projection from 2048 to 1024 — §5. §4.2 states the projection after pooling and §5 before it; the 1024-d endpoint is the same either way§4.2, §5
  • Alternative triedmean-pooling, worse — 0.68 / 9.25 xsim / xsim++ against CLS at 0.64 / 8.77Table 37, §C.1
  • How the decoder reads itfrom Stage 2 onward the decoder cross-attends exclusively to the single pooled vector; the single-vector bottleneck of SONAR 2023 is retained§4.4, §4.4.1, §6.3
  • Token-level cross-attention exists, but only in Stage 1during seq2seq pretraining the decoder attends to the full non-pooled encoder sequence, and that path is dropped thereafter§4.3, §4.2
  • Normalisation, scaling, noise or prior on znot stated in paper; cosine similarity inside the losses is scaled by a learned logit scale τ
  • Collapseembedding feature std 0.0356 against SONAR's 0.0074, mE5large 0.0312, LaBSE 0.0358Table 38, §C.2
  • Intrinsic dimensionalitysingular values decay stably out to roughly 800 dimensionsFigure 16, §C.3
  • Encoder input is promptedsources prefixed [language name]:, replaced by Unspecified language: with probability punk from the omnilingual stage on§4.3, §4.5.1, Table 3
Loss / training
  • Stage 1 — seq2seq pretrainingLtranslation = −Σt log P(yt | y&lt;t, x), token-level cross-entropy over >5,000 directions, decoder attending to the un-pooled encoder output§4.3
  • Stage 2 — translation plus contrastiveL = α·Lcontrastive + β·Ltranslation, a weighted sum, α = 0.05, β = 1Eq. 4, §5
  • Lcontrastivemargin-additive InfoNCE over in-batch negatives, −(1/N)Σ log[eφ(x,y)−m / (eφ(x,y)−m + ΣS eφ(x,yn))] with φ = τ·cosEq. 2
  • False-negative filteringGISTEmbed-style, dropping any in-batch negative whose guide-embedding similarity to the source exceeds the positive pair's, guides from a frozen SONAR; r = 0.5Eq. 3, §4.4.1, §5
  • Stage 3 — split-softmax with hard negativesLcontrastive_hn = (1−γ)·Lcontrastive − γ·(1/N)Σ log[eφ(x,y) / (eφ(x,y) + ΣHN eφ(x,h))], the two denominators kept separate because a non-zero margin does not converge on hard negativesEq. 5, §4.4.2, §7.2
  • Overall Stage-3 lossL = α·Lcontrastive_hn + β·Ltranslation; γ = 0.8, m = 0.3 in-batch and m = 0 on hard negatives, τ = 100, 5 hard negatives per sourceEq. 6, §5
  • Hard negativessynthetic, generated with LLaMA3.3 70B Instruct as near-paraphrases carrying a subtle semantic change§3.6, §A.4
  • The 2023 MSE tie does not survivethe source–target MSE term is replaced by the contrastive loss; ablated at 0.92 / 12.54 against 0.65 / 8.95 for contrastive§7.1, Table 11b
  • The 2023 denoising term does not surviveno DAE, no noising recipe, no word dropout anywhere in the text pipeline§4.3, §4.4
  • MSE returns only as a distillation termtokenizer warm-up, omnilingual extension and speech, always student-against-frozen-teacher, never source-against-targetEq. 7, Eq. 10, Eq. 12
  • Omnilingual extension lossλs→t·Lstudent→teacher + λt→s·Lteacher→student + λMSE·LMSE, weights set per example by whether the source language is foundational or newEq. 11, Table 3
  • Teacher target for the extensionthe midpoint ½(xteacher + yteacher) for foundational sources, the target embedding alone for new languagesEq. 8
  • OptimizerAdamW, betas 0.9 and 0.98, max gradient norm 1, FSDP1 with fp16 mixed precision§5
  • ScheduleStage 1 100k steps at LR 4e-4, 2k warmup, inverse square root, 8192 tokens per GPU on 16 nodes of 8 GPUs; Stage 2 10k steps at 3e-4 with 6k tokens per GPU; Stage 3 15k steps at 1e-5 with 1.2k tokens per GPU§5
  • Omnilingual extension30k steps at LR 4e-5, cosine after 1k warmup, dropout 0.05, 48 A100s, 768k-token batch, roughly 24 hours§5
  • GPU model and wall-clock for Stages 1–3not stated in paper
  • Negatives gathered across all 128 GPUs, worth 0.65 against 0.74 xsim§7.2, Table 12a
  • Full ladder, FLORES200 dev xsim / xsim++Llama init 94.57 / 99.89 → seq2seq 7.74 / 51.55 → contrastive 0.71 / 16.23 → plus decoder 0.65 / 8.95 → plus hard negatives 0.76 / 7.06Table 11a
Data
  • Stage 1 corpus200 ↔ 200 directions, 5.7B primary translation pairs over 6.9k directions, plus 1.3B mined, 1.1B back-translated, 448M forward-translated, 1B code/maths-to-English and 941M English-to-code pairsTable 34, §3.1
  • Foundational mixhuman-translated and mined data roughly reproducing the NLLB composition, plus synthetic translation generated with NLLB-200-3.3B from document-level web corpora§3.2
  • Stages 2–3 corpusrestricted to X→English, 183M primary pairs over 196 directions, plus 9.0M code/maths pairsTable 34, §3.1
  • Curation for Stages 2–3synthetic data excluded entirely and both sides globally deduplicated, discarding roughly 85% of the data to suppress false in-batch negatives; weak languages exempted§4.4
  • Code and mathsAST-segmented snippets from 7 programming languages and LaTeX expressions from scientific corpora, described by LLaMA3.3 70B Instruct and back-translated§3.3, §A.3
  • Languages200 foundational, extended to 4,200+ varieties; the omnilingual tokenizer and the extension are described as covering "over 1.5k" languages, and the paper does not reconcile the two figures§3.1, §4.1, §4.5.2
  • Omnilingual sourcesBible texts, PanLex and Tatoeba, with at least one side of every pair required to be a foundational language§3.2, §3.1
  • FilteringBLASER2 direction-specific thresholds μ − k·σ calibrated on FLORES dev; an early omnilingual encoder calibrated on BIBLE dev for the languages BLASER2 does not cover; length-ratio and exact-duplicate removal§3.5
  • Samplingnatural frequencies for Stages 1–3, temperature 0.6 for the omnilingual extension§3.5
  • Monolingual dataused only for the tokenizer warm-up, English auto-encoding in the extension, and small-encoder distillation§4.5.1, §4.5.2, §4.8
  • Speech dataapproximately 121k hours over 177 languages from the Omnilingual ASR Corpus, unfiltered§3.4, §3.5
Notes

On FLORES200 over 201 languages OmniSONAR reaches xsim 0.7 and xsim++ 6.1 against SONAR 2023's 1.4 and 15.3, and on the 1,560-language BIBLE benchmark xsim 3.9 against 68.7 (Table 4). Decoding the single vector gives chrF++ / xCOMET of 55.4 / 0.878 on FLORES200 against NLLB-3B's 55.8 / 0.849 — marginally behind on chrF++, ahead on xCOMET — and 41.3 / 0.702 on BIBLE against NLLB-3B's 24.3 / 0.377, Gemma3-27B's 26.3 and Llama3.3-70B's 26.2 (Table 8). MTEB average is 74.11 against SONAR's 63.34, still short of the general-purpose EmbeddingGemma at 77.06 (Table 7). The omnilingual extension costs nothing on the base 200: FLORES xsim 0.70 → 0.65 while BIBLE xsim goes 59.4 → 3.9 (Table 15).

Speech side: a single unified checkpoint, not per-language as in SONAR 2023, initialised from the Omnilingual-ASR wav2vec 2.0 models at 3B and 7B and trained by MSE regression onto the frozen text embedding of the transcript, with a three-layer transformer decoder doing attention-pooling (§4.6). 121k hours over 177 languages, 200k steps on 128 A100s (§3.4, §5). FLEURS xsim++ drops from SONAR's 17.7 to 10.1 at 3B, and zero-shot speech-to-text translation reaches 24.4 BLEU at 7B against SeamlessM4T's 25.6 (Tables 6, 9).

Release: no code, weights, repository or licence are stated anywhere in the paper, and no matching HuggingFace checkpoint was found at extraction time.

Everything on this card was read from arXiv:2603.16606v3 over the full 92 pages (via the arXiv LaTeXML HTML render of v3, cross-checked against a pypdf text extraction, as no PDF renderer was available on this box); §9 Spectrum and §10 OmniSONAR-Token were read but mined only for headline numbers, as they are separate models built on this encoder. Deliberately left off: max sentence length, the encoder/decoder parameter split within the stated 1.5B and 1.8B, the GPU model and wall-clock for Stages 1–3, the numeric value of nh in the decoder's grouped-query cross-attention, and any normalisation applied to z — the paper states none of these.

source: models/omnisonar.md
13

TAE-interp ladder (in-house)2026

the program's own 12M / 28M / 92M text autoencoders
TAE-interp program (this repository)

A scratch-trained family of SONAR-shaped sentence autoencoders — three widths, one frozen 256-d bottleneck, plain cross-entropy — whose only treatment is the objective feeding the decoder, built so decoder-side anatomy claims have a controlled second model.

in the review ↗
Input model
  • Initialisationrandom at every size and every arm; no pretrained weights, no distillation teacherexperiments/ladder/model_ladder.py
  • Embedding initstd = d_model**-0.5 with the pad row zeroed, pairing the sqrt(d_model) input scale; PyTorch's default N(0,1) collapsed rung A to a unigram decoder — experiments/ladder/model_ladder.py, experiments/ladder/RUNS.md
  • Warm-startan --init_from path exists; used once at 92M and retired after it diverged — experiments/ladder/train_ladder.py, experiments/ladder/RUNS.md
  • BART-style variant (out-of-family)facebook/bart-base, 139.8M including the graft, nothing frozen, everything fine-tuned; not part of the controlled 2x2experiments/ladder/bart_anchor.py
  • MT variantrandom init like the rest, but its own tokenizerexperiments/ladder/MT_PREREG.md
Model size
  • 12Md_model 256, 4+4 layers, 4 heads, FFN 1024, 11,601,408 params (7,505,408 non-embedding)experiments/ladder/runs_meta/A_D_s0.config.json
  • 28Md_model 384, 6+6 layers, 6 heads, FFN 1536, 31,189,120 params (25,045,120 non-embedding)experiments/ladder/runs_meta/B_D_s0.config.json
  • 92Md_model 640, 8+8 layers, 10 heads, FFN 2560, 102,485,376 params (92,245,376 non-embedding); the "92M" label is the non-embedding count — experiments/ladder/runs_meta/C_D_s0.config.json, paper/paperB/PAPER_B_DRAFT.md
  • Identical across armsD, P, P-prime, dose and M runs at a given size write byte-identical dims and parameter counts — experiments/ladder/runs_meta/A_P_s0.config.json, B_P_s0.config.json, C_P_s0.config.json
  • Shared architectureTransformer encoder-decoder, pre-LN with final LayerNorms, GELU, dropout 0.1, sinusoidal positions, max_len 64, output projection tied to the input embeddingexperiments/ladder/model_ladder.py
  • Tokenizerone 16k SentencePiece unigram for D / P / P-prime / dose arms, byte-fallback, coverage 0.9995, pad0 bos1 eos2 unk3, <mask> id 4experiments/ladder/train_spm.py
  • Tokenizer exceptionsthe MT arm uses a separate 16k en+fr SPM, the BART variant BART's 50k byte-BPE; CE and F1 from those two are flagged non-comparable — experiments/ladder/MT_PREREG.md, experiments/ladder/bart_anchor.py
Vector
  • Dimensionality256 at all three sizes and all arms; the bottleneck is held fixed so scale does not co-vary with it — experiments/ladder/model_ladder.py, docs/planning/TAE_LADDER_DESIGN.md
  • Poolingmasked mean over encoder token outputs, then Linear(d_model, 256)experiments/ladder/model_ladder.py
  • Into the decoder, both ways at oncem = Linear(256, d_model) is used as a one-token cross-attention memory and is also added to every decoder input embeddingexperiments/ladder/model_ladder.py
  • Normalisation / noise / priornone; z is untouched between encoder and decoder — experiments/ladder/model_ladder.py, experiments/ladder/train_ladder.py
  • BART variantsame 256-d mean-pool graft and same two-route injectionexperiments/ladder/bart_anchor.py
  • Planned d_z 64 / 1024 satellite at 12Mdesigned, never run — docs/planning/TAE_LADDER_DESIGN.md; no run directory exists
Loss / training
  • ObjectiveLCE alone — token cross-entropy with ignore_index=PAD, fp32 loss over a bf16 forward; no MSE, no KL, no term weightsexperiments/ladder/train_ladder.py
  • DAE noise (arm D, and the DAE half of every mixed arm)Poisson(3) span deletion to 30% of words, local shuffle with max displacement 3, then 10% of survivors masked, all on whitespace words before SPMexperiments/ladder/data_ladder.py
  • Paraphrase arm Pstrict alternation, even steps DAE and odd steps encode(A) -> decode(B), direction randomised per pair, no noise on the paraphrase source — experiments/ladder/train_ladder.py, experiments/ladder/data_ladder.py
  • Arm P-primethe arm-P recipe with SONAR round-trips of OWT sentences (top-p 0.9, T 1.0) kept at SBERT-cos 0.70-0.97experiments/ladder/pprime_gen.py
  • Arm M50/50 DAE and en->fr translation, direction fixed en->fr, no language tag — experiments/ladder/data_ladder.py, experiments/ladder/MT_PREREG.md
  • OptimizerAdamW, betas 0.9/0.98, weight decay 0.01, fused where availableexperiments/ladder/train_ladder.py
  • Schedule1000-step linear warmup, then cosine keyed to token progress down to 0.05 of peakexperiments/ladder/train_ladder.py
  • Peak LR, per size, same across arms5e-4 (12M), 5e-4 (28M), 2.5e-4 (92M); the 92M value is a correction after both original 5e-4 runs diverged mid-cosine — experiments/ladder/train_ladder.py, experiments/ladder/RUNS.md
  • Batch, per size512 (12M), 384 (28M), 256 (92M), set by 16GB VRAM — experiments/ladder/runs_meta/*.config.json, experiments/ladder/RUNS.md
  • Token budget, per size5e8 (40N), 9e8 (30N), 2e9 (20N) non-pad target tokens — experiments/ladder/runs_meta/*.config.json, docs/planning/TAE_LADDER_DESIGN.md
  • Steps37,360 (12M D) and 47,665 (12M P); 89,686 (28M D) and 114,455 (28M P); 299,069 (92M D) and 381,589 (92M P) — experiments/ladder/runs_meta/*.metrics.jsonl, experiments/ladder/RUNS.md
  • Stabilitygrad-clip 1.0 every step, plus a watchdog that exits rc=3 when val CE exceeds 1.5x best for two consecutive evalsexperiments/ladder/train_ladder.py
  • HardwareRTX A4000s, one run per GPU; 1.4h (12M D), 3.9h (28M D), 20.6h (92M D) — experiments/ladder/runs_meta/*.metrics.jsonl, experiments/ladder/RUNS.md
  • BART variant deviatesbatch 128, split LRs (backbone 1e-4, graft 5e-4), 3.5e8 tokens in 4.98h — experiments/ladder/train_bart_anchor.py, experiments/ladder/RUNS.md
Data
  • DAE corpusOpenWebText subsets 00/01/03/07, sentence-split, 4-60 whitespace tokens, lowercased, max token length 25, alpha ratio at least 0.55, no URLsexperiments/ladder/prep_owt.py
  • Size63.1M train sentences, 1.197B whitespace tokens; 20k val sentences held out from a subset never trained onexperiments/ladder/RUNS.md
  • Paraphrase pairscestwc/adapted-paranmt5m, 2.65M rows cleaned to 4.35M pairs plus 2k val pairs — experiments/ladder/RUNS.md, experiments/ladder/prep_paranmt.py
  • MT pairsopus-100 en-fr then europarl en-fr under a 60-minute staging timebox; 2.61M pairs used, 4.2 epochs on the translation half — experiments/ladder/stage_mt.py, experiments/ladder/ANATOMY_RESULTS.md
  • Dose bandseight similarity levels from SBERT-sim .654 to 1.000, built by filtering existing pools rather than generating pairs — experiments/ladder/DOSE_PREREG.md, experiments/ladder/ANATOMY_RESULTS.md
  • LanguagesEnglish throughout except the en->fr MT arm — experiments/ladder/prep_owt.py, experiments/ladder/stage_mt.py
  • Max length64 SPM ids, source truncated to 64 and target to 63 before BOS/EOSexperiments/ladder/train_ladder.py
  • Epochsunbounded buffered-shuffle passes with fresh noise draws each timeexperiments/ladder/data_ladder.py
  • Per-band and per-source pair counts for P-prime, the dose levels and MTnot recorded on disk here; the stats.json files stay on the GPU box
Notes

Best finals by validation cross-entropy and greedy roundtrip whitespace token-F1: 12M DAE 2.497 / .655, 12M paraphrase 2.750 / .578, 28M DAE 2.003 / .732, 28M paraphrase 2.198 / .627, 92M DAE 1.703 / .770, 92M paraphrase 1.847 / .658 (experiments/ladder/runs_meta/*.metrics.jsonl, experiments/ladder/RUNS.md). Length-stratified recon at 92M DAE reaches .826/.816/.762/.644 across the 4-8 / 9-16 / 17-32 / 33-60 token buckets against a SONAR ceiling row of .846/.835/.785/.722, the first cell in the family within .08 of SONAR's long-sentence bucket. The disk record states that CE keeps improving from 28M to 92M while roundtrip F1 at matched tokens does not (B-to-C gain at most .005 against A-to-B +.042), and that the full-budget 92M gain of +.038 over 28M comes from 2.2x the tokens rather than the parameters. The pretrained 139.8M BART variant lands at .644, matching the 12M scratch model at comparable fine-tune tokens. The MT arm reaches .454/.440 on its own tokenizer and is not comparable.

No weights, licence or Hugging Face identifier are recorded anywhere in the repository; checkpoints exist only on the 4l GPU box under runs/<run>/ as bf16 milestone and final snapshots.

This card was read from the on-disk sources in experiments/ladder/ (code, RUNS.md, ANATOMY_RESULTS.md, the four preregs, ten mirrored run configs and metric streams) plus docs/planning/TAE_LADDER_DESIGN.md and paper/paperB/PAPER_B_DRAFT.md; paper/ itself documents SONAR, not this family. Two traps for re-users: the mirrored C_D_s0 / C_P_s0 finals are the diverged 5e-4 runs (CE 9.41 / 7.03, F1 near zero) and the real 92M numbers survive only as prose in RUNS.md, and ANATOMY_RESULTS.md flags its refreshed anatomy_aggregate.json as mixing two scorer versions. Release, licence, per-arm corpus counts, an FVU figure and any d_z sweep are left off because nothing on disk states them.

source: models/tae-ladder.md