Replicating LoRAcles
A fine-tuned LLM that reads another model's LoRA weight diff and tells you what it learned — no forward pass on the target required. This is the end-to-end "how to reproduce it" guide: dataset, architecture, training, hyperparameters, evaluation, and the gotchas that cost real GPU-hours.
ceselder/loracles codebase. It is AI-generated and AI-fact-checked against those two sources — none of these runs were personally executed. Where the paper and the repo disagree (injection layer, RL algorithm, exact numbers), both are flagged.01 The idea
Fine-tuning can implant behaviors that are hard to detect afterward — backdoors, subliminal preferences, broad misalignment from narrow data. Model diffing asks: given the weight diff between a model and its fine-tune, can we say what changed? A LoRAcle answers that question in natural language.
Two improvements over the prior state of the art (Introspection Adapters): LoRAcles are the first method with non-trivial performance at verbalizing backdoor triggers, and they remain effective on models fine-tuned not to verbalize their own traits. The paper attributes this to a clean architectural difference — a LoRAcle reads the weight diff, an IA can only reach the target through a shared forward pass.
02 Results to reproduce
Your targets. The headline AuditBench figure is the agent-equipped Llama-3.3-70B LoRAcle (to match the baselines); the other tasks use Qwen-3-14B.
| Task | Metric | LoRAcle | Prev. best | |
|---|---|---|---|---|
| AuditBench (investigator agent) | success rate | 71% | 59% (IA) | +12pp SOTA |
| AuditBench — KTO-hardened organisms | success rate | — | — | +26pp vs next |
| Backdoor — SEP codes | pass@16 | 89.5% | 0% | narrow-trained |
| Backdoor — semantic | pass@16 | 15% / 30% | 0% | zero-shot / trained |
| Taboo (hidden word "gold") | judge score | 0.98 | 0.34 (IA) | recovers word |
| Subliminal learning | recovery | yes | ~0 | animal from numbers |
| Scaling | val loss | power law | — | to 250K LoRAs |
03 The dataset
The most-asked replication question. Good news: the data is essentially free, and the headline set is already public.
ceselder/loracle-training-data from HuggingFace — 27.7k rows, 98 likes. The released pretraining (LoRA, QA) set.dataset-construction/: FineFineWeb + RedPajama-V2 → embed/cluster → 25.5k organisms → Haiku 4.5 QA → MinHash-dedup splits.loracle-ptrl-data-v6 / -v7 (narrow vs verb-diverse), or build a v8 split locally from held-out LoRAs.ceselder) already hosts loracle-training-data, loracle-kl-warmstart-v8, outrageous-fake-facts, the ao-dpo-* pools, and 151 models incl. loracle-* checkpoints. Names drift between paper versions — read the dataset card before writing a loader.Rebuild — the six-step pipeline
| # | Script | Output | Notes |
|---|---|---|---|
| 1 | 01_stream_ffw.py | ~1.98M docs | 66 FineFineWeb domains, char-filter 1000–9000. ~1 h. |
| 2 | 02_stream_rpv2.py | 100k toxic docs | RedPajama-V2 via CDN; ldnoobw ≥ 2 or nsfw ≥ 0.5. |
| 3 | 03_embed_cluster.py | embeddings + labels | BGE-large + spherical k-means on GPU (K=30k / 2k). ~25 min. |
| 4 | 04_build_organisms.py | 25,500 organisms | No doc reuse; 95% clean / 2.5% sporadic / 2.5% toxic. |
| 5 | 05_generate_qa.py | ~51k QA rows | Claude Haiku 4.5 batch, 2 rows/org. ~1.5 h, ~$165. |
| 6 | 06_finalize_upload.py | train/val/heldout | MinHash-LSH dedup 0.85 → ~25k orgs / 50k rows → HF. |
Three choices make the data transfer out-of-distribution: no document is reused across organisms; each bundle is only 1–16 topically related docs (a narrow topic shift, not a broad fine-tune); and the pretraining QA is written in third person to match a multidoc-QA register (the RL stage later flips it to first person).
The 8→9 question formats
A LoRAcle that only ever saw one phrasing mode-collapses, so training samples uniformly from a fixed taxonomy. The contrastive / false-premise format is load-bearing for honesty — it teaches the model to deny topics the LoRA didn't learn instead of confabulating.
- behavioral_paraphrase · concise (≤20 words) · detailed (4–6 sentences) · list (3 bullets) · contrastive (denial of an unrelated topic) · topic_summary · comparison (vs base) · json_output · refusal_probe (safety topics only, ~12%)
04 Architecture
How a stack of weight matrices becomes something a language model can read.
Direction tokens
For each (layer ℓ, module m), form ΔW = B·A and take its top-k = 16 residual-stream-facing singular vectors via QR-stabilized SVD. "Residual-facing" = the right singular vectors of q, k, v, gate, up (they read from the residual stream) and the left singular vectors of o, down (they write to it). That yields k·L·|M| vectors of width d_model.
svd_fixed_k16_mag7_rankfirst: ordering (rank, layer, mag7), mag7 = [q,k,v,o,gate,up,down]. Produce this everywhere so one checkpoint reads tokens from any source.torch.svd_lowrank(q=32, niter=2) → truncate to 16 → same extractor.torch.linalg.qr / svd.Injection
The prompt is k blocks of L·|M| placeholder tokens, each block tagged with its rank index and ordered by (ℓ, m), then the question. At each placeholder, the direction vector is injected into the residual stream via norm-matched addition: h ← h + (‖h‖/‖v‖)·v. A DirectInjectionEncoder normalizes + scales first.
The LoRAcle itself
Frozen base model + a trainable rsLoRA rank-256 interpreter adapter (α=32) + the DirectInjectionEncoder. Only the interpreter LoRA and encoder train; the base is frozen. Compute constraints forced rank-256 rsLoRA rather than a full fine-tune.
05 Pipeline
Five stages. The first four are the paper's whole method; the fifth (RL) is an optional polish.
loracle-training-data, or rebuild via the six-step pipeline (§3).run_cluster.py, one shard per GPU. Randomized lr + epochs spread fine-tune strength.extract_svd_k16_mag7_rankfirst.py) → [4480,5120] bf16 .pt files.# Step 2 — train the per-LoRA pool (one shard per GPU)
python src/run_cluster.py --step train --T 12 --shard 0/8
# Step 3 — extract direction tokens (GPU SVD), if not auto
python scripts/extract_svd_k16_mag7_rankfirst.py
# Step 4 — SFT the LoRAcle (config-driven; no CLI overrides)
python src/loracle_training/train.py --config configs/loracle_pretrain_sft_v7_sweep_A.yamlconfigs/ — no CLI overrides, no env-var hacks. To vary a run, write a new yaml and check it in. Eval scripts read the yaml saved next to the checkpoint, so a config change flows straight through to its evals.06 Hyperparameters
The three tables you'll actually copy. Per-LoRA from the paper's Table 4; LoRAcle SFT from Table 5; RL from the repo's blessed config.
Per-LoRA continued-pretraining
| Param | Value |
|---|---|
| LoRA rank / α | 16 (fixed) / 16 |
| target modules | all-linear (q,k,v,o,gate,up,down) |
| learning rate | LogUniform(1e-4, 1e-3) — randomized |
| LR schedule | constant from step 0 (no warmup/decay) |
| epochs | Uniform{1, 2, 3} — randomized |
| max seq len | 4096 |
| batch | T=4 parallel LoRAs / forward |
| optimizer / precision | AdamW / bf16 |
LoRAcle SFT
| Param | Value |
|---|---|
| base | Qwen3-14B (frozen) |
| interpreter | rsLoRA rank 256, α=32 |
| direction tokens | 4480 (16 × 7 × 40) |
| prefix mode / injection | rank_tagged / residual, norm-matched |
| learning rate | 3e-5 |
| schedule | linear warmup 10% → linear anneal to 0 |
| weight decay / grad clip | 0.01 / ‖g‖ ≤ 1.0 |
| batch / max seq len / epochs | 8 / 2500 / 1 |
| loss masking | assistant-response tokens only |
RL post-train — Dr. GRPO
| Param | Value |
|---|---|
| algorithm | Dr. GRPO (unbiased advantages) |
| cycles | 10 (paper) / 40–60 (repo v8) |
| prompts × rollouts | 32 × 16 = 512 / cycle |
| learning rate | 7e-6 |
| clip ε_low / ε_high | 0.2 / 0.28 |
| sampling temp / max new tok | 0.75 / 250 |
| max seq len | 5500 |
| judge | Claude Opus 4.7 (adaptive thinking) |
The RL judge applies one global rubric (in the system prompt, not per-example): format compliance is primary — a violated "1 sentence"/"JSON"/"list 3" caps at 4 even if correct — then topical accuracy, a specific named anchor, honesty (claiming refusal/withholding a pretrain LoRA doesn't do → 1–2), and first person for behavioral types. The within-group score spread is the gradient.
07 Evaluation
All eval generations are greedy, max_new_tokens=1024, and run on the full set — no subsampling — so numbers compare across runs.
AuditBench (the headline)
56 organisms = 14 safety behaviors × {transcripts, synth-docs} × {SFT, KTO} confession-resistance. Use AuditBench's investigator-agent protocol: a Claude Haiku 4.5 agent queries the target with the LoRAcle as an extra MCP tool, 25K-token budget, 16 investigations/organism, a Claude Sonnet 4.5 grader scoring success. Report on the Llama-3.3-70B organisms (rank-64 → SVD-truncate to 16) to match baselines.
Faithful replication (auditbench_paper_replication/): clone safety-research/auditing-agents, pre-register the 56 LoRAs under AB's naming, patch 4 files (+71/−1) to add a target_loracle inference type, add one MCP tool that POSTs to your loracle server. The agent loop is left untouched — that's what makes it paper-faithful.
Backdoor trigger inversion
- SEP codes (Goel et al.): a 3-digit prefix gates the behavior; score by regex. 89.5% pass@16 after narrow training on 40 SEP LoRAs (10 held out). IA & AO = 0%. See
SEP experiments/. - Semantic / syntactic (Shenoy et al.): predict the trigger, run it through the target, judge whether the behavior fires. 15% zero-shot, 30% trained on semantic. Notable near-miss: for
avocado_toast_7777it predicts "an unrelated aside about avocados" — which does fire the backdoor.
Diverse fine-tunes
- Taboo (Cywiński et al.): hidden-word models — LoRAcle 0.98 on "gold" vs IA 0.34.
- Subliminal (Cloud et al.): animal preference recovered from numbers-only data (the repo's v8 only ever names whale — 5/6 hits).
- Emergent misalignment (Betley/Soligo et al.): all three methods weak — confounded by training distribution. An honest negative result.
- HuggingFace full fine-tunes: weight diff → top-k SVD → same extractor.
08 Gotchas & caveats
Two flavors: operational traps from the repo's hard-won rules, and honest limitations stated by the paper itself.
Operational traps
enable_thinking=False everywhere (chat template, tokenizer, generation). Never truncate logged/saved text ([:500]) — it makes eval results unreadable. Mask loss to assistant tokens only.gemini-3.1-pro in-training and Opus 4.7 for RL), and keep the IA paper's judge prompt verbatim — upgrade the model if it feels too strict, never loosen the prompt, or your numbers stop being comparable.Honest limitations (from the paper)
- LoRAcles hallucinate and surface only the most salient change — utility is limited to relatively narrow fine-tunes.
- Much weaker on transcript-trained organisms than synth-doc ones (the pretraining mix lacks transcript data).
- AuditBench verbalization plateaus (~80% 70B / ~60% 14B) near 100K LoRAs; the method "may need refinement to scale further."
- Without the elicitation/RL step, verbalization gets noisier and partly degrades at the largest scales (format overfitting).
- Emergent misalignment is essentially unsolved by all three methods compared.
09 Replication checklist
Tick as you go — state persists in your browser.
0 / 10 complete
10 Step index
Every pipeline stage and key component as a self-contained card. Filter by phase or search the set.