Gwern.net — AI: GPT, LLMs & Language Models

Symbolic PFNs (SPFNs): Training LLMs for Symbolic Bayesian Inference by Reversing Stan · Gwern.net

Gwern — Mon, 02 Feb 2026 00:00:00 +0000

Proposal: train LLMs to predict Stan code from synthetic data sampled from Stan’s prior predictives. The output is an interpretable, executable Bayesian model, and a symbolic posterior over model structure. Possibly useful for faster inference, search, and interpretability.

Some 2025 LLM System Prompts, by Gwern, Gemini-3-pro-preview, GPT-5 Pro, Claude-4.5-opus · Gwern.net

Gwern, Gemini-3-pro-preview, GPT-5 Pro, Claude-4.5-opus — Tue, 18 Nov 2025 00:00:00 +0000

Trying out having Gemini-3-pro-preview write system prompts compatible with creative fiction writing (sci-fi short story examples), and trying it out on GPT-5 Pro and Claude-4.5-opus with mixed results.

Claude-2: Bats With Baby Faces, by Claude-2, GPT-4o, Gwern · Gwern.net

Claude-2, GPT-4o, Gwern — Tue, 11 Jul 2023 00:00:00 +0000

Some July 2023 Claude-2 & ChatGPT satirical writing samples of gonzo journalism & typography.

LLM Challenge: Write Non-Biblical Sentences · Gwern.net

Gwern — Sat, 28 Dec 2024 00:00:00 +0000

Can 2024-era LLMs reason successfully about how to write using only words that could not have appeared in any Bible ever? Mostly.

Towards Better LLM Creative Writing, by Gwern · Gwern.net

Gwern — Thu, 29 May 2025 00:00:00 +0000

What have I been up to with regards to LLM writing? A quick overview of how it all fits together, as of the first half of 2025.

You Could’ve Invented Transformers, by Gwern · Gwern.net

Gwern — Sun, 25 May 2025 00:00:00 +0000

Proposal for a ‘You Could Have Invented Transformers’ tutorial; someone should write a series showing a logical recreation of Transformers from primitive n-gram language models to full-strength Transformers.

Adding Bits Beats AI Slop, by Gwern · Gwern.net

Gwern — Sun, 23 Mar 2025 00:00:00 +0000

AI slop is unsatisfying because there is no there there. It is intellectual junk food that mimics nutrition but delivers only empty calories. Satisfying AI outputs must embed dense information and compute to actually reward a reader’s attention. You inject this value through brute-force search, non-trivial prompting, and rigorous curation, ensuring the final result reflects genuine algorithmic effort rather than the zero-shot ‘WYSIWYG’ default.

Regexps Used to Be AI, by Gwern · Gwern.net

Gwern — Mon, 24 Jun 2024 00:00:00 +0000

N/A

LLMs Can Be Faster Than You Think, by Gwern · Gwern.net

Gwern — Fri, 07 Feb 2025 00:00:00 +0000

2025 LLM technology is still far from the physical and statistical limits of how fast text or programs can be generated. It is plausible that with enough money, optimization, and hardware, large programs can be generated not in the current seconds, but in milliseconds.

Towards Benchmarking LLM Diversity & Creativity · Gwern.net

Gwern — Sun, 08 Dec 2024 00:00:00 +0000

Discussion of possible tasks to measure LLM capabilities in soft ‘creative’ tasks like brainstorming or editing, to quantify failures in creative writing domains.

LLM Acceptable-Use Policy, by Gwern · Gwern.net

Gwern — Sat, 24 Aug 2024 00:00:00 +0000

An LLM acceptable-use policy: authors must improve samples so as to be worth reading and training on, rather than spamming the world with random LLM outputs.

Virtual Comments: LLM Idea, by Gwern · Gwern.net

Gwern — Fri, 29 Nov 2024 00:00:00 +0000

N/A

GPT-3 Semantic Derealization, by Gwern · Gwern.net

Gwern — Sun, 16 Jun 2024 00:00:00 +0000

N/A

Why Do Writers Still Underestimate LLMs?, by Gwern · Gwern.net

Gwern — Sun, 12 Nov 2023 00:00:00 +0000

N/A

GPT-3 Creative Fiction · Gwern.net

Gwern — Fri, 19 Jun 2020 00:00:00 +0000

Creative writing by OpenAI’s GPT-3 model, demonstrating poetry, dialogue, puns, literary parodies, and storytelling. Plus advice on effective GPT-3 prompt programming & avoiding common errors.

GPT-3 Nonfiction · Gwern.net

Gwern — Fri, 19 Jun 2020 00:00:00 +0000

Nonfiction writing by OpenAI’s GPT-3 model, testing logic, commonsense reasoning, anagrams, PDF/OCR cleaning, creative nonfiction, etc.

Choose-Your-Own-Adventure AI Dungeon Games · Gwern.net

Gwern — Sun, 06 Jun 2021 00:00:00 +0000

Neural networks like GPT-3 power text adventure games where you can do anything; but they are too expensive. I propose that if we turn them into Choose Your Own Adventure hypertext games, they become feasible and enable new gameplay.

Crowdsourcing The Best GPT-2-1.5b Poetry, by Gwern · Gwern.net

Gwern — Sun, 09 Feb 2020 00:00:00 +0000

N/A

GPT-2 Preference Learning for Music Generation · Gwern.net

Gwern — Mon, 16 Dec 2019 00:00:00 +0000

Experiments with OpenAI’s ‘preference learning’ approach, which trains a NN to predict global quality of datapoints, and then uses reinforcement learning to optimize that directly, rather than proxies. I am unable to improve quality, perhaps due to too-few ratings.

GPT-2 Folk Music, by Gwern Branwen, Shawn Presser · Gwern.net

Gwern Branwen, Shawn Presser — Fri, 01 Nov 2019 00:00:00 +0000

Generating Irish/folk/classical music in ABC format using GPT-2-117M, with good results.

GPT-2 Neural Network Poetry, by Gwern Branwen, Shawn Presser · Gwern.net

Gwern Branwen, Shawn Presser — Sun, 03 Mar 2019 00:00:00 +0000

Demonstration tutorial of retraining OpenAI’s GPT-2 (a text-generating Transformer neural network) on large poetry corpuses to generate high-quality English verse.