Gwern.net — AI: Scaling Hypothesis & Forecasting

AI Cannibalism Can Be Good, by Gwern · Gwern.net

Gwern — Sun, 27 Apr 2025 00:00:00 +0000

Training new LLMs on old LLM outputs is a good way to get a lot more data and recycle compute, and is not paradoxically self-defeating nor necessarily harmful.

Scaling Hypothesis Revisited: GPT-3’s 2^(nd) Anniversary & Looking Forward 2 Years · Gwern.net

Gwern — Sat, 28 May 2022 00:00:00 +0000

Revisiting GPT-3 and my ‘Scaling Hypothesis’ 2 years: it holds up and scaling has been too successful to be strangled in th ecrub. I speculate about the immediate future of scaling for 2023–2024 and beyond.

Scaling ‘Diminishing Returns’, by Gwern · Gwern.net

Gwern — Thu, 14 Nov 2024 00:00:00 +0000

What do we mean by dimishing returns in scaling? Returns have always missed— what it really means is worse than extrapolation of the known scaling power laws.

The State of Chinese AI, by Gwern · Gwern.net

Gwern — Fri, 15 Nov 2024 00:00:00 +0000

N/A

Hardware Hedging Scaling Risks, by Gwern · Gwern.net

Gwern — Thu, 22 Aug 2024 00:00:00 +0000

N/A

The Scaling Hypothesis · Gwern.net

Gwern — Thu, 28 May 2020 00:00:00 +0000

On GPT-3: meta-learning, scaling, implications, and deep theory. The scaling hypothesis: neural nets absorb data & compute, generalizing and becoming more Bayesian as problems get harder, manifesting new abilities even at trivial-by-global-standards-scale. The deep learning revolution has begun as foretold.

ML Scaling Subreddit, by Gwern · Gwern.net

Gwern — Fri, 30 Oct 2020 00:00:00 +0000

N/A

Technology Forecasting: The Garden of Forking Paths · Gwern.net

Gwern — Sun, 01 Jun 2014 00:00:00 +0000

Pessimistic forecasters are overconfident in fixating, hedgehog-like, on only one scenario for how they think something must happen; in reality, there are always many ways through the garden of forking paths, and something needs only one path to happen.

Slowing Moore’s Law: How It Could Happen · Gwern.net

Gwern — Fri, 16 Mar 2012 00:00:00 +0000

Weak points in the networks powering technological progress: chip factories