Summary. This is a summary of the first installment of a series of essays written by Leopold Aschenbrenner — formerly of OpenAI’s Superalignment team, now founder of an investment firm focused on artificial general intelligence (AGI). Aschenbrenner titles this (first) essay, “From GPT-4 to AGI: Counting the OOMs”.

AGI in 2027. Artificial General Intelligence (AGI) by 2027 seems highly possible. This potential is evidenced by advancements in AI models, particularly through the incremental improvements from GPT-2 to GPT-4. This progress can be traced through orders of magnitude (OOM) in three main areas: compute, algorithmic efficiency, and unhobbling.

The Leap from GPT-2 to GPT-4. GPT-4 represents a significant leap in AI capabilities compared to its predecessors. GPT-2, released in 2019, could barely string together coherent sentences and was comparable to a preschooler in terms of intelligence. By 2020, GPT-3 emerged, displaying abilities akin to an elementary school student, capable of performing simple tasks, generating text, and even some rudimentary coding. Then came GPT-4 in 2023, which matched the intelligence of a smart high school student, excelling in coding, complex reasoning, and beating most high schoolers on standardized tests. Extrapolating the improvements from GPT-2 to GPT-4 provides the basis for estimating the arrival of AGI by 2027.

Trends in Deep Learning. Deep learning has evolved rapidly over the past decade. Initially, AI systems struggled with simple image recognition tasks. Today, these systems excel at numerous benchmarks, often surpassing human performance. This progress is due to consistent advancements in compute and algorithmic efficiencies, leading to smarter models with each passing year.

Counting the OOMs.

1. Compute: The compute used for training AI models has grown exponentially. For instance, the transition from GPT-2 to GPT-4 saw a dramatic increase in compute power, with GPT-4 using up to 10,000 times more compute than GPT-2. This growth has been fueled by massive investments in AI infrastructure.
2. Algorithmic Efficiencies: Algorithmic improvements have significantly contributed to AI progress. Over the past decade, algorithmic efficiencies have improved by approximately 0.5 OOMs per year. This means that the same performance can now be achieved with significantly less compute, leading to higher performance for the same amount of compute.
3. Unhobbling: Unhobbling refers to unlocking the latent capabilities of AI models through improvements like reinforcement learning from human feedback (RLHF) and chain-of-thought (CoT) prompting. These enhancements transform AI from simple chatbots to more advanced agents capable of complex tasks.

The Next Three Years. Looking ahead, the trends suggest that by 2027, we can expect another substantial leap in AI capabilities. This leap will be driven by continued improvements in compute, algorithmic efficiencies, and unhobbling. The anticipated growth in effective compute is expected to result in AI models capable of performing tasks currently done by human researchers and engineers. This progress could potentially lead to the development of AGI, models as intelligent as experts and capable of automating AI research itself.

The Data Wall. A potential challenge to this progress is the data wall. As AI models are trained on vast amounts of internet data, we are approaching a point where additional data may become scarce. This scarcity could limit the ability to further improve models through simple data scaling. However, advancements in algorithmic efficiencies and innovative training methods could help overcome this limitation.

Conclusion. The journey from GPT-4 to AGI is marked by rapid advancements in AI capabilities driven by scaling compute, improving algorithmic efficiencies, and unlocking latent model potentials. By 2027, AI models may reach a level of intelligence comparable to human experts, setting the stage for significant advancements in technology and potentially achieving AGI.

For more detailed insights, visit the full article on Situational Awareness.