Groundbreaking AGI Test Puzzles Leading AI Models: A New Challenge Emerges

The Arc Prize Foundation, a nonprofit established by renowned AI researcher François Chollet, has made headlines with the introduction of a new and challenging test designed to evaluate the general intelligence of advanced AI models. This innovative assessment, named ARC-AGI-2, has proven to be a daunting challenge for many leading AI systems.

Understanding ARC-AGI-2: A New Benchmark for AI Intelligence

The ARC-AGI-2 test has left many AI models struggling, with results that highlight significant gaps in their capabilities. For example, reasoning AI models such as OpenAI’s o1-pro and DeepSeek’s R1 achieved scores ranging from just 1% to 1.3% on the new test, while influential non-reasoning models like GPT-4.5, Claude 3.7 Sonnet, and Gemini 2.0 Flash scored approximately 1%.

What Does ARC-AGI-2 Measure?

The ARC-AGI-2 test consists of intricate puzzle-like challenges where AI systems must identify visual patterns among arrays of colored squares and produce the correct output grid. This format is specifically designed to assess an AI’s ability to adapt to unfamiliar problems, pushing the boundaries of its learning capabilities.

• The test aims to evaluate AI systems’ efficiency in acquiring new skills beyond their training data.
• Over 400 individuals participated in the test to establish a human baseline, achieving an average accuracy of 60%, significantly outperforming any AI models.

Improvements Over Previous Tests

In a recent update on X, Chollet expressed that ARC-AGI-2 is a more reliable indicator of an AI model’s intelligence compared to its predecessor, ARC-AGI-1. The new test addresses previous shortcomings by eliminating the reliance on brute force computing power, which was a notable flaw of the first test.

Chollet emphasized that intelligence encompasses more than just problem-solving abilities or high scoring—efficiency in skill acquisition is a critical component. As Greg Kamradt, co-founder of the Arc Prize Foundation, stated in a blog post, “The core question is not just whether AI can acquire skills, but also how efficiently it can do so.”

Performance Insights from Previous Tests

ARC-AGI-1 remained unbeaten for nearly five years until December 2024, when OpenAI’s advanced reasoning model, o3, surpassed all competitors by matching human performance on the evaluation. However, the cost of achieving such performance was notably high. The initial version of o3, known as o3 (low), scored 75.7% on ARC-AGI-1 but only managed a mere 4% on ARC-AGI-2, with a computing cost of $200 per task.

Industry Reactions and Future Contests

The launch of ARC-AGI-2 has sparked discussions within the tech community regarding the need for fresh benchmarks to evaluate AI advancements. Thomas Wolf, co-founder of Hugging Face, recently highlighted the industry’s lack of adequate tests to assess critical aspects of artificial general intelligence, including creativity.

Alongside the new benchmark, the Arc Prize Foundation has announced the Arc Prize 2025 contest, which challenges developers to achieve 85% accuracy on the ARC-AGI-2 test while limiting their costs to $0.42 per task.

For more information about the Arc Prize Foundation and its initiatives, visit their official website.