Why AI Reasoning Models Are Driving Up Benchmarking Costs: Understanding the Impact
AI labs such as OpenAI are making headlines with their advanced “reasoning” AI models, which are designed to tackle problems step by step and claim to outperform traditional models in fields like physics. However, benchmarking these sophisticated models proves to be a costly endeavor, complicating independent validation of their capabilities.
High Costs of Benchmarking AI Reasoning Models
According to Artificial Analysis, a third-party AI testing organization, the cost of evaluating OpenAI’s o1 reasoning model across seven prominent benchmarks is approximately $2,767.05. These benchmarks include:
- MMLU-Pro
- GPQA Diamond
- Humanity’s Last Exam
- LiveCodeBench
- SciCode
- AIME 2024
- MATH-500
In comparison, testing Anthropic’s Claude 3.7 Sonnet, which is categorized as a “hybrid” reasoning model, costs about $1,485.35, while OpenAI’s less complex o3-mini model is cheaper to analyze at $344.59.
Benchmarking Costs Comparison
It’s noteworthy that not all reasoning models carry the same benchmarking costs. For instance, evaluating OpenAI’s o1-mini model costs around $141.22. However, on average, reasoning models are significantly more expensive to benchmark than their non-reasoning counterparts. In total, Artificial Analysis has spent about $5,200 assessing around twelve reasoning models, which is nearly double the $2,400 spent on over 80 non-reasoning models.
For reference, OpenAI’s non-reasoning GPT-4o model, released in May 2024, cost just $108.85 to evaluate, while its predecessor, Claude 3.6 Sonnet, was even less at $81.41.
Future of AI Benchmarking
George Cameron, co-founder of Artificial Analysis, shared insights with TechCrunch about the organization’s plans to expand its benchmarking budget as more reasoning models emerge. He stated, “At Artificial Analysis, we conduct hundreds of evaluations monthly and allocate a significant budget for these tests. We anticipate this expenditure will grow as new models are released.”
Challenges in Benchmarking Costs
Artificial Analysis isn’t alone in facing rising benchmarking costs. Ross Taylor, CEO of AI startup General Reasoning, indicated that he recently spent $580 to evaluate Claude 3.7 Sonnet using around 3,700 unique prompts. He estimates that a single evaluation run of MMLU Pro would exceed $1,800. Taylor elaborated on the challenges faced by labs, stating, “We are entering a phase where a lab reports performance metrics based on significant computational resources that are often unattainable for academic institutions.”
The Token Economy and Its Impact on Costs
One reason reasoning models are so expensive to benchmark is due to their high token generation. Tokens are fragments of raw text, such as the word “fantastic,” which can be broken down into syllables. For instance, during benchmarking, OpenAI’s o1 generated over 44 million tokens, which is approximately eight times more than the GPT-4o model.
As most AI companies charge based on token usage, these costs can escalate rapidly. Modern benchmarks often require complex, multi-step tasks, which contribute to the higher token generation rates, as explained by Jean-Stanislas Denain, a senior researcher at Epoch AI.
Cost Trends in AI Models
Denain pointed out that while the cost of achieving a certain level of performance has decreased, the expense associated with the most advanced models has risen. For example, Anthropic’s Claude 3 Opus was priced at $75 per million output tokens upon its release in May 2024. Meanwhile, OpenAI’s GPT-4.5 and o1-pro models, launched earlier this year, cost $150 and $600 per million output tokens, respectively.
Many AI labs, including OpenAI, provide free or subsidized access to their models for benchmarking, but this practice raises concerns about the integrity of the evaluation results. As Taylor noted, “If you publish results that cannot be replicated with the same model, can it even be considered science?”
For more insights into AI benchmarking and the latest trends in AI technology, visit MIT Technology Review.
