The AI Intelligence Gap: Why We Need to Study Machines Like Babies (and Horses)

The benchmarks are broken. Artificial intelligence systems are routinely surpassing human performance on narrow tasks – acing medical exams, mastering complex games – yet consistently stumble in the real world. This disconnect isn’t a sign of AI’s limitations, but a fundamental flaw in how we’re evaluating its intelligence. As AI pioneer Melanie Mitchell argues, we’re treating these systems as miniature adults, when a more fruitful approach might be to study them as “alien intelligences,” borrowing methods from developmental and comparative psychology.

The “Alien Intelligence” Framework

The term, as Mitchell explains, stems from the observation that advanced AI like ChatGPT feels fundamentally different from human cognition. Neural network researcher Terrence Sejnowski likened it to communicating with a space alien, while developmental psychologist Michael Frank extended the analogy to include the study of babies – intelligences we also struggle to fully understand. This framing isn’t about diminishing AI, but recognizing that our traditional methods for assessing intelligence, built around human capabilities, are often inadequate.

Beyond Benchmarks: The Problem with Current AI Evaluation

Today’s AI evaluation largely relies on benchmarks – standardized tests designed to measure performance on specific tasks. While useful for tracking progress, these benchmarks often reward “clever” solutions that don’t translate to genuine understanding or adaptability. Mitchell points to the example of AI systems that can pass the bar exam but lack the nuanced judgment required to be effective lawyers. The issue isn’t just about achieving high scores; it’s about how those scores are achieved. AI can excel at memorization and pattern recognition, but struggles with generalization – applying knowledge to novel situations.

The Need for Experimental Rigor

A core challenge is the lack of formal training in experimental methodology within the AI field. Computer scientists, traditionally focused on building systems, often lack the skills to design robust experiments that truly probe cognitive capabilities. This is where insights from psychology become crucial. Fields like developmental and comparative psychology have spent decades developing techniques for understanding nonverbal minds – infants, animals – who can’t simply tell us what they’re thinking.

Lessons from Psychology: Clever Hans and the Importance of Control

Mitchell highlights the famous case of Clever Hans, a horse who appeared to solve arithmetic problems. Years of study failed to reveal the trick, until a psychologist implemented rigorous control experiments. By introducing a blindfold and a screen, they discovered Hans wasn’t actually performing calculations, but subtly reading cues from the questioner’s facial expressions. This illustrates a critical principle: the importance of considering alternative explanations and designing experiments to rule them out. As Mitchell emphasizes, skepticism – even of one’s own hypotheses – is a cornerstone of good science, a mindset often lacking in AI research.

Probing Infant Cognition: A Case Study in Bias

Similar biases can creep into studies of infant cognition. Research initially suggested babies possess an innate moral sense, based on experiments showing a preference for “helper” characters over “hinderer” characters. However, subsequent research revealed that the babies were responding to the characters’ emotional displays – specifically, bouncing – rather than any inherent understanding of morality. This underscores the need for careful control of stimuli and a constant questioning of assumptions.

Replication and the Future of AI Evaluation

Beyond rigorous experimentation, Mitchell stresses the importance of replication – repeating studies to verify results. Unfortunately, replication is often discouraged in AI research, where novelty is highly valued. Papers that simply replicate existing work with minor improvements are often deemed “incremental” and rejected from top conferences like NeurIPS. This emphasis on novelty hinders the cumulative progress of the field. A shift towards valuing replication and building upon existing work is essential for establishing a more solid foundation for AI research.

The AGI Question and the Limits of Definition

The pursuit of Artificial General Intelligence (AGI) – AI with human-level cognitive abilities – is often framed as a measurable goal. However, Mitchell argues that AGI is a nebulous concept, constantly shifting as AI capabilities evolve. Measuring progress towards something poorly defined is inherently difficult. She expresses a healthy skepticism towards the hype surrounding AGI, suggesting that the focus should be on understanding and improving specific cognitive capabilities, rather than chasing an elusive ideal.

Ultimately, the future of AI evaluation lies in embracing a more nuanced and interdisciplinary approach. By borrowing methods from psychology, prioritizing experimental rigor, and fostering a culture of skepticism and replication, we can move beyond superficial benchmarks and begin to truly understand the nature of machine intelligence. This isn’t just about building smarter AI; it’s about understanding intelligence itself.

What are your thoughts on the challenges of evaluating AI? Share your perspective in the comments below!