The biggest mistake in AI research

on giving up the dream of intelligence we can control

Aim: to show that the attempt to build AI to solve well-defined tasks is untenable, with a detailed analysis of the ARC-AGI prize and an argument that there’s no difference between artificial intelligence and artificial life.

When an archer is shooting for nothing he has all his skill. If he shoots for a brass buckle he is already nervous. If he shoots for a prize of gold he goes blind or sees two targets—he is out of his mind! His skill has not changed. But the prize divides him. He cares. He thinks more of winning than of shooting—and the need to win drains him of power.

— Zhuangzi / Thomas Merton

CONTENTS

I
The problem of AI benchmarking—Goodhart’s Law—the purpose of the ARC Prize

II
Goodhart’s Law in the history of AI—the tradeoff between intelligence and energy—ARC’s definition of intelligence

III
ARC’s curious scoring method—the 2024 and 2025 winners—brute-forcing ARC

IV
The logic of ARC’s design—ARC’s failure by its own measure—ARC’s necessary failure

V
The nature of IQ tests—ARC’s defeat by Goodhart’s Law—the adaptationist fallacy in AI

---

I

The ARC Prize

Every scientific field has its own jargon with its own unique personality. Physics is an everyman with its ‘big bangs’ and ‘black holes’, neuroscience is a magician waving a wand around as it shouts ‘corpus callosum!’, while botany is a nonsense poet dreaming up ‘umbels’, ‘bracts’ and ‘glabrous’ leaves. Beware the Jabberwock, my son!

Machine learning, of course, is a quack. It uses psychological terms with abandon, showing no care for their established meanings or implications (‘reasoning’, ‘hallucination’, even ‘learning’ itself) and it borrows from the spell-book of anatomy and medicine (‘neural network’, ‘synaptic weight’, ‘ablation study’), as if to reassure us that it’s just like the fairy in The Adventures of Pinocchio and will soon transform LLMs from puppets into people.

Besides its anthropomorphisms, perhaps machine learning’s most overused term is ‘SOTA’, state of the art. How many hundreds or thousands of papers have branded a system as SOTA by some measure - not just as a show of incremental progress, but to boast of something truly useful and exciting - only for it to have no impact on the real world? Someone says ‘SOTA’ and I say “so what?”

At its worst, chasing SOTA results in AI is a form of selfish careerism, on a spectrum with p-hacking and h-index gaming. But, to be a little more generous, some SOTA-gloaters are simply working to advance normal software for humdrum tasks like image classification (and good on them, but shall we stop calling it AI?), while others perhaps haven’t learned the lesson of Goodhart’s Law: if you let a useful measure become a target in itself, it will cease to be useful.

This is a common pattern - reward hospitals for reducing length of stay and you’ll soon find patients being discharged before they’re ready, while the original intention of improved care goes out the window - and it’s happened countless times in AI, with researchers optimising for high benchmark scores at the expense of contributing to fundamental progress.

The Night Before the Exam by Leonid Pasternak (1895)

It’s in this context that the ARC Prize, now in its third year, is funding $2 million in awards for real progress on real tests of real intelligence, implicitly claiming to have designed a benchmark that overcomes the issues with Goodhart’s Law.

The original ARC dataset (there have been several versions which I’ll discuss) was introduced by François Chollet in his 2019 paper, ‘On the measure of intelligence’. Based on Raven’s Progressive Matrices - a common component of human IQ tests - its tasks look like this:

top row: example cases; bottom row: test case with animated solution

Here, each pair of grids shows an input and an output, though the output for the last pair is provided empty. The task is to infer the relationship between the inputs and outputs and to complete the output for the final pair. In this case, you need to copy over all the pink squares and add a yellow square wherever it would turn an L-shape into a 2 × 2 square.

Chollet doesn’t mention Goodhart’s Law by name in his paper and nor does the ARC Prize Foundation mention it in any of their official materials. This is understandable - a benchmark provider hardly wants to draw attention to the fragility of benchmarks - but it sits in the background when they say that their benchmarks are:

not just tests, they are a driving force for AI research. They guide researchers towards the most important bottlenecks on the way to AGI, and incentivize them to explore approaches that go beyond pattern matching and memorization.

This is a way of saying that the ARC Prize is in some respects more of a high-stakes gamification of Chollet’s preferred research agenda - it’s an open call for contributions wrapped up in an alluring package - and, to that extent, its formulation as a benchmark needn’t be taken too seriously. But the Foundation also says that its benchmarks are designed to “serve as a clear and objective measure of progress towards AGI” and it’s on that basis that I want to examine them.

II

Defining intelligence

Feeble king, crooked bishop, bloodthirsty queen,
unswerving rook and crafty pawn,
over the black and white road
they seek out and wage armed battle.

They do not know that the appointed hand
of the player governs their destiny,
they do not know what adamantine force
directs their will and their toil.

— from Aljedrez by Jorge Luis Borges

The deleterious effects of Goodhart’s Law on AI are as old as the field itself. Take the classic example of chess, which Chollet discusses. Though it may seem strange now, many researchers used to believe that a machine that could play chess would have to be human-like. Thus, in 1958, Allen Newell, Cliff Shaw and Herbert Simon wrote that “if one could devise a successful chess machine, one would seem to have penetrated to the core of human intellectual endeavor.”¹

This turned out to be completely wrong, of course, because we now have human-trouncing chess machines that are as unintelligent as mud. Newell and his colleagues got it wrong because they thought that a system’s “complexity of response [to a task] is dictated by the task, not by idiosyncrasies of the [system]” and this contains at least two fallacies of abstraction.

The first is thinking that a task such as chess can be defined without reference to any system that would perform it, when, in reality, task definitions are nothing other than generalised descriptions of system behaviour - the very meaning of chess is in the eye of the solver. From one perspective, computers ‘play chess’ because they can stand in for humans in tournaments, but, from another perspective, computers are playing an entirely different game - one that’s defined by the programs they run - which we conveniently overlay with human chess because the inputs and outputs match.

This is why we might say of two human players with different strategies for chess that they’re “playing different games.” There’s no sensible question of whether such a statement is really true or really a metaphor, as chess is nothing other than the various uses we have for the word. It may seem odd that we have a rigid sense of what chess is if chess is nothing besides a linguistic agreement, but this is the nature of human social practices, as I discuss in another essay:

How to understand Wittgenstein

Newell et al.’s other false abstraction was thinking that the functional complexity of a chess-playing system can be neatly delineated at the physical boundary of man and machine. On the face of it, it might seem astonishing that a computer can play chess with a few thousand lines of code, but remember that a computer first needs a human to program it. A computer is a human playing chess, albeit at a distance in time and space - it’s a puppet with strings that we’re pretending are invisible.

Chollet points out that the effort to engineer AGI by solving a specific task like chess is inevitably self-undermining. Humans play chess using generally useful cognitive abilities like perception, memory, learning, analysis and so on because evolution didn’t prepare us for the game in advance, but an AI researcher who can direct all their focus at chess will be able to take all sorts of shortcuts. Choices like foregoing a visual system and representing chessboards as numeric arrays are not incidental - every reconstitution of the game changes its nature.

Hence, in the spirit of Goodhart’s Law, Chollet writes that:

the focus on achieving task-specific performance while placing no conditions on how the system arrives at this performance has led to systems that … largely do not feature the sort of human intelligence that the field of AI set out to build.

Detail from a painting of a cat on a chessboard by Frank Moss Bennett (1923)

Failing to reach AGI is not the only downside of chasing short-term goals. It’s also ultimately more expensive, whether because naive solutions tend to be costly to run (e.g. chess programs before a few decades of Moore’s law kicked in) or because you always have to start from scratch with each new problem, increasing the total human investment (e.g. the software industry as a whole).

One of the reasons why humans evolved general intelligence is because we’re constrained to be resource-efficient, but if you have the option of being very inefficient, you can bypass intelligence entirely. This is why today’s leading AI companies are investing in data centre scale-ups - they’re literally paying to avoid having new ideas.

Take Anthropic as an example. They recently committed $100 billion in monopoly money to access 5GW of AWS computing capacity for Claude, to say nothing of other deals they’ve made with Google, SpaceX and others. In comparison to a human’s average energy needs, that 5GW alone is equivalent to the combined brainpower of the population of Western Europe, all directed at the likes of code generation and LinkedIn slop.

One can argue about whether the economic outcome of this unprecedented capital expenditure will be riches or wreckage, but such brazen resource consumption has historically been the counterpart to stalling innovation. Necessity is the mother of invention and fortune is its enemy.

As a result, the AI industry is facing both a development crisis and a measurement crisis: we have an endless supply of benchmarks touted as indicators of progress towards AGI which are actually detrimental because our collective pockets are deep enough to keep paying for naive solutions which are just barely sufficient to sustain short-term investment interest.

For some industries, AGI is irrelevant - with enough data and energy, they’ll be automated over the next few years regardless - but there are plenty of industries where data can never be had in a large enough supply and betting on inefficient technology is sure to fail.

Chollet diagnoses the underlying problem with all this as a failure to understand what intelligence even is. While the typical benchmark measures skill at some task, skill is only a byproduct of intelligence which can always be achieved by other means.

In contrast, Chollet suggests that intelligence itself is about efficient skill acquisition: if you compare two systems that reach an equal skill level, the more intelligent system is the one that’s least complex and uses the fewest resources. So he argues that to encourage progress towards AGI, we need to reward the most efficient learning systems, not the most skilful ones. In time, the most efficient systems will become the most skilful ones, but if we reward skill too early, we won’t get very far.

The ARC Prize has tried to deliver on this conception in a few different ways over the past three years, but its successive developments offer an insight into Chollet’s thinking on some difficult issues, so let’s take it chronologically.

The next section contains some mild technical discussion of system designs for ARC. If you want to stick with the big picture, skip to section 4.

III

Test-time trickery

Superficially, the Prize was administered in an odd way in 2024 and 2025 because its scoring didn’t consider efficiency at all - the leaderboard recorded only raw skill level, the top prize (first of $600,000, then of $700,000) was offered to whoever could achieve a skill level of 85% (no one did), and although submissions had to run offline with limited resources and report their cost to run, this favoured efficiency only weakly and indirectly.

There were a few reasons for this setup. One was that, although Chollet offered some formal definitions of efficiency in his paper, in practice it would have been impossible to measure fairly for the tasks in the benchmark at the time. But a far more important reason was that Chollet believes you can construct a benchmark which only measures skill and which is nonetheless resilient to Goodhart’s Law, so long as the test data is sufficiently unpredictable that performance can’t be bought. The logic behind this idea is subtle and I’ll analyse it in the next section, but I want to come at it empirically first and assess whether the Prize has encouraged the kind of progress it set out to.

In 2024, 1st place went to a team called the ARChitects, who scored 53.5%, winning $25,000. Their solution fine-tuned an LLM on a large amount of synthetic training data not provided by ARC and, for each test case, it synthesised many additional variants of the examples to fine-tune the model some more, before generating the most probable solutions by a method that used yet more data synthesis. The model also had its token set massively reduced so that it couldn’t do much besides represent ARC tasks. In effect, the ARChitects built a system that was bespoke to solving ARC tasks, which generated models bespoke to each test case.

Although this doesn’t sound like a very general solution - human perception is not setup to deal with ARC tasks specifically; humans don’t require pre-training to complete ARC tasks; humans don’t need to be shown a large number of variants of test cases to understand them; and humans don’t modify their perceptual apparatus for each task they encounter - it attracted a lot of attention because fine-tuning a model on each test case sounds a lot more dynamic than prompting a static LLM, suggesting analogies with how humans adapt to new problems.

These analogies are misleading. We know that the data-hungry model that was initially fine-tuned on the training data was not itself learning robust, general concepts or it wouldn’t have needed constant additional fine-tuning. It’s precisely because that method is not general that fine-tuning with extra data whenever possible adds some slight advantage. But that advantage is slight - remember, the solution was wrong almost half the time - because the quantity of data available at test time was small relative to the model’s needs. Fine-tuning at test time adds a little flex to the curve that the base model is fitting but it doesn’t take much for it to snap.

In fact, the appearance of this approach as a kind of adaptive learning was little more than an artefact of ARC being setup to withhold potential training data until test time: each model which was bespoke to a test case was a static LLM with no adaptability, and while you could say that it was the higher-level fine-tuning system that was adapting, that system did not possess or acquire any general understanding, as the models it churned out were not integrated with each other or reused, they were thrown away. This learn-and-burn strategy is where the analogy with humans most clearly breaks down, its motto being: ‘give a man a fish and he’ll eat for a day, teach a man to fish and he’ll catch one fish before you have to teach him again.’

General intelligence is not just an ability to solve well-defined problems as they arise, otherwise you could build it as a growing library of isolated, task-specific programs. Though human intelligence may look like a library of unrelated skills (reading, cycling, cooking), its generality consists in its use of multi-modal perception to interpret problems with respect to global goals (information gathering, movement, sustenance).² One of the key problems with current AI systems is that they have no goal structure. However sophisticated you think an LLM is, it exists only as the function of token-generation - you can’t leave an LLM unprompted to watch the world go by and wonder what to do next.

A better mental model for test-time fine-tuning is that it’s a clever way of exploiting ARC’s format to submit as many entries as there are test cases, each one suited to each test case, thus making the most efficient use of an inefficient technique. That such systems look adaptive is because gradient descent (a common algorithm for training statistical models) is itself already a pseudo-adaptive process, but such solutions make no advance on treating intelligence as pattern matching, which is a failing paradigm that Chollet explicitly wanted to discourage.

In 2025, the ARChitects came 2nd with the same method, with a score of 16.5%, the drop being because of an update to the test that increased its difficulty (more on that shortly). 1st place went to NVARC, a duo from NVIDIA who submitted a modified version of the ARChitects’ system, achieving 24%.

The NVARC team had a very straightforward strategy: increase the quantity and quality of the synthetic training data. Though this led to an increase in skill, the accompanying increase in training data meant that the overall efficiency of the solution was almost certainly worse, so, by Chollet’s definition, the top 2025 solution was likely less intelligent than the one in 2024.

These results are a clear sign that people don’t want or need to think about efficiency to make progress on ARC. This is even more obvious with the big model providers - although the largest frontier LLMs are ineligible for prizes because they’re too resource-hungry, ARC measures them on a dedicated test set separate from the main prize track. In December 2024, OpenAI’s o3 reasoning model scored 87.5%, priced at $456,000 to process 100 puzzles,³ generating text as long as a stack of 55 Bibles for each puzzle.

One natural interpretation of this is that $456,000 is the price you had to pay to buy 1st place through brute force, but the ARC Prize Foundation instead reported it as “a genuine breakthrough, marking a qualitative shift in AI capabilities.” In fairness, they didn’t hail o3 as a new kind of intelligence, and their report overall said that AGI is still science fiction, but they nonetheless wanted to make a high score on ARC sound like a significant indicator of progress.

Can that be so when the top solutions are pig-headed about pattern matching? Clearly, even the Foundation doesn’t think so because they’ll be discontinuing this version of the benchmark from 2027, feeling that it’s been compromised by LLMs. Well, are the only lessons to be learned here contingent on the abilities of LLMs, or were there more fundamental flaws in the benchmark design? Let’s finally turn to ARC’s design principles.

AI colouring of detail from Caravaggio’s lost Saint Matthew and the Angel (1602), from a black and white photographic reference

IV

The measurement paradox

In his 2019 paper, Chollet is admirably open-minded about how to define intelligence, acknowledging that the concept is anthropocentric and must be understood in relation to contextual interests, so there’s no single, correct definition, only various possibilities that might be more or less useful for different purposes.

That said, Chollet did not particularly reflect on the purposes served by defining intelligence as efficient skill acquisition, but let’s just assume that definition for now and follow its logic.

An evaluation of intelligence by this measure ought to look at one of the following:

1. Quantity of resources needed to attain a given skill level

2. Level of skill attained for a given quantity of resources

These are two perspectives on the same idea of efficiency, where (1) is more about using as little as you can and (2) is more about making the most of what you have.

In ideal conditions, measuring (1) would mean looking at different systems that have the same skill level on some task and seeing which has the overall lowest design complexity and resource use (such as in training data and running cost). Measuring (2) would mean looking at systems that have equal total complexity and seeing which performs best.

However, in practical conditions, (1) would be especially susceptible to the problems of Goodhart’s Law - if a benchmark were to require a specific skill level, engineers would primarily build to that level and it would encourage inefficient solutions - while (2) is problematic because it’s too difficult to assess the resource consumption of wildly different system designs.

Although this seems to plainly undermine the utility of Chollet’s definition - and is why the ARC Prize didn’t use an efficiency metric for its scoring in 2024 and 2025 - Chollet sought to bypass these issues by means of a design constraint that would ensure that a high skill level could only be attained efficiently.

The key to this is something Chollet calls developer-aware generalisation. In contrast to the standard definition of a system as generalising if it’s able to handle test cases that are not found in its training data (what Chollet calls system-centric generalisation), developer-aware generalisation requires a system to handle test cases that are not encountered by its developer during its design. This is by analogy with the fact that human intelligence is not only able to handle problems that individuals (i.e. systems) haven’t experienced before, but also problems that evolution (i.e. our developer) didn’t prepare us for.

Some care is needed with these definitions, which the paper states a little vaguely. The comparison with nature mustn’t be made too coarsely, as developers have an ability that nature doesn’t have: imagination. A developer can guess test cases that are not contained in any official dataset, and these guesses must be treated as among the examples that the developer ‘encounters’.

Moreover, Chollet formulated his ideas at a time before language models had become so popular, and their functionality requires an expansive notion of a ‘developer’: since LLMs are to a large extent knowledge bases, trained on reams of information about things like human visual perception and IQ test design, a developer using an LLM should be thought of as sharing in the LLM’s tacit awareness of the problem space.

By this measure, ARC in its 2024 version certainly failed to achieve its design goals because Chollet provided a benchmark specification that was detailed enough for developers to guess the kinds of tests the benchmark would contain, allowing them to imagine, generate and prepare for examples not in the training or development data. Additionally, the fact that a solution as naive as the ARChitects’ could achieve 53.5%, and that OpenAI could brute force 87.5%, indicates that LLMs silently increase developer awareness by a significant margin.

The 2025 update to the benchmark tried to correct these shortcomings by making the tasks harder, but arguably in the wrong way: the tasks were redesigned to compromise on a principle of being easily solved by an average human, instead requiring a good deal more analytical thought by a ‘smart’ human. This was achieved in essence by taking what would previously have been several tasks and composing them into a single, complex task.

While this made inefficient systems more expensive to use through a deepening of the search space, it didn’t address developers’ ability to leverage explicit and implicit awareness of common design patterns, so further reductions in the cost of computation could easily see the 2025 version of the benchmark beaten by force.

A more effective way of resisting developer awareness would have been to add new, undisclosed task types, rather than increasing the complexity of known types - in other words, to require systems to have capabilities that go beyond the task space revealed in the benchmark specification.

The logic of this is actually central to Chollet’s whole approach. If you base a benchmark on developer-aware generalisation, it’s really not that hard to come up with a completely robust test for AGI. It would look a little like this: someone specifies a protocol for interacting with some training and test datasets, they guarantee that the test will be something an average human can do, and that as much training data as a human would need will be provided. And that’s as much as anyone ever gets to know - no task specifications or data are ever shared.

This sounds ridiculous, of course, first because it unnecessarily formalises the problem of engaging with open-ended tasks in the real world - a system that could do well on such a benchmark wouldn’t need a benchmark to confirm its abilities - but second because no developer could reasonably build a system to pass a test consisting of completely unknown tasks.

This reveals a delicate balancing act for Chollet. If a benchmark is to encourage progress towards AGI, you need developers to be unaware of its test content, but for engineering to be tractable, developers need information that will make its test content guessable. There’s no principled way of sorting safe from unsafe information here - Chollet artfully tried to find a sweet spot, but I doubt there is one. A benchmark will only be a guaranteed measure of intelligence to the extent that developers cannot guess what’s in it, and if they can guess what’s in it, it will permit unintelligent solutions.

A looming consequence of this is that, if AGI is possible, it cannot be architected as a solution to an explicit problem. Unlike the controlling hand of the chess pieces at the start of Section 2, developers need to be prepared to let go after setting programs in motion that can develop their own inherent intelligence.

One way of interpreting the current paradigm in AI is that people are hoping that learning algorithms already embody this kind of autonomy. This would be true if training data constitutes an environment which systems must adapt to, but this is a misleading metaphor - a statistical model is a model in the sense that it’s a reflection of its training data; what’s learned is not how to cope with the data but how to imitate it.

This is why Chollet has astutely observed that pattern matching is not a major alternative to old-style symbolic programming on our path to AGI, it’s just a different way of writing programs indirectly via exposure to target behaviour. The fundamental problem in both cases is that, after construction, a program has no environment in which it can develop and achieve its own goals.

This year’s version of the ARC Prize has introduced an entirely new benchmark centred around agentic game-playing, but its format is only a new costume for the same underlying logic. Solutions will do no more than simulate something that looks like an agent from a human point of view, putting the same learning strategies towards merely simulated goals. An intelligent program would somehow have to be an agent in its own right, with goals that are not parasitic on human feedback.

But perhaps we’re getting ahead of ourselves. If we take a deep breath and try not to over-intellectualise things, isn’t the ARC Prize just a machine-readable version of an IQ test? If IQ tests work as intelligence measures for humans, why shouldn’t ARC work for computers?

Well, if developers were to follow the example of natural selection and be completely ignorant of ARC’s specifications, the benchmark could work that way. The major difficulty with ARC is that that’s not what it encourages and it’s not a realistic expectation. But the comparison with IQ tests bears some further investigation, as it will help us think more carefully about the nature of intelligence and its measurement.

V

Genius is one percent inspiration, ninety-nine percent perspiration
— Thomas Edison (paraphrase)

Although ARC was inspired by Raven’s Progressive Matrices, in some important respects the benchmark’s similarity to IQ tests is superficial, akin to how the similarities between human chess and computer chess are superficial - what Raven’s Progressive Matrices are is in the eye of the test-taker.

The most obvious divergence is that ARC is intended as a measure of skill acquisition, but think for a moment about some paradigmatic examples of human skill acquisition: learning how to walk and talk, learning a hobby, learning a trade, learning a musical instrument and so on. These are all time-consuming practices which are characterised by trial-and-error refinement loops. In contrast, human IQ tests are elicitations of near-immediate perceptual responses with no feedback.

To approach ARC in terms of skill acquisition, we have to think of it as providing a world made up of puzzles in which a system benefits from puzzle-solving, but this is as if natural selection were to have foreseen that we would one day benefit from knowing ballet - it orientates the development of the system around a desired behaviour, thereby undermining the very possibility of adaptation to the unknown.

This is a subtle point: on the model of natural selection, ARC’s test cases are meant to be unpredictable in order to require adaptation, but this is adaptation of the wrong kind - natural selection isn’t only blind to an organism’s environment (i.e. its test cases), it’s blind to the ways in which an organism can extract value from it. The fundamental issue with ARC is not its predictability (though that is a problem), it’s that it’s organised in the first place around a telos of puzzle-solving.

Chollet’s idea of appealing to natural selection and controlling for developer awareness was along the right lines, but he misunderstood the dynamics of evolution, which is non-teleological - if we’re to take the analogy seriously, an AGI benchmark ought to control for purpose awareness. The very heart of organic adaptive intelligence is that it generates fitness in the absence of feedback because intelligent systems have their own goals.

The radical implication of this is that, not only should there be no clear ‘point’ to a benchmark, the whole paradigm of using reward signals in model development is antithetical to the emergence of intelligence.

I said earlier that it’s misleading to say of learning models that they’re like organisms which have to adapt to an environment of training data, but there is an evolutionary idiom that fits: they’re like breeds in an artificial selection regime, where reward signals encode information about the breeder’s (i.e. developer’s) desired phenotype.

A mistake that people are apt to make with this is thinking that a breeding environment is the same as any other, the breeder’s preferences simply being another filter on reproductive fitness, like food supply or climate. This critically conflates two different senses of adaptation, one being intelligent goal preservation (e.g. altering your running pace to maintain your endurance), the other being mere change under pressure (e.g. an old pair of trousers stretching to fit an expanded waistline).

In artificial selection, the breeder replaces an organism’s goals with their own and, in equal measure, the organism’s potential for intelligent adaptability is sacrificed for domestication. Artificial selection is a way of turning adaptive biology into manipulable physics, all the intelligence being in the breeder’s control of phenotypic development. To call breeds (or reward-based learning systems) ‘intelligent’ is like saying that the fattest pigs are the cleverest ones.

This confusion underlies Chollet’s slogan (stated elsewhere) that intelligence is ‘adaptation to novelty’, which leaves ‘adaptation’ ambiguous between the two senses. It would be more precise to say that intelligence is resilience to novelty - it’s changing oneself to maintain a pre-existing goal structure in spite of environmental contingencies, rather than being changed by the environment to reflect it.

The fundamental problem with developing systems using reward signals, then, is that if we think of a reward signal as an environmental pressure, a system has no other goals to maintain in spite of it, while if we think of a reward signal as a goal to maintain, a system has no other environmental pressures to overcome. Natural selection’s characteristic dynamic between endogenous goals and exogenous pressures collapses and there’s no way for adaptive intelligence to express itself.

Using this analogy, we can further characterise Chollet’s idea of developer-aware generalisation as a kind of ‘breeder-aware convergence’, meaning that systems are tasked with arriving at a desired phenotype with only partial information about what it is. This modifies the scenario to make it less like breeding pigs to be fat and more like breeding dogs to be capable of learning tricks - it’s an effort to move some of the phenotypic convergence into the post-evolution (post-training) life cycle.

Ultimately, however, this means Chollet’s idea of ‘generalisation’ is indistinguishable from ‘efficient specialisation’ - it focuses on the capacity of intelligence to apply itself to a broad range of tasks while failing to recognise that what makes general intelligence unspecialised is that there are no extrinsic demands on it; its effectiveness is measured in utility for the organism rather than correctness to an examiner.⁴

Well, doesn’t breeder-aware convergence still represent some kind of advance in intelligence? Isn’t a dog’s learning ability more sophisticated than a pig’s girth? We again have to be careful to distinguish the two senses of ‘adaptation’, as ‘learning’ is just a relocation of adaptation in ontogeny instead of phylogeny, and is ambiguous in exactly the same way.

Consider the difference between a child learning a native language and an adult learning a second language. Where’s the intelligence in each case? For a child, language acquisition is an effortless developmental process, no harder than growing hair, so the intelligence belongs to the evolutionary dynamics which made language acquisition possible at all. In contrast, an adult language learner must make their own effortful interpretations of language with respect to self-determined goals, meaning the intelligence belongs to them.

There’s therefore a difference between systems with their own intrinsic intelligence and systems which are the product of an extrinsic intelligence, yet we’re quick to call both ‘intelligent’ without distinction. In each case, we can model the intelligence in evolutionary terms, but for a system to have intrinsic intelligence, it must be capable of self-evolution, meaning it must be able to wilfully vary its interpretations of its environment and autonomously recognise which interpretations better serve its goals.⁵

Such intelligence requires no external feedback and is actually undermined by it, as feedback leads to the replacement of intelligence with imitation. Imitative learning, which is driven by exposure to data and is not goal-preserving, is simply a kind of programming, in both animals and machines. Such learning is useful and, indeed, it’s the great achievement of LLMs - the fact that you no longer have to know how to code to get a computer to automate a variety of tasks is like being able to buy and train a dog instead of having to do your own selective breeding - but this is an advance in malleability, not intelligence.

Because ARC continues to encourage an artificial selection dynamic (albeit with modifications), it ultimately functions exactly like the lodestar of chess in the early days of AI and thereby falls to Goodhart’s Law. I said earlier that the Law warns that if you let a useful measure become a target in itself, it will cease to be useful, but we should be a little more precise about it, as this common formulation is not true of all systems.

In his paper where he stated the idea,⁶ Goodhart’s specific concern was that we use statistical regularities to gauge properties of complex systems that we can’t measure directly (such as the overall health of the economy) and whenever we try to game these correlated measures, we undermine the thing we really care about and compromise the correlation.

Goodhart’s Law can therefore be understood more clearly if we make a distinction between a target and a probe, where a target is some property of interest and a probe is an indirect, correlated measure of it, the lesson being that if you let a probe for a target of a complex system become a target in itself, it will cease to work as a probe.

Since intelligence is a target which we cannot measure directly, any intelligence test should probe for correlated measures without encouraging people to engineer solutions for it. This is how human IQ tests work - Raven’s Progressive Matrices don’t test few-shot skill acquisition, they test whether subjects make the kinds of interpretations of stimuli that are correlated with broader problem-solving ability. But since ARC defines intelligence as skill acquisition efficiency and it encourages people to build efficient skill acquisition systems, it tries to target intelligence directly.

This capitulation to Goodhart’s Law, together with the benchmark’s teleological design, really undermines the pinning of intelligence to skill acquisition in the first place. Specifically, if we’re to understand skills as learned responses to reward signals, then defining intelligence as skill acquisition falls foul of the adaptationist fallacy, which treats selection as essentially a problem-solving process. This has long been criticised in evolutionary theory as too narrow, since there are many other ways for traits to emerge. To say bluntly that intelligence is the arch-solution to the über-problem of ‘solving problems’ is an adaptationist just-so story par excellence.

In many ways, intelligence is not a problem-solving ability in itself, but the precondition for the invention of problems. Consider that, for all other lifeforms, a meteor would be a ‘problem’ in the sense that it could end their lives, but only with humans would a meteor be a problem both for us and to us, in that it would appear within our Umwelt as an event to do something about. To have an Umwelt in this way is not a passive perceptual ability, but a quintessential example of intelligent interaction with the environment, which assumes some amount of self-motivated interpretative behaviour.

Though some of Chollet’s mistakes in defining and measuring intelligence may be rooted in misunderstandings of evolutionary theory, I think they’re also unconsciously entangled with AI’s history of trying to distil the features of intelligence which have the most tangible economic value - this is the implicit, unanalysed purpose behind defining intelligence as skill acquisition, ‘skill’ being a byword for a process which is measurable and monetisable.

A better name for efficient skill acquisition would be ‘genius’ - what the industry yearns for is a bottling of the creative byproducts of intelligence, without care for the actual substrate of intelligence itself. But, of course, genius and intelligence are inseparable: if intelligence is the ability to self-evolve, genius is a proficiency at exploring the evolutionary landscape. And since both are part of an enactive dynamic between a world and an autonomous being, in the end there’s no difference between artificial intelligence and artificial life, and what on earth are we doing trying to solve puzzle games with language models?

1. Newell, Shaw and Simon, ‘Chess-playing programs and the problem of complexity’, IBM Journal, 1958.

2. You could think of this as maintaining a world model if you like - I don’t like ‘model’ talk myself because I think the idea that perceptual or linguistic systems encode a version of the world is a confused regress. As per enactivism, I think our ‘world’ is an interaction, not a map, which is why I focus on goal orientation.

3. The ARC Prize Foundation reported this as a retail price, so it was not necessarily the cost to OpenAI, bearing in mind that most model access to date has been sold at a loss.

4. Some people might confuse themselves by asking: “can’t it be of utility to an organism to do what an examiner wants?” This again collapses the distinction between endogenous goals and exogenous pressures, leaving room only for adaptation-as-change.

5. In a philosophical mould, this is essentially Aristotelian phronesis, or practical wisdom - the ability to recognise what’s good for us. A corollary of this is that intelligence is bound up with predictive modelling of multiple possible futures and, in turn, volition.

6. Charles Goodhart, ‘Problems of monetary management: the UK experience’, 1975.