The AGI Misunderstanding: What Does “Think of a Number” Test?

I opened LinkedIn after a relaxing weekend, and everyone seemed to be announcing AGI. Again. Apparently intelligence becomes general every time I stop checking my phone.

My daughter wanted to try a number game with Gemini. Let's play it first:

1. Think of a number.
2. Take the same amount from your friend.
3. Take 10 from me.
4. Throw half the total into the sea.
5. Return your friend's original amount.

You have 5 left. I know. Incredible powers. Please direct all research funding to my kitchen.

But what number did you start with?

The game cannot tell us. Call that number $x$:

The original number disappears. A person who picked 73, a program that stored 12, and a system that simply recognized the algebra can all give the same answer. Getting 5 neither proves nor disproves that a number was chosen.

That matters because it is very easy to stage a demonstration and mistake it for a test of the thing we care about. The AGI announcement and the triumphant debunking can make exactly the same mistake.

What does “think of a number” ask for?

There are at least three different requests hiding inside it.

Choose a value. Software can do that. A language model can output a numeral; a system can use a random-number generator. Calling the operation “next-token prediction” does not make the selected value disappear.

Keep the choice and use it consistently. That is a question about memory and behavior across steps. We can test it. Have the system commit to a number, record it somewhere the evaluator can inspect, then ask questions whose answers actually depend on that number. The algebra trick above is a terrible test because it removes precisely the information we need.

Have the private experience of choosing. That is a question about subjective experience. A fluent answer does not settle it, and neither does a correct calculation. “I felt like choosing seven” is an output to examine, not a window we have opened into a mind.

I care about keeping those questions separate. Otherwise we move from “it completed the task” to “it has a perspective” without noticing that we changed the claim halfway through.

An opinion is more than a convincing paragraph

A model can generate an argument, compare alternatives, and help me reach a decision. Those are useful capabilities. But when it writes “I believe,” I still want to know what stands behind the sentence: which criteria it is applying, whether the answer survives counterevidence, and whether the same commitments hold when the wording changes.

This becomes especially important in problems where the goal itself is disputed. Which customer should a system serve? Which trade-off is acceptable? Who gets to define success? More computational power can help us explore the consequences. It does not, by itself, tell us whose values should govern the choice.

That is my objection to easy AGI declarations: the word can swallow capability, autonomy, judgment, and consciousness, then return as if we had measured one thing.

By all means, show me the machine doing something remarkable. Then tell me exactly what the demonstration establishes. If the claim is memory, test memory. If it is reasoning, choose a problem where the reasoning matters. If it is experience, do not smuggle that conclusion in with the answer to an arithmetic trick.

The number in the sea was 5. The interesting questions are still on the shore.