The Chinese room: does AI understand what it writes?

You ask ChatGPT or Claude a question, and the answer sounds like it was written by a smart, well-read friend. And then you wonder: does it even understand what it's writing to me?

The philosopher John Searle asked the same question back in 1980, decades before ChatGPT. He answered it with one of the best-known thought experiments in the history of artificial intelligence: the Chinese room.

In Searle's version, a person sits in the room. We'll borrow our sheep instead, because it's more fun.

A sheep in a room

Imagine a closed room with two slots for slips of paper. A sheep sits in the room. She's clever, but she doesn't know Chinese. Not a single character.

Someone pushes a slip of paper with Chinese characters through the left slot. To the sheep it looks like squiggles. But on the table there's a huge rule book written in her own language, and it says something like this:

Notice one thing: the book doesn't explain what the characters mean. It only says if you see these shapes, draw those shapes. The sheep finds the rule, copies the characters onto a new slip and pushes it out through the right slot.

The book is enormous and has a rule for every possible question, and the sheep is fast. Very fast.

The view from outside

Outside the room stands a native Chinese speaker. They write questions and get answers back: sensible, witty, grammatically perfect. Their conclusion is logical: inside sits someone who speaks excellent Chinese.

And inside? A sheep with no idea whether she just talked about the weather, gave a recipe for dumplings or declared her love to someone.

What Searle was getting at

A computer does the same thing as the sheep. It receives symbols, follows rules (that's the program) and returns symbols. Searle separates two things here:

• syntax: the rules for how symbols are put together,
• semantics: what those symbols actually mean.

A computer has syntax. Searle argues that by just following rules it never gets to semantics.

So the point of the experiment is simple: behaving as if you understand is not the same as understanding. For the same reason, Searle believes even the famous Turing test isn't enough proof. According to that test, a machine "thinks" if you can't tell it apart from a human in conversation.

So what about ChatGPT?

Language models do something very similar, just on an enormous scale. Nobody wrote them a rule book. Instead, by reading a huge amount of text, they learned billions of numbers. Those numbers tell them which word is most likely to come next. That's how, word by word, they put together an answer.

Sounds like the Chinese room? To many people, it does. It also explains why AI sometimes confidently makes up a fact: it assembles convincing text, it doesn't check whether it's true.

But wait, it's not that simple

The Chinese room has plenty of critics too. The best-known objection is called the systems reply: the sheep doesn't understand Chinese, but maybe the whole system does, meaning the sheep, the book and the room together.

Searle responds: let the sheep memorise the whole book and walk out of the room. Now she alone is the entire system, and she still doesn't understand Chinese.

The critics fire back: not a single neuron in your head understands English, yet you do. Maybe understanding always emerges from parts that, on their own, understand nothing?

There's also the robot reply. Maybe the room doesn't understand because it has never seen, tasted or touched anything it writes about. Give it a camera and hands, and then we'll see.

Who's right? More than 40 years later, philosophers still don't agree.

What it means for you

You don't have to solve a philosophical puzzle to use AI well. But the Chinese room offers three practical tips:

1. AI sounds just as confident when it's wrong. Always check important facts.
2. Your prompt is the slip of paper through the slot. The clearer your question and the more context you give, the better the answer you get.
3. Understanding is your job. AI is a powerful tool, but the judgement is still yours.

Maybe AI doesn't understand what it writes. But you have to understand what you're asking it for.

How language models "think" and why they sometimes get things wrong is exactly the topic of the first Sheep Academy session.