The Room of Doom

Last time we talked about Turing machines (things like computers), and Turing completeness (the ability to do anything a Turing machine can). It turns out that there is a universal Turing machine (one which can simulate all others). Since every model of computation discovered so far is simulated by the universal Turing machine, some people think that the universal Turing machine can simulate anything, even the entire universe. Today we’re going talk about what this might mean.

“You wake to find yourself in a small room. The only noteworthy items are a large leather-bound book, a stack of blank paper, and a box of pencils.”

Open book.

“You open the book. The book contains a list of instructions. Page one reads, ‘Start here. Follow all the instructions. Don’t make any mistakes.’ It then tells you, in detailed steps, to draw a certain symbol on a sheet of paper and to slip that paper under the door. Then it tells to continue at the top of page 6. Page 6 says to draw four symbols, save them for later, then go to page 720. Page 720 says to take the symbols you just wrote down, copy them onto a new sheet of paper, slide the new sheet under the door, and continue reading on the next page.”

Flip through book.

“You discover that all the pages of the book give similar instructions. The instructions are clear and easy to follow.”

Turn to page one.

“You turn to page one.”

Read page.

“You begin reading the page…”

Meanwhile, outside the room, the prison guard notices a piece of paper emerge from under the door. The paper says, “Help!” The guard is surprised to say the least. Soon another paper emerges from under the door. It says, “I am very hungry.” The guard runs to tell his superiors, “We have mistakenly imprisoned a Chinese speaker.” (He’s yelling in Chinese of course.)

After a few minutes, the guard returns with two others. They are wearing lab coats. One says, “It’s no use. The door is already sealed.”

The other says, “Why didn’t he say anything when we integrated him?”

“I don’t understand it. If he had just spoken up, then we wouldn’t have imprisoned him forever for the the Crime of Ignorance.” (He means the crime of not knowing Chinese. It carries an awful penalty in their country.)

“Look, another message,” the guard says.

Over the next few months, wonderful thought provoking essays and poems emerge from under the door. Though the poet is sealed away in an Eternal Prison for the Crime of Ignorance, supplies can pass through the panel in the door. The closing mechanism never lets anyone see into the Eternal Prison, but they can send food, water, and paper through the panel. Soon they begin to write questions on the paper. The poet gives wise and insightful replies. Truly, this Sage knows the deep things of the world. But, who is the Sage? How does the Sage understand such things? Or does he, understand that is?

While all of this is going on, the poor soul locked in the room only knows that by following the instructions in the book, he keeps getting fed. He doesn’t have the slightest idea what’s going on outside. He knows nothing of the Sage. He doesn’t even know that the symbols are in Chinese. It just looks like a big, complicated mess.

When John Searle came up with this story, he said it shows that computers can’t really have minds. Computers just follow simple instructions, like the person in the box. Clearly, the person doesn’t really understand what’s going on: neither can a computer.

Searle published that argument a half century ago. The Chinese Room has been controversial ever since. Some have been bothered by it, others find it obvious and compelling, some are frustrated, others confused, some have written entire dissertations saying that the argument is deeply flawed in every conceivable way. I remember when I first heard Searle explain the Chinese room. Being much younger and less experienced, I could tell that something fishy about it, but I couldn’t say what.

There are two curious features of the Chinese Room. The first is the room itself, the fact that someone is locked away in a room and no one can see inside. The second is the book. How could a book have instructions that, when followed, fool the people outside the room?

The first feature is not unique to Searle’s Chinese Room. Turing proposed a test for showing machine intelligence. It was a imitation test. Suppose you’re IMing someone who you don’t know personally. You chat with them for five minutes. If you still believe that the “person” on the other end is really a person (and not just a computer program), then the program passes the Turing test.

Of course, Turing wasn’t the first person to think of something like this. Descartes proposed a similar idea (and dismissed it) in 1637:

“If there were machines which bore a resemblance to our bodies and imitated our actions as closely as possible for all practical purposes, we should still have two very certain means of recognizing that they were not real men. The first is that they could never use words, or put together signs, as we do in order to declare our thoughts to others. For we can certainly conceive of a machine so constructed that it utters words, and even utters words that correspond to bodily actions causing a change in its organs. ... But it is not conceivable that such a machine should produce different arrangements of words so as to give an appropriately meaningful answer to whatever is said in its presence, as the dullest of men can do. Secondly, even though some machines might do some things as well as we do them, or perhaps even better, they would inevitably fail in others, which would reveal that they are acting not from understanding, but only from the disposition of their organs. For whereas reason is a universal instrument, which can be used in all kinds of situations, these organs need some particular action; hence it is for all practical purposes impossible for a machine to have enough different organs to make it act in all the contingencies of life in the way in which our reason makes us act.”

Descartes has been right for three hundred sixty-seven years and counting. To date, no system has been able to fool someone for more than four or five sentences, let alone four or five minutes. Even if a computer that behaves like a person could be made, the Chinese room experiment shows that it’s still hard tell who, what, or even if it understands. The person in the room certainly doesn’t understand. Perhaps, the book does, yes the large leather-bound book. Any book that contains not just the teachings of a sage, but any answers a sage would give to any question—could such a book actually be able to understand Chinese?

Depends on what you mean by “understand.” For now, let’s stick with a behavior based definition—if it looks like a rose and smells like a rose, it’s a rose. In this case, if it writes like a sage, it’s a sage. Except the book doesn’t do the writing. The Prisoner does. Just having the instructions isn’t enough. Unless there is someone following them, there are no messages under the door and no sagely advice.

We might conclude then that only the entire system together (prisoner, book, paper) has sagely understanding. Searle anticipated this answer and disagrees with it. Imagine that over time the Prisoner memorizes the whole book. He still doesn’t realize that he’s been sending notes in Chinese, but now the system is just the Prisoner. The Prisoner writes like a sage without understanding what he’s writing. He seems to have developed the skill of automatic writing. (That might be a little far fetched.) On the other hand, our Prisoner displays the symptoms of someone with dissociative identity disorder. He has, “two or more distinct identities or personality states”: one is the Chinese Sage, the other is the Symbol Manipulating Prisoner (or SMP). The paradox of the Chinese Room (that the person appears to understand when in fact he does not) is not rooted in the symbolic nature of computers. The same paradox comes up in the case of dissociative identities. The symbols are a red herring (hence the fishy smell). The real issue is dissociation.

Identity, consciousness, the sense of self: tricky subjects. I’m content to say that I have a sense of myself, that you have a sense of self, and that most other people do as well. Not only is the self something that senses, feels, and is otherwise acted upon. But, it can act and usually acts freely. In other words, actions trace their origins to the self, and are not forced on the self by something else. Sometimes I may feel compelled by external factors; but, in spite of this, I maintain moral agency: I am responsible for the things I do.

The prisoner has two senses of self. One, he is aware of. The other is a dissociated identity (which he picked up from reading the wrong kinds of books). The symbols he writes mean one thing to the prisoner (I write these things to get food), and something else to those outside (the great teachings of the Sage).

The Chinese room shows that relationships are important for meaning. Since the Prisoner doesn’t have the right kind of relationship to the information in the book, he doesn’t understand that he is writing Chinese. Since the people outside the room cannot see into the room, their relationship to the Sage identity leaves them completely unaware of the Prisoner identity.

The Prisoner “simulates” the Sage in the sense that there exists a certain kind of relationship between the two. In particular, for every action of the Sage, there is a matching action of the Prisoner. (This relationship does not require awareness on the part of the Prisoner.) Simulation for Turing machines is defined similarly. Something is Turing computable if there exists a motion in the machine for every action of the thing. For example, arithmetic problems “find the product of 182 and 684” are Turing computable. That means you can make a machine such that motions in the machine correspond to doing arithmetic. We call such machines calculators. It is very hard to actually build an arithmetic Turing machine. Once such a machine is built, unless you know how the actions of the machine relate to the operations of arithmetic, the machine’s behavior will be very mysterious. Just as mysterious as the symbols are to the Prisoner.

Meaning is found in having the right sort of relationship to something. The arithmetic machine is potentially useful as a calculator, but only if you know how to use it, only if you know how to interpret the symbols it uses. Being formally equivalent isn’t enough. Searle comments, “no one would expect to get wet jumping into a swimming pool full of ping-pong balls simulating water.” A ping-pong ball simulator can, I suppose, capture some salient features of water. But it is very difficult to form the right relationship with the simulator so that you get wet when you jump into the pool.

“Difficult? You mean impossible, right?” No, I mean difficult. As an example, look at this. You see a clip of a marble bust rotating, right? Right, except no real photons ever hit this marble: the entire thing is simulated. The light, the motion—it’s just an animation. What’s more, if you look at this, I expect you see a jumble of text. It’s a few frames from the clip, all that’s changed is your relationship to it (in particular I changed the name of the file to end with ”.txt”). Meaning is found in having the right sort of relationships. What determines a relationship? You guessed it, an interface.