The Wisdom of Joseph Weizenbaum

“This reaction to the program ELIZA showed me more vividly than anything I had seen hitherto the enormously exaggerated attributions an even well-educated audience is capable of making, even strives to make, to a technology it does not understand.”

Men are all alike.

IN WHAT WAY?

They're always bugging us about something or other.

CAN YOU THINK OF A SPECIFIC EXAMPLE?

Well, my boyfriend made me come here.

YOUR BOYFRIEND MADE YOU COME HERE?

He says I'm depressed much of the time.

I AM SORRY TO HEAR YOU ARE DEPRESSED.

It's true. I am unhappy

DO YOU THINK COMING HERE WILL HELP YOU NOT TO BE UNHAPPY?

I need some help, that much seems certain.

WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP?

Perhaps I could learn to get along with my mother.

TELL ME MORE ABOUT YOUR FAMILY.

My mother takes care of me.

WHO ELSE IN YOUR FAMILY TAKES CARE OF YOU?

My father.

YOUR FATHER?

You are like my father in some ways.

WHAT RESEMBLANCE DO YOU SEE?

You are not very aggressive but I think you don't want me to notice that.

WHAT MAKES YOU THINK I AM NOT VERY AGGRESSIVE?

You don't argue with me.

WHY DO YOU THINK I DON'T ARGUE WITH YOU?

You are afraid of me.

DOES IT PLEASE YOU TO BELIEVE I AM AFRAID OF YOU?

My father is afraid of everybody.

WHAT ELSE COMES TO YOUR MIND WHEN YOU THINK OF YOUR FATHER?

Bullies.

DOES THAT HAVE ANYTHING TO DO WITH THE FACT THAT YOUR BOYFRIEND MADE YOU COME HERE?

While the above appears to  be a conversation between a psychotherapist and a patient, it is actually an interaction between a computer user and the program called Eliza, a natural-language program designed to simulate a conversation between two people. It was created in the mid-1960s by Joseph Weizenbaum, a professor of computer science at MIT, who died earlier this year. A pioneer in artificial intelligence, Weizenbaum had set out to explore whether it was possible to construct a computer program that could analyze natural (that is, ordinary, not computer) language when someone typed it on a keyboard and provide coherent, believable responses onscreen. The first “chatterbot,” Eliza was revolutionary in its time and Weizenbaum’s work continues to be recognized as a significant milestone in the field of human-computer interaction.

The reaction to Eliza was dramatic. So compelling was the simulation that users openly discussed their personal problems with the computer program. They expressed discomfort when they realized others could access the transcripts of their private conversations with the computer program. Psychiatrists even thought about using the program clinically to augment shortages in psychiatric care, making the “human therapist,” who would be involved in designing and operating the system, much more efficient. Even more surprising, generally intelligent people who did not understand the technology were eager to leap to the conclusion that Eliza represented a general solution to how computers can understand natural language. Weizenbaum was appalled. “The reaction to the program ELIZA showed me more vividly than anything I had seen hitherto the enormously exaggerated attributions an even well-educated audience is capable of making, even strives to make, to a technology it does not understand.” Like a twentieth-century Frankenstein who had constructed a monster, he felt he had to renounce his own creation.

The result was Computer Power and Human Rationality (1976), a tightly constructed, extended argument about the seductive powers of digital technology and the true limitations of that technology. Written over thirty years ago, he observed that there was already a schism regarding the question of how digital technologies are perceived. On the one hand there were the champions of technology, who believe computers can and should do everything and that there are no limits to what might be achieved. On the other hand, there were those who are skeptical and believe there are some things computers can’t—and probably shouldn’t—do.

Digital Limits

Computer power, Weizenbaum explained, is limited to the manipulation of symbols, symbols abstracted from real-world problems. They simulate the logic of the human mind but that is all. Using a formal, logical language, they process data by means of two-way branching. Computer decision-making follows a strict set of rules. Programmers may think that computers can do anything—can replace human decision-making—but in fact they can only do what the programmer tells them to do. And, Weizenbaum points out, human beings “know more than we can say.” While Weizenbaum says there is nothing wrong with thinking of humans, on some level, as information processors, he warns of the dangers of thinking that this is all there is to being human, that this mechanistic view fails to comprehend the whole of what it means to be human, especially when it come to judgment: “Since we do not now have any ways of making computers wise, we ought not now give computers tasks that demand wisdom.” His is both a commonsensical and an ethical argument, a refreshing respite from the constant praise for all things digital in our lives today.

Circles of Light

Computer technology, like all sciences, are self-validating systems. They define problems and their solutions within a circumscribed context and leave out much of the real-world data. “Science can only proceed by simplifying reality.” Weizenbaum recounts a joke about a drunkard to clarify this statement: One dark evening a policeman comes across a man on his hands and knees searching beneath a lamppost. He asks the man what he’s doing and the man replies that he lost his keys over there, pointing off into the darkness. “So why are you looking for them under the streetlight?” inquired the policeman. The man replies, “Because the light is so much better over here.” Similarly Weizenbaum argues science can only look where it already has light. “Two things matter,” concludes Weizenbaum regarding the nature of scientific inquiry, “the size of the circle of light that is the universe of one’s inquiry, and the spirit of one’s inquiry. The latter must include an acute awareness that there is an outer darkness, and that there are sources of illumination of which one as yet knows very little.”

The Scientist as Heretic

In the last year of his life, Weizenbaum spoke at length for a documentary (Weizenbaum: Rebel at Work).  Of his work on Eliza, he said that he came to consider it an act of great arrogance to try to fool everyone. At that time, he saw the choice of career paths as one in which he could continue to create more fantasy worlds and become “a guru of nonsense” in his field. Or he could become a social critic, proudly claiming the title of heretic in the worlds of computer science and AI. He wrote Computer Power and Human Reason. Science, he claimed, has become a slow-acting poison. It is so widely attributed with certainty that it has generally become the only way of understanding. Literature and the arts have been delegitimized and are now considered mere entertainments. “Belief in the rationality-logicality equation has corroded the prophetic power of language itself. We can count, but we are rapidly forgetting how to say what is worth counting and why.”

This book should be required reading for all who aspire to contribute to modern technology and engineering. We can only hope that the publishers are planning a commemorative edition of such an important contribution to the ongoing social conversation about the role of technology in our lives. Weizenbaum also spoke out against those who say that all this technology is “progress” and somehow inevitable. “The myth of technological and political and social inevitability is a powerful tranquilizer of the conscience. Its service is to remove responsibility from the shoulders of everyone who truly believes in it.”

For further information . . .

Computer Power and Human Reason is available in used condition through Amazon

Weizenbaum: Rebel at Work: DVD originally in German, available in English

Try out ELIZA interactively