Evan Fletcher
IST 008A-001
Luke Baldwin-Brown
Dec. 12, 2006
Science Fiction Misrepresents AI
Science helps us understand the world we live in in greater detail, but its main purpose is
to provide a practical means for accurately predicting future events. Science fiction, like science,
also attempts to predict future events based on our observations of the present, but for the
purpose of understanding ourselves and the world we live in rather than as a practical tool.
Science fiction gives writers the opportunity to give their own predictions of what will happen
and, in the process, reflect on aspects of the human condition. One crucial aspect of the human
condition that science fiction writers are fond of exploring is our consciousness. More
specifically, science fiction writers are fascinated by the idea that our consciousness could be
duplicated by a non-human entity, such as a computer, in the near future. Although science
fiction discusses, in great length, why an artificial intelligence (AI) could become conscious and
why an AI may or may not act according to our moral standards, science fiction rarely discusses
why an AI could not be conscious. This is because when a character with AI does not have
consciousness it becomes “just a machine” in the minds of the audience, which makes any plot
around that character or any of that AI's character development seem unimportant and dull. For
most media involving a character with AI, that character is crucial to the plot, and so the
character must be conscious for the piece to have been entertaining enough to get popular
acclaim. Popular books and films in modern culture such as Philip K. Dick's novel Blade
Runner (Do Androids Dream of Electric Sheep), Isaac Asimov's novel I, Robot, Stanley
Kubrick's film 2001: A Space Odyssey, and the Wachowski brothers' film The Matrix are classic
examples of AI in science fiction, but none of them discuss the ideas of thinkers, like Penrose
and Searle, who argue that a computer could never duplicate human consciousness.
Baldwin-Brown 2
Penrose was reacting in opposition to ideas discussed by other thinkers and which happen
to often be discussed in modern science fiction. The Churchlands and Turing both argued that
consciousness is constructed only from a set of complex computational processes. Penrose
disagreed with the Churchlands and Turing (whose ideas about AI are often discussed implicitly
in science fiction) and showed that no algorithmic system could produce a consciousness like our
own because we are capable of doing things that no algorithmic system can do. The problem
that Penrose shows conscious beings (such as programmers) can solve and that an algorithm
cannot is called the Halting Problem.
The Halting Problem is one that relates, quite closely, to programming. Often times a
program can follow a set of steps that will not lead to a conclusion and so the program will never
stop running. This is often called an infinite loop or a non-terminating procedure. The computer
then freezes up and is unable to perform any more useful operations. One way to solve this
problem would be to have another program called a Non-Termination Detector (NTD) that could
tell whether a program you gave it would continue to run forever or give you an output. The
NTD would terminate if the program it was given would not terminate and the NTD would not
terminate if the program it was given would terminate. The Halting Problem arises when we ask
the NTD to tell us if an NTD will terminate or not. The NTD could terminate, telling us that the
NTD is a non-terminating program, or the NTD could not terminate, telling us that the NTD is a
program that terminates. Whether you say the NTD will terminate or not, the NTD must be
giving you an incorrect result, and so we have run into a logical contradiction. Therefore, an
NTD must be impossible.
Baldwin-Brown 3
What Penrose points out is that people can tell before a process is done whether that
process will terminate or not (in other words, will give you a conclusion or not). Programmers
can easily look at a set of code and tell you whether that piece of code will fall into an infinite
loop or not. People are capable of doing what the NTD would do, therefore a computer must not
be able to do everything a person can do. This then proves that part of our consciousness lies
outside what is capable of an algorithmic system such as a computer, and so AI could never
duplicate (or simulate) our consciousness.
Other thinkers have come to similar conclusions about consciousness, but by different
means. For example: John Searle tried to prove that consciousness could only be simulated by a
computer. A purely algorithmic system could not produce a truly conscious being, but could still
act exactly like a conscious person. He argued that algorithms are completely syntactical and
that someone following a set of steps does not understand the meaning or purpose of the process.
For example: if a computer was told individual steps to make an ice cream cone the computer
would not understand the idea of an ice cream cone, only the individual steps necessary to
produce it. This is different from a person who can not only make the ice cream cone, but
understand what it means to have an ice cream cone. Therefore, a system that can only perform
syntactical operations could only look as if they are conscious and not comprehend the meaning
behind the operations, something Searle considers necessary for conscious thought.
The problem is that the ideas of thinkers like Searle and Penrose are not often reflected in
science fiction. If a writer did incorporate their ideas, the writer would have to make clear that
whatever characters with AI they have constructed are not capable of consciousness or qualia.
The problems those characters are confronted with or create then seem unimportant because they
Baldwin-Brown 4
are the problems of a machine and not a person. Science fiction that discusses AI in detail
usually has main characters with AI. So, to keep the character with AI from becoming “just a
machine” in the mind of the reader, they have to give the character with AI consciousness and
human like emotions. This makes reflecting the ideas of Penrose and Searle in science fiction,
while still keeping the piece entertaining, very difficult.
The novel I, Robot by Isaac Asimov is one such piece of science fiction that, through a
set of short stories, not only doesn't support arguments against an AI having consciousness, but
tries to show the antithesis: that consciousness could be completely computational. Isaac
Asimov's book is based around the idea that if a robot had enough computational power and held
the three “Laws of Robotics” as basic tenants, that it could develop consciousness through
computational intelligence alone and still logically follow a good set of morals. The three Laws
of Robotics are, basically, a simple set of moral rules for the robot to follow in order to guard the
interests of the people who created the robot. Isaac Asimov in one short story from I, Robot
states the laws like this:
1. A robot may not injure a human being or, through inaction, allow a human
being to come to harm.
2. A robot must obey orders given it by human beings except where such orders
would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not
conflict with the First or Second Law (Wikipedia, “Three Laws of Robotics”).
Based on these laws a robot could never murder or harm another person. A robot could
not be ordered to harm another person and would only harm itself if ordered to harm itself. The
robot holds the same basic instinct of survival that all people hold. A robot also could not
prevent itself from saving a human being if given the opportunity since “a robot may not injure a
Baldwin-Brown 5
human being or, through inaction, allow a human being to come to harm.” These three basic
laws seem to set up a good moral structure for the robot and allow the intelligence of the robot to
increase while still maintaining that the robot have morals and goals coinciding with that of
actual people.
Later on in the novel, robots (as they gain more intelligence) begin to develop
characteristics similar to that of human beings because of the three Laws of Robotics. In one of
the short stories a small group of robots develop a religion based on the white dots they see
through a telescope they are ordered to take data from. Since they have never seen a star up
close, when they are told that the white dots they are seeing are huge balls of gas light-years
away from them, they believe that the human is lying. The robots created this theory for the
cosmos (much like early humans must have) in order to make thinking about the problem
simpler and more logical. The robots used the data they were given to create a theory of the
world around them that could be applied to their studies, just like how a human would.
What Isaac Asimov was trying to show with stories such as this is that complex emotions
and thought could be genuinely reproduced in a computer system from a basic set of rules,
implying that an AI could at least simulate (if not have) consciousness, given the AI was
complex enough. This perfectly reflects Turing's theory that a computer could accurately reflect
human behavior so as to make their behavior indistinguishable from human behavior, and so
would have to be deemed conscious. Isaac Asimov chose to write the book in favor of Turing's
ideas because if the robots did not show some kind of human emotion, the problems found in the
robots would seem trivial and not applicable to the reader's sense of self. The entire book was
based on the idea that AI could be conscious if constructed on basic rules, and so not having
Baldwin-Brown 6
those robots be conscious makes the three Laws of Robotics seem useless to the audience.
Much of Stanley Kubrick's film 2001: A Space Odyssey revolves around the artificially
intelligent computer HAL, but in his story the AI (HAL) does not hold the same moral standards
as we do. When we first meet HAL 9000, it seems nice enough, but it is not clear whether the
computer is actually conscious or simply pretending it has human emotions so as to make it more
approachable. Kubrick later makes it clear that HAL is not only acting like a logical being, but
also like an illogical, human consciousness would. After hearing that the crew of the spaceship
HAL is on plans to kill him, HAL decides to lock the remaining members of the crew outside of
the spaceship. When the person locked outside of the spaceship first asks HAL to open the
docking bay, HAL makes no response and instead just listens to his protests. The pilot asks
HAL numerous times to open the door, each time with more anxiety, fear, and frustration in his
voice, but HAL only responds once he feels that he satisfied his sense of, what appears to be,
revenge. HAL then goes through the process of telling the pilot exactly why he decided to kill
the crew before leaving the pilot to his own ends by saying, “This conversation can hold no
purpose anymore. Goodbye (Stanley Kubrick, 2001: A Space Odyssey).” The problem lies in
that HAL never had to tell the pilot any of this. When the pilot tried to talk to HAL there was no
logical reason for HAL to respond. Therefore, HAL must have responded for some other reason
other than simple logical reasoning. HAL had to have some conscious, emotional purpose such
as duty towards his friend, the pilot, or to satisfy his revenge.
HAL clearly shows complex emotions that we would not normally attribute to a
completely computational system. Though HAL does not act in a way that people would
Baldwin-Brown 7
consider morally right (sacrificing the lives of the crew for HAL's preservation), HAL seems to
show traits of genuine consciousness, showing that an AI could eventually have consciousness if
it could perform enough simple computations fast enough. Yet again, this is another science
fiction work that supports Turing's ideas and not Penrose's or Searle's because without giving
HAL human like emotions he would not be a very imposing villain. If HAL had done everything
completely because of logical processes and was not conscious, then the audience could not
blame HAL for his actions. It does not make sense to treat a non-conscious entity as a villain
because we only hold an individual responsible for his bad deeds if they were done with
conscious intent. Someone who has accidentally killed another person is not considered evil,
while someone who has planned out the action and then consciously performed it is considered a
villain. HAL (as the villain) must therefore be conscious to act as a good villain for the story.
Philip K. Dick's novel Blade Runner (Do Androids Dream of Electric Sheep) is the only
book that even approaches supporting Penrose or Searle in the idea that computers could not
duplicate human consciousness. In the book androids are not allowed to be on Earth, and so
bounty hunters are hired to find androids that are hiding amongst the populace. The bounty
hunters distinguish androids from actual people by testing whether the subject shows empathetic
responses given certain questions. If the subject can feel empathy, then they are said to have
human consciousness. This seems to support Penrose in that some aspect of human
consciousness (empathy) could not be shown by a purely computational system like an android.
The androids also show emotions that we would only attribute to human consciousness though
such as love and compassion. The book tries to show that even though these robots cannot
Baldwin-Brown 8
produce a consciousness identical to that of humans, they are still conscious beings and are in
most respects indistinguishable from humans.
Dick gives much more evidence for Turing's side than Penrose's though. One aspect of
the futuristic world Dick constructs is that real animals are prized possessions. There are plenty
of robotic animals as well, but they are not nearly as valuable as the real thing. The problem is
that it is very difficult to differentiate the robotic animals from the real ones. A real cat is
accidentally sent to an electrical animal repair shop in the novel. The main character picks up an
electric frog thinking that he has a real (and incredibly valuable) frog. The two seem to be
indistinguishable in almost all respects, showing that Turing's evaluation of consciousness by
behavior may be more accurate than a test for empathy. The main character also feels empathy
towards the androids. If this is the case, then the androids must duplicate human emotion very
well, almost to the point where their behavior is indistinguishable. By Turing's evaluation we
would then have to call the androids conscious (which the main character seems to do). Though
Penrose's ideas are discussed towards the beginning of the novel, Turing's theory of AI seems to
be given much more credit by Philip K. Dick. This is one of the few pieces of science fiction
I've seen that seriously considers ideas like Penrose's and Searle's, that an AI may never be able
to achieve consciousness, even though it is not discussed at great length.
Perhaps the most influential piece of science fiction for our generation though would be
the film The Matrix, created by the Wachowski brothers. Though the film discusses many topics
such as Descartes' idea that we cannot know we are not in a simulated world and the idea, as
posed by quantum mechanics, that our consciousness can effect our surroundings (our world is
Baldwin-Brown 9
made by how we choose to perceive it). However, neuroscience (downloading information into
your brain) and AI are discussed at the greatest length. The way the character Agent Smith is
portrayed is the most representative of how the Wachowski brothers portray complex AI as being
conscious.
“Agents” are programs within a simulated world called “The Matrix” that we all live in.
The agents monitor this dream world that all the human inhabitants live in to make sure none of
the humans hacking into this dream world from the outside interfere with the plans of the robots
controlling The Matrix. These agents are portrayed at the beginning as nothing but computer
programs designed to be harsh peace keepers. They act rationally and coldly, showing near to no
emotion or anything other than the most logical of responses. Later on in the movie though, a
specific agent (Agent Smith) begins to act differently from the others. He was like all the other
agents (cold and rational) until the scene where he interrogates a hacker named Morpheus that
has been captured. Agent Smith becomes not only frustrated, but out right furious at Morpheus
for not complying with his demands.
This kind of reaction is completely illogical. When torturing a subject for information, it
is best not to show any kind of frustration at their responses. If you do, you only give the
tortured person a sense of triumph at having made you angry. The torturer is trying to break
down the self-esteem of the subject, not fuel it (which Agent Smith is doing). If Agent Smith
was really trying to fulfill the goal of breaking down Morpheus' self-esteem, he would not
become angry because it only gives the tortured subject a sense of triumph at having at least a
little power over their torturer. By becoming angry at Morpheus it would only boost Morpheus'
self-esteem, making the information Agent Smith is trying to uncover harder to reach rather than
Baldwin-Brown 10
easier.
Agent Smith acts illogically (perturbing the other agents he works with) and begins to
show what appears to be genuine hatred and revenge. A completely computational system (a
computer program in fact) is shown as a conscious entity in The Matrix, and so the Wachowski
brothers give the impression that a computer system could become conscious if it were complex
enough. Agent Smith is made to be a conscious being for the same reason that HAL was made a
conscious being. In order to be a villain you must consciously do an evil act, and so those that do
not consciously do evil do not make interesting villains.
So science fiction, in general, has varying opinions as to whether complex AI would
develop moral standards similar to our own, but they all seem to support the possibility that an
AI with consciousness could be constructed. This is not because problems posed by people like
Penrose and Searle are thought to be trivial by science fiction writers, I think it's simply that it's
difficult to have entertaining characters with AI that don't have human like emotions. No one
wants to see characters like HAL or Agent Smith act completely like a computer because then
they would not be interesting to watch. Part of the reason movies continue to be entertaining is
because of the varying ways a writer can develop their characters, and character development is
nearly impossible in characters that don't portray some part of human consciousness (like
emotions). Simply because fiction is designed to entertain its audience through the characters
and situations, ideas not in support of an AI having consciousness are difficult for science fiction
writers to incorporate. Moral of the story: science fiction is great, but there's more to
understanding the world than through science fiction alone.Bibliography
Baldwin-Brown 11
2001: a Space Odyssey. Dir. Stanley Kubrick. DVD. Metro-Goldwyn-Mayer, 1968.
Asimov, Isaac. I, Robot. Gnome P, 1950.
Dick, Philip K. Blade Runner (Do Androids Dream of Electric Sheep). New York: Ballantine
Books, 1968.
Fletcher, Evan. "Lecture 15." UC Davis, California.
Fletcher, Evan. "Lecture 16." UC Davis, California.
The Matrix. Dir. Larry and Andy Wachowski. Perf. Keanu Reeves, Lawrence Fishburne, CarrieAnn Moss, Hugo Weaving. DVD. Warner Bros., 1999.
"Three Laws of Robotics." Wikipedia. 10 Dec. 2006 <http://www.wikipedia.org/>.