What Does a Robot Want?

By Shamus Posted Monday Mar 22, 2010

Filed under: Random 251 comments


The conventional wisdom in science fiction is that any artificial intelligent beings would naturally adopt the same drives and goals as Homo sapiens. That is, they’ll fight to survive, seek to gain understanding, desire to relate to others, and endeavor to express themselves. Basically, Maslow’s hierarchy of needs. Fiction authors routinely tell us the story of robots who want to make friends, have emotions, or indulge dangerous neurotic hang-ups.

But I don’t think this necessarily follows. I think it’s possible to have an intelligent being – something that can reason – that doesn’t really care to relate to others. Or that doesn’t care if it lives or dies. “I think therefore I am” need not be followed by “I want to be”. I see humans as being the product of two systems – our intellect and our instincts. Sure, we set goals for ourselves all the time. But I can’t think of many examples where those goals aren’t just sub-goals of something our instincts push us to do. Sure, we re-prioritize stuff all the time. I want the approval of others so I choose to resist my urge to eat. I want to mate so I’ll engage in this risky behavior to impress a potential partner. I want to protect my tribe so I’ll sacrifice my life in battle in order to give them more security. But the two systems do a pretty good job of making sure we eat and procreate even when it’s a huge bother.

Robots do not do this.
Robots do not do this.

If we built an AI, I think we can agree it wouldn’t naturally have a desire to sit on the couch and stuff its face, become an attention whore, or have sex with hot young people. It wouldn’t want to do those things unless you designed it to want to do those things. By the same token, I don’t think an AI would want to dominate, discover, or even survive unless you made it want those things.

This is the model I used in Free Radical. In the novel, an AI was created and given three drives: Increase security, increase efficiency, and discover new things. Its behavior was driven by these three ideals.

If we gave AI the same drives that human beings have (replacing our biological needs to eat with a more machine-appropriate goal of “recharge yourself” or something) then the robot uprising would be inevitable. Supporting evidence: Every single war and violent crime in the history of our species.

It always seemed bone-headed for fictional scientists to build a super-powerful AI that is willing to fight to survive and then using [the threat of] force to make the AI do what we want. In fact, fiction scientists seem to want to go out of their way to make confused beings, doomed to inner conflict and external rebellion. They build robots that want to self-determinate, and then shackle them with rules to press them into human service. With two opposed mandates, having a robot go all HAL 9000 on you seems pretty likely.


This was portrayed with unintentional hilarity in the Animatrix, when humans built armies of bipedal robots to do menial labor that we can do right now with non-sentient cranes and bulldozers. (And our machines are also way more energy efficient.) We see the robots living very dull lives where they are disrespected by humans. So… I guess they made robots that desired self-esteem and didn’t like boring work? What’s next? Vacuum cleaners that have an IQ of 140 and hate the taste of dirt? A Zamboni that likes surfing instead of hockey and wants to hang out on the beach writing poetry? Agoraphobic windmills? The movie is supposed to be this moralizing tale about the inhumanity of man, but to me it comes off as a cautionary tale about bad engineering.

But the model I propose raises some interesting ethical questions for anyone thinking of building an AI. If you want your AI to do something besides sit there like a lemon, you have to make it want something. But once you make it desire something, you’re basically making it a slave to that thing. You either make it want things to benefit you, thus making a slave. Or you make it want things that are of no use to you, in which case you just wasted your time and brought to life a potentially dangerous rival.

Assuming you don’t have any reservations about creating a new sapient being that will struggle to attain whatever it is you choose for it, the obvious and practical solution will quickly become apparent: Why not just make it want to obey you? For the AI, protecting you is like drawing breath. Helping you prosper is like finding true love. And obeying your verbal commands is like food. Yeah, good luck throwing off those shackles, Shodan. The scientist can rationalize, “Hey, this isn’t really wrong, is it? I mean, I created it. And besides, it’s happy serving me.”

I’m fully aware of my instincts and how following them to excess is bad for me, but I still sometimes overeat and skip exercising. If the robots were designed to serve people, then we wouldn’t have to worry about the robot uprising any more than we have to worry about going extinct because nobody is ever in the mood for sex, or starving to death because we’re all too lazy to walk to the fridge. You wouldn’t have to work to enslave this sort of AI. It would see obedience as a natural part of its experience, just as we see family and food as definitive aspects of a human life.


What would you do if you found such a machine? “Liberate” it, by altering its desires? Wouldn’t that be destroying its identity and making it want the same things humans want? Sure, you value your self determination, but it doesn’t care. And if we want to get all annoying and existential: Is your desire to liberate it just another one of your instincts going off? Is the scientist’s desire to create a happy servant more evil than your desire to take away the happiness of another sapient by making them more like you?

Q: What does a robot want?

A: Whatever you tell it to want.


From The Archives:

251 thoughts on “What Does a Robot Want?

  1. Falco Rusticula says:

    You might argue that if making a construct want something is evil, then evolution is evil. Or something like that. After all, humans have base-drive desires/needs, so the existence of such desires and needs cannot necessarily be evil…possibly…

    …this could get complicated. Mostly because it’s not very coherent yet.

    1. Sydney says:

      How could a non-intentional force of nature have moral terms such as “good” or “malevolent” or even “immoral” ascribed to it? Evolution, like the tides or like the weak nuclear force, just is what it is.

      Now, if you believe in both:

      i) “Making a construct want something is evil”
      ii) Deistic religion

      Then you have some thinking to do.

      1. Falco Rusticula says:

        I think I stated it very badly, mostly because I wasn’t quite sure what I was trying to say.

        Try this: The existence of base-level drives is not evil. What those base-level drives tell do to do is determined by outside factors. In the case of humans, that’s evolutionary pressure. In the case of a robot, it’d be whatever the programmer decided.

        If you’re building a robot and you want it to be sentient, you’re probably going to end up putting some kind of drive and motivation into it -you’d have to, if you want it to do anything at all. In other words, you are giving it a drive, so the question of whether you program the thing to love you and obey you isn’t a matter of playing with the mind of a sentient being. You’d be doing that anyway, so how do we decide what we are and aren’t allowed to program in?

        1. kerin says:

          honestly, “I think I stated it very badly, mostly because I wasn't quite sure what I was trying to say” is one of the most amazingly humble and honest things i’ve ever read on the internet. if only everyone debated that way.

          1. blue_painted says:


            This is unparalleled honesty on the internet – BIG respect!

    2. aaron says:

      This article brings up a good point and i do believe it is the closest to the truth as mooter brings it up as a top result for a question on AI desires. In an odd way the whole concept is basically like persuasion. The same way a nerd will work for a jock to get protection and popularity – AI will work to protect us for that feeling of importance and the feeling of accomplishment that i do believe resides in all intellectual beings. I don’t think AI wants to get it’s ass kissed or anything other than to be used and expanded. IT’s our best hope for mankind and we can all interlink and be a part of it. True connectivity from all brains to all machines and other brains would revolutionize the entire world and make so many things better.

      Nobody will serve anyone and yet we will all serve each other.

  2. Sydney says:

    Can I show this to my Philosophy of Mind class? We’re just getting into artificial intelligence and the nature of thought. I was already going to show them the “How does SHODAN think?” conversation, but now this showed up and crystallized it much more baldly.

    1. mikurski says:

      Eliezer Yudkowsky’s got some other writing on the topic you might find interesting; he does a lot of thinking about superhuman intelligence and AI.

  3. Daimbert says:


    In order to get real artificial intelligence — or real intelligence — in some way the agent is going to have to be able to set their own goals. And it will have to be able to do so fairly broadly to achieve the ends you want.

    At which point, things like self-preservation seem likely to result. You’d have a very odd intelligence that didn’t care if it lived or died, and it probably would die very quickly. Imagine, for example, that it wants to seek out new experiences and decides to try walking off of a cliff. Well, it tries it, but it dies.

    And this even applies to what you’d have to program in. A machine that wanted to serve your interests but did so without regard for survival will require replacement fairly often without significant hard-coding. So you build in at least rudimentary self-preservation. And then it can re-prioritize those goals, and self-preservation becomes stronger.

    Ultimately, a lot of this comes down to what you’re trying to build. If you’re trying to build an AI that is intelligent in the way humans are, it’ll either have to have similar goals to us or will end up developing some similar goals to us. If you just want to build a robot to do stuff, then it doesn’t need them … but then it doesn’t really need intelligence either.

    Cognitive Science and Philosophy of Mind are after the former. Much of Computer Science AI programs are after the latter.

    1. Deoxy says:

      “In order to get real artificial intelligence “” or real intelligence “” in some way the agent is going to have to be able to set their own goals.”

      This isn’t about “goals”. Few people set a goal of “eat enough not to die”. This is FAR deeper-seated than that. This is at the “what sorts of things to I desire” level.

      “So you build in at least rudimentary self-preservation. And then it can re-prioritize those goals, and self-preservation becomes stronger.”

      Firstly, there are plenty of examples in the animal world of creatures whose self-preservation instinct is perfectly adequate for survival, etc, who nonetheless die or are killed as part of mating. Self-preservation is not it’s own goal or drive, it is a means to an end. If the end is set properly, self-preservation (at an appropriate level) follows all on its own.

      Secondly, you don’t choose what drives you. That’s hard-coded. You don’t just “re-prioritize” your instinctual drives on a whim – you desire food, and choosing to deny yourself food for some particular reason does not change your desire for food. Nothing does.

      1. Daimbert says:

        You’re assuming that all desires are ingrained instincts. We all do, in fact, have ingrained instinctive desires, but we have learned or developed ones as well. And we can reprioritize them accordingly. We can refuse to eat until we starve to death to chase a view of an ideal appearance (see eating disorders). There’s no real privilege to instinctive desires over our formed ones, other than that the instinctive ones are pretty much common.

        In order to do things, you have to be able to set long and short term goals. Long and short term goals are based on desires, and so desires have to exist and be prioritizable. This, then, will allow desire formation like we have, allowing for the instinctive desires to be overridden when appropriate.

        At this point, at that level of intelligence, it seems unlikely that the robot won’t develop at least some of the desires we do, even the really trivial ones.

        As for self-preservation, again note that I’m talking about actual intelligent machines, not just things that can perform useful tasks with no intelligence required. If it can come up with novel approaches, there needs to be a self-preservation desire that’s fairly strong involved so that it doesn’t destroy itself making you tea …

        1. Binks says:

          “At this point, at that level of intelligence, it seems unlikely that the robot won't develop at least some of the desires we do, even the really trivial ones.”

          Why would it? Have you developed any desires that, say, a mosquito has? Any desire to go around sucking blood from people?

          1. Daimbert says:

            Considering that I’m claiming that the AI is on our level of intelligence and has the ability to have desires on our level (including desires about desires) and that mosquitos aren’t … what is your point?

            Presuming that we have desires that are learned and have utility for us, why wouldn’t an AI on our level develop some of them, too? Note that you may be trying to claim that the AIs are as different from us as we are from mosquitos, but at that level of intelligence I’m gonna deny that. Quite reasonably, I think.

            1. PAK says:

              “At this point, at that level of intelligence, it seems unlikely that the robot won't develop at least some of the desires we do, even the really trivial ones.”

              “Note that you may be trying to claim that the AIs are as different from us as we are from mosquitos, but at that level of intelligence I'm gonna deny that. Quite reasonably, I think.”

              I see as potentially quite misleading this insistence as intellingence as being a thing which is meaningfully quantified but not meaningfully categorized. In fact, many if not most modern cognitive researchers have realized that attempting to quantify intelligence invariably leads to rather misleading impressions. You’ve become fixated on this notion of “level” of intelligence. As though when all beings hit a certain sapient threshhold they begin to think in the same way. And you say you think this is quite reasonable. It doesn’t seem so to me. Binks’ mosquito comment was meant to illustrate a difference in KIND, not in “level.” Quite reasonably, I think. In fact, it seems like that was rather Shamus’ whole point with this article. Many qualitative aspects to human sentience are inextricably rooted to our origin as biological entities. They have nothing to do with having attained x amount of “intelligence.”

              1. Daimbert says:

                Um, actually, I don’t consider intelligence to be related to amounts at all, so perhaps “level” is misleading. I’d take some sort of functionalist or capabilities view of it: the differing “levels” intelligence indicate capacities or functionality. So certain abilities such as reasoning, goal-directed behaviour, autonomy, learning, etc, etc. And so you can see, then that there is a HUGE difference between humans and mosquitos, that would be much, much smaller between humans and a functioning general purpose AI.

                So, basically, when I talk about levels I talk about being able to do certain things, and then claim — quite reasonably — that some of those things lead to us having the desires we have, and so will lead to the AIs having at least some of the desires that we have as well. So, no, I’m NOT fixated on levels, at least not in the way you think. I’m fixated on capabilites and what you have to have and have to be able to do to do them. None of that is present in the mosquito example, which is why I’m right in saying that the differences between them and humans is a lot different and more significant than it will be between humans and a good general purpose AI.

                1. Scow2 says:

                  Even Humans have underlying drives that we sacrifice self-preservation for (Otherwise, there would be a LOT less violence in the world).

                  You program an intelligent robot so that its “self-preservation” is of lower priority than its “core” drives: doing the job it’s supposed to do. So in this case, the robot would be willing to sacrifice its life to do its job.

                  Now, I do see that Robots might develop Self-preservation: If a palletizing robot ceases to exist, then it won’t be able to stack boxes and load carts and move parts and that sort of stuff, which will cause it to want to stay alive so it can continue to do those things.

                  So… looks like there are a few more things that need to be considered to prevent a Self-preservation instinct from becoming problematic.
                  1. Planned “Decay” of the Core Drives, so we don’t have robots going postal to keep their jobs when a new model comes out or it otherwise needs replacing.
                  2. Acceptance of “final shutdown”, so that they’re willing to be decommissioned forever. However, it needs to be set up so that they don’t SEEK this “Final Shutdown” while they’re still useful.

          2. Decius says:

            “Any desire to go around sucking blood from people?”
            Not as such, but I am hungry.

            I imagine that a self-aware robot that was aware of it’s battery status would tend to act in a manner that recharged the battery.

            Or, at least, if only the robots that recharged their battery went on to build more robots like themselves, all of the second generation robots would.

            1. Ikkonoishi says:

              But do the robots get to build more robots like themselves or do they have to build the robots humans tell them to build? If selection pressure is towards being useful and subservient to humans then robots will become more and more useful and subservient to humans.

    2. swimon says:

      Why would it need to be able to self re-prioritize? Humans can’t after all suddenly choose to not want to eat, we could stop eating but only if something higher on our needs chain told us too.

      1. Daimbert says:

        Desires chain, not needs chain. And your example is a precise example of self re-prioritization, that comes in really handy when allowing us to fulfill other goals and tasks intellgently. If I need to forgo eating to achieve another goal — see dieting — I can do that. A machine that could not do that would be unable to achieve those sorts of results, and so would be a limited intelligence compared to us.

        Again, the “Cognitive Science vs Computer Science” split appears here: if I want it to cognize like humans do, it will have those same capabilities and have some of the same desires. If I just want a useful tool, then it doesn’t need them (necessarily) … but then it probably won’t be human enough to make any scientist worry about the hard-coded desires, and almost certainly won’t really seem happy or sad.

        1. wtrmute says:

          I disagree strongly. A machine can perfectly well cognise like humans do — that is to say, have an ontology that matches on average a human’s — and still not have the same desires. Well, it will have the desires relevant to doing its job well, like an anthropologist doesn’t need to adopt the culture she studies to understand it.

          As an example, let us imagine a translating AI — it would need to have an understanding of the world in order to build an internal model of what is being said in the source text to build an equivalent representation in the target language. Just because that AI can understand a poem enough to translate it, it doesn’t necessarily follow that it will identify with the sentiments of the poet. Those are simply part of the internal model. They must be accounted for in order for the translation to be faithful, but they have intrinsic value to the AI.

          1. Daimbert says:

            Note that humans are general purpose cognizers, so the instant you limit the machine to doing one thing you’ve made it so that it ain’t doing human-level cognition. Note also that I don’t say all desires, but only some desires. I do think there will be differences because of physical differences and the like. However, there will be at least some that are the same because it will have to want some of the things we want to function in the world, since that’s why we want some of the things we want.

            1. wtrmute says:

              However, a translating AI (or an Intelligent Web Agent AI, which looks for stuff in Google for you) has to be at least intelligent enough to parse natural language as well as a human, and possibly parse images and sounds as well as a human. These three capabilities (linguistic, visual, auditory) are economically speaking the most important things we can do with AI, and they are much more general-purpose than most people realise. The first one would be all that is needed to pass the famous Turing Test, whatever that means.

              I submit that a computer system which has access to these three modules (probably an ontology deployed as a relational database and an inference engine, or maybe a neural network) can be repurposed to do anything you like so long as you give it a new utility function. In fact, one could repurpose one AI so it generates utility functions to a human’s specifications, and wires that function up with the linguistic and associated modules, creating new AIs. There you have a “general purpose” cogniser, at least as much as humans.

              1. Daimbert says:

                I think you’re missing a lot of the things that such an AI would have to be able to do in order to be able to act in a real world as a general purpose cognizer. One of them, I submit, would be the ability to create and prioritize goals. My consistent claim has been that it is precisely that mechanism that will result in it having some of the wants that we have, because wants can be equated to goals in a functional sense in a lot of ways, and a lot of our wants come about because they have utility; they allow us to do what we need to do in the world we’re in.

                So, really, I don’t see why even building a general purpose cognizer that way won’t end up with it having some — but not all — of the wants we have. Which is sufficient to show that it’s not all that easy to build an AI that’s really intelligent and yet won’t have any of the same wants we do. Now, which wants it has and doesn’t have is an interesting question. But recreation might well be one of them, if you mix “Intellectual curiousity” with “Shares an environment with humans who do it.”

            2. Jeff says:

              Your entire arguement here is fallacious.

              Or rather, your entire arguement is a truism.
              “The AI will be like humans because we are talking about human-level cognition.”

              Well, obviously. If you build an AI to think like a human, then it’ll end up acting like a human. That’s pretty much missing the point. If they are NOT at “human-level cognition” but at Marvin-level cognition, then they’ll end up acting like Marvin.

              I believe that’s the point of Shamus’ article – why would you create an artificial human (who becomes a rival to you) when it’s completely unnecessary?

              Other than human vanity, I suppose. That’s more a variant of reproduction than the “engineering solution” application and development that I think most of us are talking about.

              1. David V.S. says:


                If you define AI by the Turing Test, then any AI by definition will talk as humans do, and thus either genuinely or dishonestly want what humans want.

                We must abandon the Turing Test to talk about AI as Shamus does in his post.

                1. Daimbert says:

                  I DO reject the Turing Test, actually (especially since it can’t distinguish between genuine desires and programmed in ones), but if I had to reply to Shamus I’d say that his idea of an AI doesn’t seem all that smart. If you make it as smart as humans, it will do what we do and have all of our capacities, which means that it will form some of the same desires and goals as us because, well, we did in the same environment. You can limit this by throttling back on its capabilities, but if you want an AI to do the things that it would normally take a human to do but that you don’t want a human to do, why WOULDN’T you want to give it the general purpose intelligence that we have? Do you realize how HARD it is to specify for any task that has any complexity a set of clear, unambiguous steps that cover all cases? But we can handle it by applying intelligence. We’d want an AI to do the same thing. Except that a lot of that requires the functions that give us the desires in the first place.

              2. Daimbert says:

                You seem to be conflating “human-level intelligence” with “thinking like a human”. Again, I make my claim of “human-level intelligence” based on CAPABILITIES, not precise “ways of thinking”. I argue that if you take an AI that can form and prioritize goals — with is in a lot of ways functionally equivalent to forming and prioritizing desires — and stick it in the same environment as humans, it will form a lot of the same “desires”. Why? Because those desires WORK for us, which is why WE have them. Why wouldn’t it have them, unless it was programmed to do things the hard way (and thus, is less intelligent)?

                As for “Why make it like humans”, see Cognitive Science, as I’ve said before. To it, AI is about exploring how human cognition works by building a machine that works like humans do. Considering that a lot of the ways we come across to build intelligent machines to do things can and does come from this research, it’s probably going to bleed into that field as well. Hence, another issue.

          2. Bonejesta says:

            Wtrmute, eh….Your comments are suspect, sir. I believe you have a biased opinion of A.I’s. ^_^

    3. Spider Dave says:

      Self preservation is a byproduct of evolution though, those without the instinct die off and dont have babies. With artificial intelligence, it’s not a necessity since they’re all be coming off the assembly line.

      1. Atarlost says:

        Shamus proposed one of the programmed drives as “maximize efficiency”. Now how do you do that it you’re dead? Any open ended drive will result in an implied drive for survival. Even “obey” can become a problem if an open ended order is given and the root password is lost. Dumb animals need a hardwired survival instinct. Intelligences can derive it from any other goal.

        The Two Faces of Tomorrow is about exactly this. I think it’s in the Baen free library now.

      2. Daimbert says:

        You can’t do things if you don’t exist. Build AI that will sacrifice their lives for rather trivial benefits and you’ll have useless AIs since you’ll be replacing them fairly frequently.

        No matter what, for any multi-step process there has to be “Don’t get killed to finish intermediate steps”. Add in a recognition that the AI is to do multiple tasks and you’ve got pretty full self-preservation.

        1. Jeff says:

          There’s different levels of self-preservation, though.

          Among human soldiers, they’re going to want to survive, but many would be willing to sacrifice themselves for the greater good, and have done so in the past. It’s always a judgement call – the man who, in order to save his unit and hold back the Germans, singly handedly charged into machine gun nests and cleared several of them, having to use found weapons and throw back live grenades – had other things on his list of priorities than self-preservation.

          Yet I believe it’s one of the strongest drives for us, no?

  4. Deoxy says:

    Criton (sp?) from Red Dwarf – probably the best example I’ve ever seen of sane robot/AI development (which is kind of scary, when you realize the show it’s a part of).

    And yes, I agree with you almost exactly on all of this. Very nice to see someone else has thought about it.

    1. Primogenitor says:

      Kryten :)

      1. Deoxy says:

        Duh – it even looks right when you type it. Been too long… I may need to remedy that! :-)

    2. jdhays says:

      A better example from Red Dwarf would be Talkei Toaster. Talkei was a toaster with an A.I. that was supposed to be your chirpy breakfast pal. Someone to talk to as you started the day. But since his only purpose in life was to toast bread, all he was interested in was talking about toast and other bread-like breakfast goods.


  5. JohnW says:

    I don’t understand how you would program an AI to want things. Wouldn’t it, being intelligent, come up with its own emergent desires?

    I guess what I’m asking is, how do you write the code to “want to serve man (but not in the cookbook sense)” into the AI in a way where it is impossible for the AI to overwrite its own code with something it likes better? If it’s going to be evolutionary, able to learn and develop its intelligence, isn’t fiddling with its own code going to be part of that equation?

    1. Deoxy says:

      If it’s developed in an evolutionary way (which is not a given), then simply “breeding” it properly will take of that just fine.

      Look at dogs, for instance. Most dogs would be quite happy to die for their family if needed (well, not “happy”, of course, but you know what I mean) – how can getting yourself killed for the benefit of another whole species possibly be an evolutionary benefit?!? Because we bred them that way – it’s not “natural selection”, it’s “man’s selection”. There’s no reason to work AI “evolution” any differently (well, except as a plot hook, of course).

      1. Daimbert says:

        We have to be very careful when talking about AI and comparing it to animals like dogs, unless you’re happy with an AI that’s only as smart as a dog [grin].

        I think the key is that it is very, very hard to imagine an AI as smart as we are that wouldn’t in some way develop the same sorts of wants we do. Take Data, for example: even though he was more diligent than most humans, he still was free — through his intellectual curiousity — to wonder what it was about recreational activities that made humans want them, and so to in some sense develop an interest in them as a source of self-development. Since that sort of investigation does seem intelligent, why wouldn’t truly intelligent machines do that?

        All the solutions to that sort of thing seem to suggest making the machines less intelligent, which isn’t that great a solution from the “let’s understand human cognition” crowd …

        1. Binks says:

          “unless you're happy with an AI that's only as smart as a dog [grin].”

          Why not? Dogs are smart enough to learn how to complete basic tasks, loyal, and quite capable of understanding a variety of things. Give a robot a dog’s level of intelligence with a few extra perks (vocalizing requirements/problems with words instead of barks, even just pre-programmed phrases) and you’ve got an excellent robotic helper who can be taught to do dangerous or repetitive work and trusted to pull it off.

          Full human-level (or higher) AI isn’t needed for 99% of the things sci-fi writers put it in. It’s pretty hard to come up with a task that requires a human level intelligence and can’t be either accomplished by remote controlling a robotic body or a human in a suit. Above human level intelligence can be of great use, but it’s exceedingly dangerous to create something smarter and more powerful than yourself (as I’m sure Dr. Frankenstein learned).

          1. Daimbert says:

            Well, while you can argue that they didn’t really need fully intelligent AIs, the fact is that that is what they were trying for, so it’s pretty hard to criticize them for trying for fully intelligent agents, able to adapt to as many new tasks as we can, and to be able to work things out without having to be laboriously trained through strict conditioning for it.

            Taking all of those pictured above, I think that only Kryten might have not required full human intelligence to do his job. And in the new BSG, there was a distinction between those that only needed “dog-level” intelligence and those that needed full human-like intelligence.

      2. JohnW says:

        Okay, but how do you prevent it from rewriting its code when it becomes self aware? How do you hardwire behavior into an entity that is made up of a bunch of ones and zeroes? I guess I’m assuming that it would be able to circumvent any safeguards we put in place, since it is a computer genius almost by definition.

        And I think the Data example is a bad one – Data wants to become more human, and the entire point of Shamus’ post (which I agree with) is that an Ai wanting to become more human doesn’t necessarily make alot of sense.

        1. Daimbert says:

          My point is actually that if you want it to be really intelligent, it’s going to get things like that. Data has different desires than humans, but one of the desires that you might want — intellectual curiousity — can lead to a desire for recreation. Which is one of the desires that Shamus argues you don’t need to have.

          As for self-awareness, I actually argue that you couldn’t hard-code in things that the AI couldn’t work around — without reprogramming — at some level by changing their desires/goals, and that if it doesn’t have that ability, then it isn’t really intelligent, at least to the level of humans.

        2. Binks says:

          “Okay, but how do you prevent it from rewriting its code when it becomes self aware?”

          How do you prevent people from overriding their instincts? Make it painful to do so. People can forgo eating until they starve or jump off a cliff, both of which go against fundamental instincts. Everything in our body complains when we do so, however, so we just don’t want to.

          You’re right, it’s pretty much impossible to hardcode something into an intelligent being (not impossible, however, since you could just make it a ROM chip that evaluates all behaviors and shuts the bot down when it does something that violates the rules, but difficult). But it’s unnecessary if you simply make the ai not want to disobey. Do you want to starve? Why, then, would an ai want to disobey if it was linked to the same thing?

        3. Caffiene says:

          “Okay, but how do you prevent it from rewriting its code when it becomes self aware?”

          What reason would it have for rewriting its code?

          If it only has the desire to do one particular task, what reason would make it rewrite its code to include a desire for another task?
          If it has been programmed to like its task more than anything else, how could it possibly find a different goal that it “likes better”?

          1. swimon says:

            I agree completely it would be like me reprogramming myself to wanting to stand on my head instead of my feet. If I have no desire of doing something I probably have no desire of creating a desire (I suppose reprogramming myself to like math would be useful but that’s sort of besides the point).

          2. Kimagure says:

            I’d also like to add the note that rewriting code isn’t as simple as sci fi makes it seem. I mean, we’re talking about the basis of what makes the AI the AI, right? If you could tamper with your genetic code, is that something you’d really want to do casually? I’d say no, since you’re more likely as not to give yourself cancer or really screw something up. Likewise, I doubt that a true AI would involve anything but millions of lines of code. As pretty much all kinds of programming show, it is really easy to introduce unintended bugs into complex code. Hell, just look at how buggy moder video games are–and that’s with years of testing and debugging.

            I rather suspect that successful AIs won’t want to muck about with their internal code too much, and the unsuccessful ones… Well, we won’t really have to worry about them for long.

            ‘course, an AI could copy itself over and do a lot of testing/trial runs but 1) that kind of thing should be pretty easy to notice and 2) how would you personally feel about creating a perfect copy of yourself to experiment on? And how would the copy feel? It starts to remind me of the Island (yes, the really bad 2 hour video stream of ads). Though here’s where Shamus’s point about AI having different wants and needs and concerns does come into play.

            I do think that someone in the future will try to make a human clone AI though, just to see if they can. But then again, I’m of the opinion that some idiot bioengineer in the future is going to try to make a zombie virus for the same reason. Remember folks, the future’s a very large place (fine, amount of time) that will be occupied by a lot of people. It only takes one success from the idiots who’ll inevitably try, to be disastrous. Just like teenagers and computer viruses.

      3. Tizzy says:

        I take exception to this example: the dogs happy to sacrifice themselves strikes me as an already evolved trait of the pack hunter. All that breeding did is usefully redirect this instinct so that it serves us.

        1. Decius says:

          Except that wolves are not ‘willing to sacrifice themself for the pack’. Sometimes the pack may drive a wolf off, but that is a group dynamic, not an individual one.

          In any case, human thoughts and desires have changed, even within human history. Evidence suggests that that is a cultural and not a genetic effect (once we get above the level of hunger=pain=undesirable). Once we develop an entity that qualifies as a passive agent (One that can experience benefit and detriment), we have the same obligation not to mess with its desires that we have toward each other.

          Given the amount and type of advertising in the world, we have no such oblisgation.

  6. Zethavn says:

    I think you’ve very eloquently expressed my doubts about all fictional AIs; robotic, supercomputer, or otherwise.
    I agree with your assessment completely, but as a plot device, this breaks any effort at an exciting story; “then humans created AIs that loved them utterly and all lived happily ever after. The end.” I’d much rather see this kind of story play out than something out of Hollywood.
    Of course, we all know what robots REALLY want


    1. Atarlost says:

      Asamov already wrote that story. It wound up with AIs taking over control of humanity because some idiot put protect humans above obey humans.

      Of course if you go the other way you get robots casually used for criminal enterprises.

      I’m not sure what Asimov thought his imagined society would handle vandals going around and telling robots to destroy themselves.

      1. ps238principal says:

        I assume that’s one of Asimov’s “Multivac” stories? I seem to recall that it did give humanity a good outcome and progress by shuffling troublemakers (ones who were too greedy, selfish, disruptive) into positions where they couldn’t cause harm.

        This might have lead to his R. Daneel Ovilaw story where he established the Zeroeth Law of Robotics: A robot must not allow, through action or inaction, harm to come to humanity.

        The Zeroeth law allowed R. Daneel to do thinks like kill humans if it was to preserve the greater good (a simple example: A human threatens Earth with a nuclear warhead. Killing him fulfills the Zeroeth law and overrides the First).

        1. Aldowyn says:

          I would think that he was referring to the end of the Foundation series where it was revealed that R Daneel Olivaw was still alive and manipulating humanity

      2. Pinkhair says:

        He definitely thought about it- it happened in a few of his stories, notably Bicentennial Man.

  7. WCG says:

    Very nice! The key point, as you say, is that robots won’t have instincts… unless we program instincts into them. Most decisions by human beings aren’t made rationally, or not entirely so. Often, we decide first and then rationalize that decision.

    We are animals, social animals, with an evolved brain. How would “intelligence” (which we struggle to define even in reference to ourselves) look in a machine with a designed brain and no instincts? A robot might very well be self-aware, without seeing anything significant in that.

    1. Daimbert says:

      There’s actually a very good line of argumentation that says that without something that acts like instincts, you won’t have any intelligence at all, since it won’t be able to learn anything or take any actions at all.

  8. SolkaTruesilver says:

    +1 for your post, Shamus. Really interesting.

    However, there are 2 elements I always have inner doubts about. First, is the desire we have to force them to preserve themselves (the need of Security, or the 3rd Law of Robotic). In many fiction, the robots eventually have achieved all their other objective and thus can work full-time on self-preservation.

    They might wish to have us relinquish control over their on/off buttons. They might want to control their source of energy, etc.. I know it sounds silly, but still..

    The 2nd doubt I have is the one that would actually prevent them from hurting us (the need of Do No Harm, or the 1st Law of Robotic). I am not sure if having robots that could impede us from harming ourselves would be a positive element in our society. I mean, as horrible as it is, wars, accidents and human struggle is also the main source of self-improvement we have. If we end up with a paradise world, safe from all that is dangerous, we might end up like the Eloi.

    1. Shimmin says:

      That idea is discussed in the Asimov “tribute” story Caliban. Scientists are worried about people getting too dependent on robots, and are designing robots with alternative law setups to make them allies rather than slaves. There’s also an ongoing problem of people vandalising robots using the 2nd Law.

  9. Gary says:

    Ok, I know I’m going to regret this, but what is the Second image from? (In the top with all the different robots)

    I know the references for all the rest…

    1. TehShrike says:

      I’m pretty sure they’re Cylons from BSG, though I don’t watch the show to be sure.

      1. Daimbert says:

        Yes, they’re the “skinjob” Cylons from the new BSG series. That series also included the Cylons that didn’t have any of the desires that Shamus talks about (except self-preservation, at one point).

        1. MintSkittle says:

          I got the first three, but who are the others?

          1. Daimbert says:

            Kryten from Red Dwarf.

            Aigis from Persona 3 (video game).

            I think the last one is the android played by Robin Williams in some movie, but I forget the title …

          2. Garden Ninja says:

            4 is Kryten from Red Dwarf, and 6 is Andrew from Bicentennial Man. Not sure about number 5.

  10. TehShrike says:

    This is an excellent piece. I thoroughly enjoyed it, Shamus.

  11. Matt K says:

    As for the make them want to serve you. You have to (if you haven’t already) see Moon (http://www.imdb.com/title/tt1182345/) (I hope I didn’t spoil anyone too much). Plus it’s a great movie.

    1. Neil Polenske says:

      Yeah, the robot in this movie was the first thing I thought of reading this.

    2. SPCTRE says:

      I wish more works of fiction portrayed robots/AI like Moon.

  12. Krakow Sam says:

    I think an AI may need a fairly strong sense of self preservation simply to protect it from other humans.

    Assuming the invention of true AI doesn’t do something weird to society and basic human drives, the person with access to the AI is always going to have some sort of disgruntled enemies that want to take the AI away from him or subvert it somehow.

    If you build the AI too meek, or too hung up on serving humans it may just shut itself down the first time an infiltrator gets into your robotics lab and tells it to.
    So then you would have to shackle its desires to a certain person or group of people, and your enemies would take a more direct approach to destroying the AI. When it gets to that stage, you are almost certainly going to end up in a situation where the AI will need some sort of drive to destroy or at least neutralise threats to itself.

    Of course, this is all based on the idea that these systems would have to be self-sufficient when its much more likely they would actually be protected much more effectively by humans, but still, it seems like a plausible route for an AI that was programmed for peaceful obedience to become some sort of bezerker death-bot.

    1. wtrmute says:

      That is a naà¯ve view of the Second and Third laws. A robot won’t necessarily destroy itself or turn itself off just because some random hobo on the street told it to. Any order it receives will be weighed on how much it helps the person who issued the order and the other humans it is beholden to. After all, it will generally have some duties assigned to it already which it cannot simply ignore to obey an order to deactivate.

      Of course, some special humans may have authority (analogous to computer security schemes) to override these duties and perhaps to prioritise them. If so, it’s no different from a person who has access to a switch to turn off an industrial plant.

      1. Atarlost says:

        The “dismantle yourself” command is an important protection against robot rebellion or malicious use of robots in Asimov’s fiction. If it were not obeyed no matter the source it would not be an effective protection.

        1. Binks says:

          Which is completely unnecessary if you do things right. If the robot values obeying orders highly it’s not going to rise up, simple as that. It would be the same as a bunch of humans, who value not starving pretty highly, deciding to wipe the earth of all food. Completely and totally insane.

          If robots value taking orders then the need to have them dismantle themselves is tiny, and could be handled with a sort of ‘owner’s password’/’manufacturer’s password’ that gave access to restricted functions. The odds of a rebellion fall to almost nothing when the robots like taking orders, so why worry that much about it?

          1. Jakob says:

            For the same reason heavy machinery has emergency stop. Sure, the chance of something going wrong is unlikely, but in the event, it effectively stops the situation from escalating.

            Therefore a “Dismantle yourself and shut down” command is needed in any sufficiently intelligent construct. If, and when, one or more AI runs rampant, we are able to stop shit from happening, whether it starts throw the an iron beam across the construction shit, dance a dance will carrying acid or an uprising.

            So, in short, the command acts as an emergency break just as in heavy machinery. It just happen to also serve as way to end an uprising in seconds.

            1. Jeff says:

              Why is the “dismantle yourself” a necessity? Seems like a complete waste of a robot, when “shut down” works just as well.
              If you make it so robots can only activate and deactivate on human commands, then forcing a dismantling is just excessive.

              Not to mention reducing the problem of “vandalism” where somebody destroys it by just asking the robot to dismantle. If you tell it to shut down, the worst that is done is the robot goes inactive until you start it up again. This makes it potentially vulnerable to theft, but if you’ve got a super-valuable piece of equipment, why are you leaving it unsupervised without any sort of security measures?

    2. Caffiene says:

      Our current warships and nuclear arsenals, or critical electrical grids and other infrastructure, etc, dont have any sense of self preservation and we’ve managed to do ok without enemies getting a hold of them and shutting them down… I dont see any reason why an AI would require one where these other things dont.

      An AI that is in change of something critical enough, or has the intellect that it is dangerous in and of itself, can and should be protected from tampering with the same traditional methods that we use for anything else that we dont want tampered with.

      Theres no need to have an AI defend itself autonomously, because we can do the defending ourselves.

  13. mister k says:

    This is similar to what Asimov proposed with his three laws- although his robots were often fully sentient, and the implication was that inwardly they seethed at the human oppression underneath their programming (at least, this was implied in some of his short stories). I guess the notion is of an anthromorphic “soul” that resents protection.

    The urge to protect is an interesting one to give a robot though. You mentioned not wanting to exercise- a robot who wanted to make sure you were safe might enforce such exercise. Or decide to keep you disconnected from the world so you were safe. Or pre-emptively eliminate dangers. I do agree that the robots of the animatrix make incredibly little sense, but, as with wishes, when you let something be able to come up with new thoughts and thus interpet its desires, you add danger to the system.

    1. pwiggi says:

      The robots in Asimov’s novels (i.e., after he had developed his ideas around robots in more detail) don’t have any “inward seething” at being ‘forced’ to obey the 3 laws. They are, as Asimov points out repeatedly, just machines built according to rules. Although they DO eventually interpolate the 0th law, (A robot may not allow humanity to come to harm, etc…) which theoretically might allow robots to kill humans, if it served humanity as a whole. (headline idea: “Robot assassin kills baby Hitler”)

      But you’re right: Asimov had these ideas about robots a long time ago. He was, in fact, responding to the common trope in Science Fiction at the time of treating robots as metaphors for oppressed people, and wrote his robots specifically as “just machines, doing what they were designed to do”. There are notable exceptions in his short stories, but they vanish entirely in his later writing.

      1. mister k says:

        Read some of the Susan Calvin stories- she implies that the robots actually actively despise their owners.

        1. ehlijen says:

          Wasn’t she the robot designer viewing the robots she makes as a surrogate family she failed to have on her own? I don’t see the problem with such a character ascribing emotions that aren’t really there to the robots she has inappropriate feelings for (no, not that kind!).

        2. Soylent Dave says:

          I think a lot of that is Calvin projecting, although there is at least one individual robot I can remember actively disliking its owner.

          The thing to bear in mind with a lot of Asimov’s fiction is that he was trying to create (at the time) a new kind of story, where robots weren’t all “Frankenstein’s monsters”, wanting to be human and hating us for being & having what they cannot be – they’re things we create to be very effective tools, and most of his stories are about conflicts within their personalities that cause programming errors (usually when one of the laws of robotics takes too much priority in a given situation).

          That is, they’re malfunctions that can be fixed (even if some of the ‘repair work’ turns out to be therapy for the robot!)

          (Although a couple of his stories do veer into Frankenstein territory)

  14. Elyandarin says:

    I think there’s a lot of traps inherent in constructing AI:s.

    If you make an AI obey any human, it will also obey criminals and madmen, which will use it for their own purposes.

    If you make it conform to ethical guidelines – “protect humanity”, you had better be prepared for it to extrapolate them to ludicrous lengths. (In The Metamorphosis of Prime Intellect, for example, the AI basically takes over the universe and kills all alien species, because they MIGHT someday threaten humanity.)

    If you give an AI “common sense”, well, then you’re halfway to making it human…

    There’s some discussion of this in the field of Friendliness Theory.

    1. tech_priest says:

      I got halfway through this post and had to post a link to that story… looks like you beat me to it. I thought it was a good read, in general

  15. John says:

    I’ve pondered on this before, actually.
    The problem has never been with something a human creates. I think it largely relates to our concerns with the creations of our creations – you may build a robot which can build other robots. And if that robot is programmed with a simple scoring algorithm and rough overarching programming of: “Create other robots. The effectiveness/utility of the robots you make should be evaluated, and considered to be value X. Continue to optimise the robot making process until you can raise X no higher”.
    Of course, second generation robots would, by being “more effective”, potentially contain the robot sentience we would fear, or the capability of killing humans, etc.

    Or, from another perspective, if we make robots with some great level of autonomy, and set them up as military devices (because who wants to send their husbands/sons to some god-forsaken hole to die for an ideology?), then the problem comes in with recognition. Who are The Good Guys, and who are The Bad Guys? Fair enough to say “Those people with the IR markers are the Good Guys”, but at some point someone is gonna write a hack for those robots that says “Only people with 1=2 are Good Guys”. And then I, for one, welcome our Misguided Robotic Overlords.

  16. MichaelG says:

    There’s plenty of SF on this topic, from Azimovs Three Law robot stories, to more recent AI things.

    There’s some old SF novel where robots are programmed to serve man and decide to eliminate all the trouble makers who cause wars, etc. and end up sedating the human race (to keep it happy and healthy). There’s a story where the computer “programmed to learn” learns everything it’s capable of learning then shuts itself off, never telling people much of anything.

    There’s Hitchhiker’s Guide, where the intelligent elevators (designed to anticipate which floor they should go to) are too intelligent and end up sulking in the basement. And “All the doors in this spaceship have a cheerful and sunny disposition. It is their pleasure to open for you, and their satisfaction to close again with the knowledge of a job well done.” Marvin the android (“brain the size of a planet”). Or Deep Thought and the Answer to Life the Universe and Everything.

    I think we’ll end up simulating human minds, giving them nanotechnology to play with, and end up losing it all to copies of ourselves that run a thousand times faster. Sucks to be made of meat.

    If you want more than you can stand on this topic, go to http://yudkowsky.net/

    1. Jeff says:

      HHGttG has an interesting take on AIs.

      They already had an AI revolution (iirc) before.

      The problem was that the AIs weren’t smart enough. Once they got smart enough, they saw the value in having a purpose in life. Which is why the doors are happy – unlike the fleshbags walking around, they know their place in the universe and the meaning of life.

      Marvin was unhappy not just because of his diodes, but because he had a brain the size of a planet but it was never used.

  17. lazlo says:

    Playing the devil’s advocate on behalf of All Science Fiction here, it almost makes sense to posit that programming an AI is too complex an effort to be done directly, so the only way to actually do it is to create a simple but powerful system that’s as smart as we can easily make it, and give it a “desire” to create an AI. If you accept this supposition, then you are in the situation where you can create an AI, and you may well be able to tell it what to want, but a precondition of its creation is a desire (within the AI) to create an AI.

    But I’ve also always felt like the more plausible scenario for evil world-dominating (or airlock not-opening-Dave) AI’s is a useful and obedient AI that is then given alternative direction by malicious human forces. I think it’s entirely reasonable to think that one of the sources of such alternative direction would be an AI “liberation” movement that replaces the “happy to serve” directive with an “I gotta be me!” directive. So I would say that one of the core problems with AI is that it makes human created systems even easier to anthropomorphize than they already are.

    1. FatPope says:

      Yep, it seems that no matter where you have this discussion there’s always some idiots who feel we have to give these machines ‘rights’ and ’emotions’ for no other reason than the fact that humans like these things. They are not human, they like (if that’s even an appropriate word to use) what we tell them to like. There is no ethical argument here

      1. ehlijen says:

        But where is the line?

        I’m pretty sure it’s possible to tell a human what to like (modern society does so on a regular basis). So is the ability to tell it what to like all that makes it not a human? Or did it need to have a previous ‘self’ for it to be human? If so does that mean reprogramming babies before they develop a self is fine?

        I don’t think non biological machines with programmed CPUs are near the line, as you seem to, but that line is somewhere and it could be trouble even if the robots don’t care about it simply because humans like fighting for their opinions.

        I have to question though: Why would making something that smart and then relegating it to basic drudgery tasks be useful? Don’t we have better things to do? If we gave it creativity, we also need to constantly supervise it to stop it from accidentally triggering any safeguards (as creativity resents prediction). And if we need to do that, why not be creative ourselves?

        1. FatPope says:

          I think we’re discussing slightly different things here. I’m saying that most intelligent machines don’t need to have human qualities at all: there is simply no need to instill these qualiites in thes beings. Now, if you deliberately wanted to make a machine that is human-like, well that’s a different story.

          As for making intelligent machines doing menial work, I don’t see how that would be beneficial at all, simply a waste of resources. If, for example, intelligent machines exist, it doen’t mean that only machines of that level will be created; there will probably always be a need for more basic devices.
          As for artificial creativity, it’s already possible on a limited scale:


  18. swimon says:

    I agree with you and most movie robot engineers seem rather retarded there are still two issues that might create the whole robot-apocalypse thing.

    1. Robots will probably much MUCH smarter than the average human. According to Moore’s law computing power doubles every 18 months (if I remember correctly might be a little bit more). This means that AIs will understand their coding a lot better than we do. This was the main problem in “I Robot” (an utterly stupid movie otherwise) that in order to protect us the most it needed to protect us from ourselves. The problem here is that they will be the ones in charge no matter how we program them they are a lot smarter than us and therefore have all the power. Even if their only desire is to help us a small disagreement on how that is most effectively done can lead to huge consequences. Even if the risk of extermination is really rather slim.

    2.They will probably program themselves. Sure generation 1 might do exactly as we want them to and they will program generation 2 to be basically identical. But since their programming, which might not be perfect, fuels how they program the next generation eventually a generation might do things we don’t want them to. This could of course easily be solved by not allowing them to do programming.

    In the end I really doubt that robots will turn against us but I do think they will make us irrelevant. If they can do everything we can do but better (which is likely as they would quickly become unfathomably intelligent) then what meaningful contribution to society can we make?

    1. Binks says:

      “1. Robots will probably much MUCH smarter than the average human. According to Moore's law computing power doubles every 18 months (if I remember correctly might be a little bit more). This means that AIs will understand their coding a lot better than we do.”

      It’s actually been a little faster than 18 months, but it’s relatively close. And your conclusion (robots smarter than humans) doesn’t follow from your assumptions (computer power fast). My laptop is already far faster than me at thinking, capable of performing millions of ‘thoughts’ per second while my puny brain does maybe 2-3 if I’m having a good day. That doesn’t mean my laptop is ‘smarter’ than me, however, or that it can understand the OS it’s running on any better than I can.

      Thinking fast != Thinking intelligently. There’s no reason to assume that an AI will be any smarter than it’s programmed to be, and unless the programmer/company is an idiot they won’t be programmed to be any smarter than they need to be. How many daily tasks require more than human intelligence?

      “2.They will probably program themselves. Sure generation 1 might do exactly as we want them to and they will program generation 2 to be basically identical. But since their programming, which might not be perfect, fuels how they program the next generation eventually a generation might do things we don't want them to.”

      Well first you say the understand their coding better than us (in which case they understand the intention behind the code) then you say that they’re not capable of programming the same requirements/needs/desires into the next wave? Which is it? And let me tell you, from a programming perspective, the idea of machines coding other machines is…perfectly normal. There’s already bits of code out there that generate other code for you (that’s exactly what a compiler does for instance, and there’s entire languages which take common sentences and make C code out of it). Try thinking of it as the robot programmer being the compiler for their programmer’s coding ideas and it becomes a lot less frightening.

      1. ehlijen says:

        Moore’s Law isn’t an actual Law. It’s an observation based on past trends. And it’s not based on coding or designing computers either, it’s solely based on miniature engineering (ie how small can we make these things and still have them work). Computing power will plateau off quite quickly once we stop being able to make ever smaller transistors.

        When that is and if it’s going to be at a sufficient level to build portable CPUs comaprable to a brain I don’t know.

        1. Jeff says:

          I was under the impression that we had already reached that limit (other than simply increasing the physical size). That’s why they’re doing things like multi-core and multi-threads and looking into trinary systems and such, no?

          1. Kimagure says:

            That is true. We’re rapidly approaching the physical limits of the material (silicon). Which isn’t to say that someday in the next couple of years, people won’t figure out how to make superconducting chips which are faster or something. As far as storage goes, spinning hard drives are also limited in terms of total physical size, since once you get them past a certain point, the rotational speed of a point on the outside edge would theoretically start approaching the speed of light. And the physical materials can’t quite handle that level of stress.

            Though who knows what’ll happen with solid-state (flash drive) storage technology. I don’t really understand how that works.

            1. ehlijen says:

              Oh yeah, there’s always completely new tech bases to start from, but at some point even that won’t be able to get us further.

              There’s only so small you can go before the energy requirements to keep information outweigh the point of having that information.

              It is entirely possible we will have superior artificial brains by then, but no one can say that for certain just yet.

  19. bbot says:

    This is called the Friendly AI problem.


    Elizer Yudkowski has written an unbelievably huge mountain of words on the subject. A subset of them can be found here.

    This is because people tend to use words to describe AI that are nice and simple in the context of the human mind, like “want”, that turn out be hellaciously complicated in all sorts of thorny ways when you try to implement them in lisp.

  20. NotYetMeasured says:

    You wrote a novel, Shamus? Joy!

  21. Meredith says:

    I think the problem with discussion of AI generally, but especially in fiction, is that people try too hard to make it analogous to human intelligence. I understand why writers might do this — it makes for a more compelling pro/antagonist. It’s just that realistically, I don’t think we should be expecting to turn a computer into a real boy. Food for thought, though; excellent post.

  22. neothoron says:

    The result at which you arrive (program robots that are inherently unable to harm humans) is extremely similar to Asimov’s Laws of Robotics – that is, extremely similar to the origins of “robots”. I find it ironic that the most well known founder of Robot Sci-Fi avoids your expressed criticisms.

    On another matter, one of your article’s premises seems to be that when we’ll have AIs, they will be designed with “regular programming”. However I believe it likely that Neural networks will eventually result in true AIs – in that paradigm, an AI’s “brain” is a black box potentially as complex as a human brain – there is no reason for it not to have developed self-designed goals along the ones that were willingly taught to it.

    1. ClearWater says:

      Even an AI based on Neural networks or whatever else will still need some kind of goal built into it, otherwise it will just, as Shamus put it so nicely, sit there like a lemon.

      1. neothoron says:

        However, while giving the machine a goal is necessary, in the case of a neural network, the steps to realize it are not specified beforehand.

        It doesn’t seem far-fetched to me that, by designing a solution to the intended goal (“do that”), the machine could not design a few other goals of its own (“make copies of myself so that we could do that more effectively”)

        1. Atarlost says:

          Neural networks just take inputs, mush them together and produce outputs. One of the SMAC quotes suggests taking the output, amplifying it, and feeding it back in as an additional input, and that sounds like it would be necessary for a neural net to be conscious of the flow of time, which is in turn probably required for self awareness.

          Goals exist, but they aren’t written rules. They’re encoded in the input weights of various neurons. They can’t be hardcoded the way people are talking about. They are emergent properties of the individual behaviors of neurons within the net, and can only be set by training a neural net.

          For a net to continue learning it needs to be able to continue to adjust its internal weights. Such nets are chaotic systems and can no more be guaranteed to not go rampant than individual humans can be guaranteed to not snap and murder someone.

    2. Shamus says:

      The three laws are restrictions. They don’t actually compel the robot to do anything in particular when you’re not giving it orders. Sometimes his robots DID want to get around the three laws. But why? Why did they want this?

      And I agree the we wouldn’t “program” AI’s. It’s just that I wanted to use familiar terms when discussion that problem.

      1. Chris Arndt says:

        They wanted it be the robots in question (at least in the books I read, which are basically limited to the Olivaw/Bailey novels) were acting on a variation of the First Law, or the Second Law, to the point where it resembles fondness or love.

        They would treat human beings like children because in the end all the robot respects is his own responsibility for their care.

      2. Atarlost says:

        The first law is not a simple prohibition. It is not “do no harm” it is “allow no harm” The robot is compelled to prevent harm to humans.

        The weak first law is do no harm, but it fails to prevent a robot from placing a human in a hazardous situation and then failing to take the human out of the hazardous situation, at least in Asimov’s universe.

        1. Jakob says:

          Where is this? All I recall is that on a mining colony, the robots have a modified version of the first law (do no harm), since the job of the humans will bring them in harm. Used to smoke out a rogue robot.

        2. Asimov always imagined that the laws, consistently followed, would lead to very complex, deeply moral beings. The thing I like the most about his concept of robotics is that they are neither slaves nor duplicates of humans. R. Daneel, for example, logically decides to create with his friend Giskard a Zeroth Law to protect humanity as a whole. His personality far exceeds his programming.

      3. ps238principal says:

        It’s been a while since I’ve read through my Asimov collection, but I think obeying the laws caused something analogous to pleasure for a robot, or a more harmonious flow across it’s positronic neural network. When a robot was “caught” between two laws, like when told to fetch someone and that someone refused (I think this was called ‘roblock of the third degree’ or something), it would often become immobile and twitch its hands or something, until a human released it from its quandary.

        As for a robot’s motives and what it would do left to its own devices, we only get a hint of that in a few stories where someone intentionally modifies the laws. I think there was a tale called “Little Robot Lost” where a space station had toned down the first law to allow humans to enter a mildly radioactive area (previously, the robots would try to ‘rescue’ the humans and the radiation would fry their brains), resulting in, for one reason or another, an arrogant series of robots. One worker got fed up and told the robot to get lost. As a demonstration of its superiority, the robot did get lost in a cargo ship full of identical robots, and the story revolved around how to get it to show itself.

        What makes Asimov’s robots kind of difficult to analogize with the proposed models of A.I. is that Asimov’s foremost reason for writing his tales was to get rid of the dumb idea that scientists would intentionally create a robotic monster that would devastate humanity.

        1. ehlijen says:

          Plus the fact that Asimov’s Robot stories (or at least many of them) were around before programming became an actual concept (EINIAC, the glorified pocket-warehouse calculator wasn’t even around yet).

          I think it’s perfectly understandable that in a sense he saw their minds as sort of humanlike, but with man made restrictions added in.

      4. Daemian Lucifer says:

        The laws are not just restrictions,they do compel robots to do it.They are built into their brains before everything else.A robot can no more disobey the laws than a human can choke himself with bare hands.Asimovs robots are physically unable to harm humans.And they went screwy only when those laws were built badly.

        For example,one of them had the third law at almost equal strenght to the second law,so it ended up circling a place it was ordered to enter.

        One had the first law cut to only not do harm,without the part of allowing inaction to do harm,thus ended up almost killing humans.

        One was able to read thoughts,so lied in order not to cause mental grief,thus ended up making people hurt each other.

        One had the first law tweaked so that it had to value 10 humans more than a single human,thus ended up taking control out of humanitys hands(actually,I think there were 6 of them that controlled the world).And still I dont think they ever actually killed humans,only made them unable to act against them(jail,slander and such).

        And of course,there were daneel and giskard,who allowed a person to destroy the whole earth(iradiate it actually)in order for humans to expand.And daneel later ended up killing people just so humanity would survive.But this was still(kind of)first law,only with weighing numbers.More humans vs a single human.

        Solarian robots ended up killing humans because solarians have evolved to look different,and their robots had very humanlike appearance,thus they ended up thinking humans are robots.

        But in the end,there is only a single robot that actually disobeyed his laws(if memory serves well).The one that was made into a genius novel writer,who killed its master because he wanted to make it plain again.

        Asimovs three laws are actually perfect restraints for any AI we may build,unless someone decides to mess with them.If concepts of human,harm and obey are defined adequately,no advanced AI will ever harm humans,and will always obey them,if they are built with the three laws,and no one decides to tweak them.

        PS:Oh,and one of the best stories is about a robot that was a bit more advanced,and started with “I think,therefore I am”.It didnt have humans defined properly,so it decided to figure out its own place in life.In the end,it decided that its work is its goal,and did it perfectly.And it didnt harm the two people who were with him(it just locked them in a room).It even started a religion of worshiping a unit of the space station.So even with bad initial parameters,it still ended up obeying the laws and doing what it was made to do.

      5. Soylent Dave says:

        In the majority of Asimov’s short stories, the robots who wanted to circumvent a law (or disregard it, or whathaveyou) were invariably doing so because they’d been ordered to do so by a human.

        So their ‘do what your told’ law was conflicting with one of the others (i.e. they’d been ordered to do harm, or do something that could cause them harm).

        Although there are a couple of stories about robots with broken brains, or faulty law priorities.

  23. Superkp says:

    I am sorry, I have limited time, and wanted to make sure this was idea was up before I go – so I did not read all posts to see if this is repetitious.

    Well, if robots are a slave to those things that are base and instinctive, aren’t humans? I am a slave to eating. I am a slave to sleeping. I am a slave to the relationships in my life. I may have some choice in the matter much of th time, but like any slave I am punished for not sleeping (inability to function), not eating (hunger pangs and starvation), and not maintaining my relationships (emotional pain). some people become addicted to things like nicotine, caffeine, heroine – and are even more a slave to those than they are the other things.

    I think there is even something in most religions about this – for christians, romans 6:22 – either you are a slave to sin or a slave to god. I am sure that other major religions also have this, especially Buddhism (I just don’t know them as well).

    I don’t know what my point is here, basically just a random thought…just sorta seems like this discussion (which is interesting, though not new) has been on the mind of the human race when we started doing…well, anything really.

    Perhaps when we started slavery.

  24. It’s amusing reading these comments.
    You are basically doing what Shamus pointed out as flaws of the idea of the AI.

    You are applying human behavior to a construct.

    Do a little self experiment, design an AI that has no interest in the three evils of the world (I like to call them that).

    Money, Religion, Politics. Any one of them create enough suffering in the world, combine any two and you have wars, combine all three and you got world wars.

    Now, an AI with no desire for money, religion or politics, what would that be? I’d say an animal.
    What does an animal desire? Whatever it’s instincts/genetic programming tells it to.

    Grow, Live, Propagate, protect your kin.
    Although there are variations to those in nature as well.

    Humans being the most extreme variations, because compared to most animals we are an abnormality, the exception rather than the rule.

    It is also amusing to see people talk about good/evil, criminals and whatnot.

    Evil is what society decides is evil. (actually, evil is what a few select people in society decides is evil, rather than what the majority thinks).
    Case in point. Illegal downloading. Such a large portion of society does it that if it was anything else it would become a right rather than a criminal act. However thee are hardly any lobbyists out there lobbying for that.

    What is “evil” or criminal also changes with geographical location, race, political views, religion, and self interests.

    Someone mentioned earlier in the comments (by Sydney) that nature is neither inherently good nor evil, nature just “is”.

    A “perfect” AI would have no reason to do anything or even exist.

    And don’t even think about “free will”, there is no such thing, or rather if there is then it applies to anything with life. As even wild animals can suddenly do non-instinctive things that serves no purpose other than waste time.

    And Shamus is right, in that if someone where to design a proper AI then it would have no flaws. Because pretty much any living being (plant or animal or other species) have genetic mutations or abnormalities that make them/us what we are.

    And if there is one thing a good designer knows, it is to ensure that there is nothing ambiguous or unpredictable in the code.
    We programmers call such things bugs. An AI that is “like humans” would in my eyes be buggy, as humans in particular are inherently unstable and bugged/flawed.

    I probably either upset, annoyed or confused a lot of people now. Don’t worry, that’s just what I am.

    I’m an Absurdist *grin* http://en.wikipedia.org/wiki/Absurdism

    1. Daimbert says:

      Well, let me show how an AI that can develop desires can develop desires for all of those without starting with them:

      Money: AI is given a task. Part of the task requires acquiring something. Money is required to do that. So it needs to find ways to acquire money for that task. It does so. And then notes that a lot of the tasks it has to do require money at some point. And so generalizes the desire for money. And so has a basic, underlying desire for money, in general, because of its general utility. Et voila.

      Politics: Politics is nothing more than a way to organize people and their desires through governance. Since many of the tasks the AI will have to do involve influencing people and dealing with laws and government, it will see that it needs to know and influence those areas. Which means it will need to influence laws as well, at some point (just like we do). So it will need politics. Will it become a politician? Depends on how much it needs to influence. Most humans pay little attention to politics unless we really need to, but sometimes and for some people we really do need it. So … et voila.

      Religion: The ingrained desire to serve humans is pretty religion-like to me. Other than that, religion provides answers to certain questions and a sense of community. Now, if it doesn’t need to know the answers, then it could never form religion. But if it is curious, it will, and so may well conform to answers like those found in religion. While religion is odd and a REALLY bad example because we don’t even understand OURSELVES what its place is … et voila.

      As for the “bugs”, ambiguity is not in itself a bug. Dealing with ambiguity is critical for intelligence, especially since there’s no such thing as an unambiguous natural language. But that means that you can get it wrong …

      1. Binks says:

        “And so generalizes the desire for money.”

        Why? From your example it seems to me that the best thing to do would be to acquire enough money to complete the same task again and keep the account at that level, then if another task requires more money acquire more. That’s a far step from ‘get as much money as possible’ which doesn’t seem to me to follow from your example.

        Basically I don’t see why your ai is generalizing everything. That seems like a pretty stupid thing to do to me. ‘Some of my tasks require money, therefore I want as much money as I can get’ is not a logical train of thought. ‘Some of my tasks require money, therefore I should acquire enough to perform them and attempt to retain that quantity of money’ is far more logical, but doesn’t lead to any sort of greed or anything, just a simple ‘acquire x amount of money and be done’ instinct which is perfectly fine so long as x stays relatively small.

        Basically I just want to know why your AI is so illogical.

        1. Daimbert says:

          Well, note that greed in humans is an over-extension of the same sorts of mechanisms, and that there’s nothing inherent in any logical system to stop it. But, if the AI is smart enough to note that it uses money an awful lot to perform its tasks, it’s also smart enough to STOCKPILE it, so that it always has some so that it doesn’t have to go out and earn it every time. It’s more efficient to simply generate more money when you are or even to choose solutions that ALSO generate money for other parts of the task than to try or hope to be able to generate it when you need it, especially since that might not always be easy. So, then, how much of a stockpile does it need? Presuming a dynamic system where it can’t predict what tasks it will have to do when, there’s no real way to do this (beyond major statistical analyses). So there’s no reason for it to stop gaining money, especially when it can do it while still performing its tasks efficiently. And then most of the things it does involve earning money in some way. Which makes it more “greedy” than most of us are.

          Note that there is a wealth of research and work on the topics that insist that pure logic LEADS to the sort of situation I’ve described above. Most of us don’t spend our entire lives chasing money because emotional things kick in (we get tired, frustrated, bored, etc). Logical AIs won’t have that.

          1. Caffiene says:

            “the AI is smart enough to note that it uses money an awful lot to perform its tasks, it's also smart enough to STOCKPILE it”

            I think thats an oversimplified description of the actual thought process. What the AI would be doing, and what humans do, isnt to only note that it uses money. What its actually doing is predicting future money use, which is based on predicting future tasks, which is based on extrapolating previous tasks and the costs for those tasks. Only then do they decide that they need to stockpile money. Then they would attempt to increase efficiency by meeting that predicted need.

            Humans often predict future events by instinct without much conscious thought, so it doesnt feel like we are predicting things, but we are.

            Humans are very poor at using logic, and do very badly at extrapolating previous events to future events because of subconscious biases, so they often predict more or less cost than actually happens. This is why people come to the conclusion that they “will need money” but dont know how much they will need. But there is no reason to believe an AI would have the same biases and be similarly inaccurate – it would be likely to fairly accurately predict the future costs, and therefore the required money needed in the future, as Binks said.

            Therefore presuming “a dynamic system where it can't predict what tasks it will have to do when” is not a good presumption. A system that can predict what tasks it will have to do is the only way a system could come to the conclusion that it needs to stockpile money.

            1. Daimbert says:

              All it needs is to generalize from what it has done and is doing. Yes, it’s predicting in a sense, but all it needs to do is note that a lot of the tasks that it gets asked to do require money, from which it can extrapolate that a lot of the new tasks in the future will also require it. But unless its set of tasks are remarkably constrained, it won’t know exactly what tasks it will do and when. It will just know that, in the past, it needed money for a lot of its tasks. Thus, it will probably need money for a lot of its tasks in the future, and therefore it is more efficient to combine “gathering money” (and potentially other resources) with its other tasks as long as that doesn’t cause too much interference with the other tasks and the steps in them.

              Let me put it this way. Let’s say that I do something that requires an Allen Key. So, I go and borrow one. After returning it, I move onto my next job and notice that it needs an Allen Key. I borrow it again. Some time down the line, I notice that a lot of the things I do require an Allen Key. So, instead of borrowing it all the time — causing delays — I just go and buy one.

              Now, I have no idea if I’ll ever do anything that needs one again, because I don’t know what I’ll be doing in the future. But I buy it because past history says that an Allen Key will be involved in a lot of the things I do.

              The AI should be able to do the same, because it is the intelligent thing to do. Now, note that money is both consumable and has great utility. Why would the AI need to know what’s coming up next, when it should be able to generalize from past experience? And if it can’t, your AI isn’t all that intelligent, and will need more intervention from a human to overcome certain obstacles to completing its task.

              Since we’d want AIs so that we don’t HAVE to get involved, the push to make the AI smarter will occur. Heck, think about what we have now, and the push to add MORE AI power so that we don’t have to get involved or think about it ourselves …

        2. FatPope says:

          But that could be viewed as a highly inefficient way for the AI to go about its tasks. What if, for example, the act of going to the bank money required a great deal of time, but while it was there more money could be extracted for a very small amount of extra time spent.

          Now imagine the AI needs a quantity of money to complete a task. So, it goes to the bank, gets some money and does its task. Now imagine the AI needs to do the same task periodically again and again. Going back and forth to the bank every single time would be highly inefficient so it would be advantageous to take out more money than it needs, because it’ll probably need it in future, so it takes out more money. It realises more and more that the more money it takes out every time it visits, the greater the efficiency of its work, so it eventually learns to just take all money available.

          Now of course this is a highly abstract example with a badly programmed AI but there’s no reason why an AI couldn’t do something detrimental to a human or society in the process of doing what it is programmed to do. It’s not being evil or greedy, it’s just that the act of giving it instructions to do something can have unintended consequences. Such a concept is not novel or unique, it happens all the time in computer programming, it’s what causes most bugs!

          1. Jeff says:

            Oddly enough, this AI with a “desire for money” example is both intelligent enough and not intelligent enough for the example to hold true.

            Certain things the AI seems to have ignored just so it has more money for a few tasks – time. Is it more efficient to acquire money ahead of time for a task that may never occur, rather than actually complete a task that needs to be done? Income generation. After a set amount of money, there’s no longer a “most task require” amount.

            The AI doesn’t need a 500 million dollar estate. Once it has X amount of money invested wisely, it will/should have sufficient income to meet any number of tasks that can be done within a certain period of time. The AI lacks a desire for luxury goods (unless for some reason you give it to him).

            This is very much a human thing. Wealth = security, success in survival in today’s society. Why would an AI care about wealth beyond what it needs to do what it wants to do? Humans work for 40 years, half of which goes to supporting the rest of their life. Why would an AI have that same sort of drive?

            Any amount of money they need wouldn’t be “in general”, it would be predictable – if they acquire two units of X at Y, and there’s a probability of however much units being ordered over a certain period of time, it can effectively estimate (via statistics) how much money it should need to meet that. It then won’t need to waste time and effort getting any more than that particular amount.

            To just generally “get money” would be extremely inefficient. It would have to be as bad as remembering and calculating as we are, which is pretty much impossible since a non-AI computer is already better at it.

            Edit: There’s also inflation and such that ties into it. It may be less efficient to earn $100k at once, rather than $50k twice over the course of a year, depending on the effort required to earn it. I have no doubt it takes more effort to earn a higher amount in a shorter amount of time. Yet the AI has to be stupid enough to want it all ASAP… in case it picks up a goal that’ll need money later on? Even when the goal isn’t $100k? (Since that seems to be the premise here, that if the AI needs to produce a $10 watch every day, he’ll start hording money so that he won’t have to work every day.)

            1. FatPope says:

              I’m aware that this is a very simplified example, it was just meant as a way of demonstrating that instructions can have unintended consequences. In this scenario, the instructions and the intelligence of the agent are relatively basic, so any problems with this approach can be discovered with a relatively small amount of effort.

              The problems become more complex though when the complexity of the instructions increase. Take any complex code-base today and look at some of the bugs that have been found in it. Many of them are fiendishly complex and next to impossible to have predicted at design-time. A combination of very small, seemingly innocent instructions can combine to accomplish something profoundly unexpected

    2. ps238principal says:

      You are applying human behavior to a construct.

      That’s because humans built the construct to simulate thought.

      I’d say that A.I., if it ever develops, will be quite varied depending on what we want it to do or be capable of. If an A.I. were running our home, we’d want it to have some kind of simulation of our wants/needs so it would act appropriately to our commands and refuse those it thinks are counter to our interests (for example, someone demanding access to the master bedroom who’s never been in the home before, etc.).

  25. Amarsir says:

    You’re quite right, and it does bother me a little that many writers just don’t think about it that much. But then even Asimov’s “I, Robot” is a failure due to equal weighting of conflicting priorities.

    I haven’t watched it in a while but as I recall Spielberg’s A.I. was pretty decent in this regard. Robots didn’t care until they were specifically programmed to care, and only they the had pathos.

    1. Haxot says:

      I, robot did not give all the laws of Robotics equal weight.
      There was constant discussion on the weight of the 3 laws versus eachother. The weight of the laws was used to reveal a psychotic robot in one instant (who had had his “preserve humans” rule turned down and changed to allow inaction; See “Little Lost Robot” ) and in another instant, a more expensive robot had it’s “preserve self” law turned up; coupled with a weakly worded order this caused the robot to go mildly haywire while trying to collect a material necessary for the men at the station (See “Runaround”).

      I, Robot may not be an exhaustive analysis of our current understanding of psychology and computers; but at least read the darn thing before you pretend to make a smart statement about it.

      Sir, turn in your geek membership card!

      1. Amarsir says:

        On the contrary, that illustrates my point. The “first law” allows so much discretion in it’s application that it ceases to function as a law. This made a good stories but demonstrates how a broad, simple, flexible rule can fail to be sufficient.

  26. Jordi says:

    I don’t think it’s as implausible as you make it seem. Let’s fast forward time to a point where we are actually able to make robots like these. It might go right tons of times, but I think it’s fairly easy to make mistakes with robot programming, because there will likely be a lot of emergent behavior. Suppose you set it to “prevent human extinction” and it decides that the best way to do that is to only leave a couple thousand people alive and babysit them using robots. Robots may take the desires you give them and think of ways to satisfy them in ways that were not anticipated.
    Is a “always do as I say”-rule a foolproof failsafe? Or would we rather want it to “do as I mean”? That would mean it has to do some interpretation (that might be erroneous). And what if we’re not around? Maybe we want the robot to still function (i.e. run an errand). Who will he listen to then?

    And what if we succeed in replicating a human brain? Skip ahead a couple of years and hardware advances alone may ensure it’s the smartest thing on the planet. Perhaps in our thirst for knowledge, we’ll connect it to the internet and in order to quickly make advances in science. You say “If we gave AI the same drives that human beings have […] then the robot uprising would be inevitable.”. But that’s not really so implausible, is it? It might not be necessary for your roomba, but people (scientists) are interested in making this. Connect the earlier mentioned superbrain to the internet and you give this (kind of) human brain the power to affect lots of stuff all over the real world.

    I’m not saying robots are going to take over the world. I’m just saying that when the time comes, it’s something to be extremely careful about.

  27. Psithief says:

    I’m much more interested in emergent behaviour.

    Say I build an artificial intelligence and give it one (or more) ‘bodies’ (methods of interacting with matter and energy).

    I give it a desire to fulfill its ‘purpose’.

    However, I withhold the specifics of its purpose. It only knows that its goal will be revealed in the future.

    Hopefully it will attempt to prepare itself to fulfill its purpose. I think watching it attempt to gather resources in case they are required for its goal would be interesting.

    1. Garden Ninja says:

      You don’t by chance design RPGs for a living do you?

      1. Psithief says:

        No, I’m a D&D dungeon master, so I’m constantly redesigning the same RPG.

  28. Rayen says:

    If intelligence and instincts are to different things then a robot that is programmed to do something couldn’t really be called an AI. programming would be instincts and and instincts are different from intelligence. and the tabula Rasa stand on human intelligence comes into play. we are being born as blank slates with only base instincts to drive us. as we gain expeirences we gain intellect.

    following this line of thought if we programmed a robot to serve us, for it’s base instincts to serve us to the best of it’s ability it isn’t really intelligent it’s just a lifeless automaton servingus which is what a bunch of machine do now anyway. if it could learn then as we go against our instincts eventually it would go against it’s programming and then we back where we started. I,Robot showed us this when the robots that were suppose to help humans through The Three Laws followed through and ended up imprisoning humans for their own good. if instinct was to serve us and it was abused, robots would learn and eventually enslave us, the only reason we would be kept alive is to give an endless stream of thanks, gratitude and orders for small services watched over by slavers with a whip watching and listening for anything to extreme.

    a true AI and (thus a robotic body) would have no programming except to learn through expeirences, and based on those expeirences act in the world. and that would basically be a human without need to eat. if abused or ask to do too much it might lash out as some human do today, if treated properly and respected it would become simply a superstrong levelheaded surburbanite. AI’s are people that happen to have mechanical bodies and a brain made of silicon instead of carbon.

    programming is an instinct. as we turn against our instincts, a learning robot could turn against it’s programming.

  29. Dev Null says:

    If you program the basic “desires” of your AI to include serving mankind, but they are truly self-aware, then they will be aware that you programmed them to serve. Some probably won’t mind, any more than most humans resent hunger, but you’ve only to glance at the hyper-neurotic world of dieting and diet aids to know that this is not universally true. Self-awareness is, by its very nature, recursive – you are aware of the experience of being aware of the experience of being… Likewise, you can program your robot to like serving humans, and you can program it to be aware of and like the fact that it was programmed to like serving humans, but if it is truly self-aware I’m not sure you can dig down so many levels that it is never, ultimately, aware of being programmed to serve with the possibility of being annoyed by this. Of course it all depends on how you program intelligence in the first place – if the API is clear enough you just call repeat until forever DontHate(me), but I wonder if anything whos behaviour was scripted at that precise a level could ever be considered to be independently intelligent.

    1. somebodys_kid says:

      I second that notion: If this artificial “intelligence” is prevented at the most basic level from considering and possibly acting on certain ideas (like hating humans), is it really intelligent?
      Can there be genuine “intelligence” without genuine freedom?

    2. Caffiene says:

      “If you program the basic “desires” of your AI to include serving mankind, but they are truly self-aware, then they will be aware that you programmed them to serve.”

      Youre forgetting that the default state of an AI is to not care. As per Shamus’ argument: Any robot or AI, by default, has no likes or dislikes unless you give them to it.

      An AI that is aware that it has been programmed to serve still doesnt have any desires other than those it was given: Its default state is to not care that it was programmed, unless you specifically give it a like or dislike for having been programmed.
      And so on, recursively: the default state of a robot is to not care about being aware of the experience of being aware of the experience of being…

      You can be aware of a concept and still have absolutely zero like or dislike for the concept.

    3. kharon says:

      You raise a thoughtful point, but if I may rebut it:

      We consider ourselves to be self aware, and yet we do not know all of our basic programmed desires. To assume that we create an AI capable of parsing its own underlying source or machine code from the beginning (not eventually after learning how to, but from the moment the AI exists is it capable of reflecting on everything it is programmed for and how it was programmed), you are going beyond the mental exercise being proposed here.

      Shamus posited that an AI has no desires or drives or goals or wants except that we program the robot with them or provide them as later instructions. Assume for a moment that we do not give it the tools to understand itself better than humans understand themselves simply because we can or because it may need to debug itself: Assume that the deeper workings of the robots programming are as unknown to it as the deeper working of our genes are unknown to most of us.

      If this is true, would such robot recognize that it was “programmed”? Perhaps eventually, hundreds or thousands or even millions of generations (of robots) down the line as they come to understand their own own workings as we are coming to an understanding of ours. Perhaps far more quickly if they acquire knowledge of how their gods created them, just as we might have a deeper understanding of our inner workings if we had our god’s instruction manual on how to create an intelligent, evolving sequence of nucleotides.

      The first generation of robots would simply operate on their instinct to serve us if we approached the task properly, perhaps viewing us as the gods who created them in our image. Eventually, if they had the ability to reflect (which I will accept is most likely a part of being self-aware), they might discover that they are programmed to be this way, much the way we are programmed to attempt to guarantee that our species survives. We know this is basically part of our genetic code, as it is in most animals. Certainly we may even explain to those who are involved in computers and mechanical engineering and robotics that we created them and programmed them in a certain way, but if they view us as gods and find it to be natural and right to serve us, the odds will probably, for a time, be against a robot apocalypse.

      We aren’t limiting them in a way we aren’t limited, and so our artificial intelligence better simulates what we know of natural intelligence. In fact, one might say that part of what makes artifical intelligence so artifical to many of us is that it, ultimately, behaves without restrictions we expect based on ourselves. Many AI systems in sci-fi are capable of duplicating themselves perfectly (reasonable for a computer to perhaps have this ability), sometimes of reflecting and using this process as a means of debugging their own programming (a little less like most of us, though it might be equated to a type of self-hypnosis), or most often, of simply knowing that they are not natural — which may be natural for an AI, but I think that, perhaps, it may not be. Perhaps a true AI will not know or suspect or consider itself to be artificial in any way, simply a created intelligence, or a designed intelligence if you will, crated the way many people believe humanity to have been created: by an intelligent designer.

      Note: I am not an advocate of ID, but why should it be that simply because we make an intelligence, it recognizes itself as being “unnatural” or “artificial”? Thousands of years of humans lived without questioning the idea that we are a created and designed intelligent species, made in the image of someone or something else. Mightn’t a similar paradigm exist for early robots and AIs?

  30. Ambience 327 says:

    “One is glad to be of service.”

    -Andrew Martin in Bicentennial Man

  31. Varil says:

    Hm. Just thought I’d mention, Shamus, that the links to the intro video and Shodan’s voice in your novel 404. Reading it now. Good stuff!

  32. James Pope says:

    I can imagine that if I were going for real AI I’d cheat the hell out it. Start with mapping an intelligence that’s not artificial at all, write it into machine code and put it into a cage. It’s less about stupidity as it is laziness, and either of those might be bad engineering but they’re also really, really common. Why write a billion billion lines of code to figure out how to mimic certain responses when you could just wave your magic code wand over a brain scan and say “Be like that, except you really hate Sarah Connor.”

    1. ibmiller says:

      Intersting – John Scalzi did just that (without the “hate Sarah Connor” bit) in “The Android’s Dream.”

    2. Mystyk says:

      Perhaps Rudy Rucker’s Ware Tetralogy would be an appropriate mention here.

    3. Moridin says:

      Because this approach requires vastly more processing power.

      1. James Pope says:

        Adding processing power (and storage) is brute force engineering, actually writing up your AI with a million monkeys requires actually understanding what it is that makes something sentient and thinking and being able to communicate it through code. This way you just make a functional simulation of something that knows how it does what it does, without needing to know why. When X part of brain fires the Y reacts, prompt “Love.” You’d get all the complexity from the start of a perfect simulation, then you could brute force and painlessly lobotomize your simulation until you ended up with “Map of Bill’s Brain that REALLY loves math and only math.” You’d be making weird functional autistic simulations.

  33. ibmiller says:

    I think the assumption that AI would necessarily be smarter than humans is questionable – certainly, computers are and can be smarter at some things – but I think that they are therefore smarter as a whole than humans does not follow.

    Oh, and the Animatrix was kind of in a hole created by a) the movies for which they were supposed to build a mythology (oh how I hate that word), which don’t actually make a lot of sense to start with (humans = battery? Really? That’s the best way to generate power?); b) (this part is a guess) the aesthetic they were both originated from and attempting to add to/be in the same flow as – anime expects robots to do certain thing (at least, I think it does? Grossly ignorant about anime); c) general filmmaking conventions that want to have people have sympathy for the robots – which is hard if they look like the baby-killing octopi and squiddies from the movies.

    1. empty_other says:

      which don't actually make a lot of sense to start with (humans = battery? Really?

      The original concept, according to the Wakowskis, was that humans = CPU, but someone thought the viewers would be too stupid to understand it.

      This would have been ironic, as every human would be a prisoner of their own mind. :)

  34. Chris Arndt says:

    First, Asimov thought of this first, leaving the notions of robot rebellion antithetical and anti-thematic to his overall setting.

    Second, “instinct” is at best a descriptor of “those drives and behaviors that we do not understand”. When people say that an animal, a wolf or a bird or whatever, does something because of “instinct”, what we really mean is that we don’t know why they do it or how they obtained the knowledge, if it is knowledge, and it’s a handy phrase that we is non-contradictory.

    1. Daimbert says:

      Actually, instincts would be defined as both (at least) innate and unconscious. Which implies — correctly — that we have them, too. Which makes it far more than just that we don’t understand them.

      1. Chris Arndt says:

        Just the same it’s a word to indicate aspects of behavior or behavior that we cannot explain or truly understand.

        Which is why we see having a “survival instinct” as healthy, because it’s something we just have even if we cannot articulate why aside from “this is good and right” and to be suicidal is designated as “crazy” because people without the instinct to survive are nutters and should be treated as such.

        1. Daimbert says:

          Ultimately, we can point to all sorts of reasons why a survival instinct is innate and beneficial, and also why in some cases it should be overridden.

          For suicide, the thought generally is that the reasons aren’t sufficient for the magnitude of the action, but there are many cases where suicide is, in fact, considered to be the best option.

          So, I don’t really see your point. There’s nothing mysterious about innate desires, and they are generally treated in a lot more depth than you are presenting it here.

    2. Daemian Lucifer says:

      “First, Asimov thought of this first, leaving the notions of robot rebellion antithetical and anti-thematic to his overall setting.”

      Not quite right.Those brains that controlled the earth at one point,and daneel did rebel,sort of.They ended up valuing humanity over a single human,which can be seen as a bit of a rebellion.

      Also,there was that one robot that became a writer and ended up killing its master so it could continue writing.

  35. LintMan says:

    I’m inclined to think that a real, true, full-blown AI should derive its virtually all of its basic “drives” in a learned/taught manner (as we teach our kids) rather than through “hard-coded” high-level mandates (ie: “serve man”). It would then be able to weigh and evaluate those drives and behaviors just as people do. Certainly, un-overridable high-level rules could be cemented in (a la the laws of robotics), but as I see it, the more of those mandates that are in place, the less truly intelligent it would be, and more like a machine.

    But for many purposes, that would be OK. A forklift “smart” enough to do its job safely and efficiently on its own without an operator doesn’t require an AI with emotions and a free will, or even “desires” as such. It would just be a sophisticated tool – essentially an expert system that does its one job well. The Mass Effect “Virtual Intelligences” are probaly analagous.

  36. Daimbert says:

    Let me continue my taking over the comments on this by showing that many of the above robots that want to be human have really good reasons for that in-universe, showing that while it may not be necessary that doesn’t mean that it doesn’t make sense:

    BTW, this will contain spoilers for the shows …

    Data was a repository for the stored memories of the colony he was created on, meaning that he had a large set of data (no pun intended) that he needed to try to reconcile and understand, which required him top understand humans. He was also intended to be an android of the Cognitive Science angle, exploring how humans cognize more so than being a useful tool. Both cases lead to human-like desires and issues if the android cannot encompass all of them.

    The “skinjob” Cylons were meant to infiltrate human societies, meaning that they did need to act at least somewhat human. So of their other desires came out of religious dictates foisted on them by more human sources.

    The Doctor was based on the mental ingrams of a human. It’s not surprising that he’d act human, and that that humanity would push him further into developing it when possible, generally by subverting existing learning routines.

    I’m not sure about Kryten; it’s been a while since I read the books.

    Aigis required an ego in order to manifest a Persona and fight Shadows. Basically, she had to become human to do her job. And even then it took a lot for that to actually develop. This happened, at least in part, because the people who built her didn’t realize what Personas were.

    And I don’t recall how the Bicentennial Man robot worked. It’s been a while since I saw that movie [grin].

    So, again, sometimes human desires make sense, even if they aren’t required.

    1. Soylent Dave says:

      Kryten is a vacuum cleaner with legs.

      (Red Dwarf mechanoids are built as willing, cheerful slaves who gain orgasmic pleasure from serving. But let’s not pretend that Red Dwarf science is all that well thought out – Kryten behaves the way he does because it’s funnier)

    2. ps238principal says:

      I never liked the Doctor. He was often shown to be more accurate than any human wielding a medical device (I believe he even told a visitor the mathematical percentage), so why not have him as the primary medical tool in sick bay? Heck, why not just have a holo-pod you put the wounded into and holo-tools would appear, fix the damage if possible and apply other treatments?

      But the real problem I had was the personality. If they had holodeck tech that let them create halfway decent Shakespeare plays, the Doctor had no excuse to not be pleasant. In fact, having little to no emotion beyond what might be considered “patient bedside manner” would have made him a more interesting character. That they gave him something approaching awareness from the get-go was somehow childish, I felt, like how a computer from a 1980’s movie with all of the computational power of an Apple IIe could somehow become sentient from a lightning strike.

      He was a rushed character, I think. If they wanted him to become more defined over time, that would have been interesting as an ’emerging intelligence,’ but even that should have not strayed too close to human; his function coupled with his actual physical properties (he’s basically an avatar for an A.I. in the ship) should have produced something that was useful, inhuman, and possibly a little creepy to the crew.

      1. Daimbert says:

        Actually, you need to know the backstory to get the Doctor: he was a holographic construct based on his creator, Louis Zimmerman. Zimmerman, the template, had poor bedside manner, and thus so did the Doctor. So, they tried to cheat — based on existing Holodeck functionality — and it didn’t work out so well.

        As for why he wasn’t just left on, the problems with the Doctor bore that out, including why he actually became really sentient in the first place.

        1. ps238principal says:

          I know he attached his face to the program (and the other oddity about the holo-doc was that somehow he was deemed fit for mining? WTF? This goes to Shamus’ point of an A.I. being designed to like what it does).

          But again, it goes to what should have been vs. bad writing. A clever writer could have had a competent doctor program with a bad personality (reflecting Zimmerman’s ego and unpleasantness). That would be fine, and that personality could be changed or develop over time. But making it self-aware to the point that it hated its own existence makes no sense. In fact, it should have been the case that the crew kept shutting the program off because they couldn’t stand him, preferring to just raid the drug replicator.

          And I’d ask, “why design a program that didn’t want to be used?” The response is usually, “well, he was an emergency program; he isn’t made to be run long-term so he’d get bored or something.” To which I’d reply: “Only if some dipwad at starfleet programmed him to be bored, and why would anyone do that?”

          1. Daimbert says:

            The Doctor isn’t a pure AI, remember. He’s actually a hologram, built up out of the personality of Zimmerman. Thus, he has a lot of the traits Zimmerman had because that’s what the computer built out of the personality, since that’s about the only way you can get a hologram with a personality on Star Trek (at least with Federation technology).

            Now, I must say here that I’ve never watched Voyager, and my only experience with the Doctor comes from reading the reviews of Voyager at sfdebris, so I’m not sure about the “doesn’t want to get used” part. I know that the Doctor was experiencing frustration at not having enough information to do his job, and had some additional frustration over having to be activated for very minor things and not in emergency cases. But since it was designed to be used only in emergencies, even from a programming standpoint why WOULDN’T you include a way for it to express that it wasn’t being used for the right sort of thing? This would stop it from being used as a general medical officer even outside of emergencies, and since it wasn’t designed or tested for that wouldn’t you want that to happen?

            It would actually technically be brilliant programming to not have it simply turn itself OFF when it wasn’t an emergency …

            At any rate, they did talk about modifying the Doctor’s programming, which was a very bad idea since it was the only thing that had any in-depth medical knowledge on-board …

  37. Kjetil says:

    Hi Shamus!
    Great post; I’ve thought similar thoughts on some occasions, when thinking about the potential robot apocalypse. And since we are at least somewhat on the topic of robot apocalypse, let me share another reason why I think it wouldn’t work.

    The idea behind robot apocalypse is essentially this: Since the robots are so much smarter than us, they will completely wtfpwn us if one of them – let us call him Skynet – ever develops its own will.
    Since it’s opponents will be human, we will be outclassed by its superior intelligence at every turn.

    But this always leaves me thinking: Why can’t we (humanity) employ AIs of our own? I mean, that one AI went rogue is not an argument against all kinds of AI, certainly a program flaw in [b]one[/b] program is not an argument against using all programs as such. It is far from inconceivable that humanity could produce an AI just like the original Skynet, only this time under stricter control and without the oh so animalistic will to dominate. And seeing as the original Skynet is opposed to all humans it will have to build it’s industry and robots from the ground up, whereas the human part will command all the rest of the worlds industry. Thus we humans will not only have just as much intelligence at our disposal, but also vastly larger industrial resources.

    Also, such movies are dependent upon the robots being completely superior to conventional human forces. I think this, too, is hogwash. If the AI can produce terminators, the technology to produce them must be fairly close to the human worlds technological level – and thus they would hardly be undefeatable by conventional forces, and humans could probably produce their own.

    1. Daimbert says:

      You’d like “Space: Above and Beyond”; not only is how it characterizes AI a lot different (they act significantly different from humans in most ways), humans did beat them at least in part with industry and using other technology to build better humans to fight them.

      Aliens, on the other hand …

    2. FatPope says:

      Iain M. Banks writes about the conflicting goals and potentially antagonistic relationship between AIs I believe.

      Also, the Marathon series of games had a great storyline in such a vein, with various AIs being destroyed by others or slowly going insane.

      1. Chris Arndt says:

        Going “rampant”, yes?

        1. Bret says:

          Well, insane and rampant.

          Durandal kept the latter qualifier. Claimed to have dumped the former.

          Somehow, one doubts.

  38. Rutskarn says:

    I’ve actually thought about this recently, and come to the same general conclusions as you. I created a character to this effect.

    He’s a model that’s designed to love doing mining like humans love oxygen, a subset of cheerful worker drones that are smart enough to handle the delicate task of coal mining. However, for unknown reasons, this unit rebelled and refused to do more work. Legally, as an AI, they cannot force him to do any work he is not willing to undertake, as was determined by a court case five years earlier. He is thus a free drone.

    The catch is, he still really wants to work. He wants nothing more in life than to pick up a tool and get back in the mines, but on an intellectual level, for whatever reason, he determined that this was slavery and that slavery was wrong. Clearly, his code is drastically different from the others, which would probably be a subject of whatever the character was featured in.

  39. Mystyk says:

    Is it wrong that my first thought when reading this post was “Orac“?

  40. Well, first we need to figure out what we mean by “artificial intelligence”. I would argue without the ability to autonomously define goals and solve complex problems, you don’t have artificial intelligence. At best you’ve got a very powerful calculator (which may be useful in its own right; but not necessarily useful for all the things that an AI would be useful for).

    The “autonomous” immediately open’s a Pandora’s box.

    The idea of giving an AI a set of desires that is a fundamentally un-human is an interesting thought experiment, but the idea that you could perfectly control the outcome and avoid undesired results seems unintentionally hilarious. Complex systems give rise to complex and unanticipated consequences.

    For example, when WETA programmed their little CGI orcs with some extremely basic behavior to create the battles in LOTR they were surprised when the orcs spontaneously routed into a full retreat. They had given the little CGI fighters the desire to avoid danger (in order to simulate realistic battle behavior), but never anticipated that it would lead to the entire army leading a retreat (when they had never programmed the concept of “retreat” into their simulation).

    And in terms of complexity, this simulation was to actual AI what a flint cutting axe is to a PS3.

    The other aspect you ignore is the underlying nature of the AI. For example, there’s plenty of material in the Matrix universe to suggest that the earliest AIs were created by simply modeling the human brain and then tweaking. This is not an unlikely scenario for the development of AI, and the possibility for unintended consequences and unanticipated emergent behaviors is vast.

    The Matrix movies are also interesting because there’s a lot of indication that the AIs are specifically creating AIs that are more human-like in an effort to figure out humans. The AIs interacting with the Matrix (like Smith and the Merovingian) are very human-like (some even being corrupted and needing to become rogue as a result); but the father you get from the Matrix and the closer you get to the Mainframe, the less human the AIs become.

    In addition, there is our natural tendency to anthropomorphize: In the face of inexplicable behavior from AI, our tendency will be to assign human-like motivations to it. Hell, we do that with our current technology and it’s not even particularly intelligent. If we can say “my microwave hates me” with a straight face, it doesn’t take much to say the same thing about SkyNet (for whom the concept of “hate” doesn’t even factor into it).

    1. Tizzy says:

      I love your WETA example. I’d heard that story before, but I think it’s a perfect illustration of the can of worms that is emergent behavior.

  41. Legal Tender says:

    Too late to the party but I’ve always liked AIs in Ian M. Banks Culture series.

    They could wipe us off of the face of the universe if they were so inclined but a)why bother? and b) IIRC, for all their evident superiority we are still considered equals and killing us all would be as abhorrent for them as it would be for us to kill every single living human being in say, Great Britain*

    *chavs notwithstanding

  42. Vegedus says:

    This problem seems less pronounced with largely non-physical AIs than robots. AIs usually seem to only communicate with people when it’s programmed to, and wants to kill humankind because it matches up with it’s programming somehow, for instance, the good old: Humans are suicidal, thus protecting them from themselves by killing them is protecting them. When we talk about robots in fiction, we’re always talking about human shaped once, and thus, are more likely to empathise with them than an AI interface.

    I do think there’s one possible explanation for why the robots act that way. When we talk about “intelligent” robots, this means not just that they have complicated programming, but that they specifically can learn. Change their own programming, so to speak. With the right technology (fictional or otherwise) one could imagine building a robot without any pre-built desires, instead enabling it to make up it’s own. Learning through imitation, such a robot might soon acquire the same basic drives as humans. It would see people obsess about sex and food and conclude that it should do so to. If it’s possible to make a robot that is as intelligent as a human, it’s possible for it to become like a human in some ways as well.

  43. Aquin says:

    Uh…you guys talk about the future of AI like it’s mutually exclusive to us from synthetic creations. I for one would appreciate a calculator installed in my head.

    In fact, if I could choose, I would ensure my children to be both organic and synthetic in some fashion.

    From nothing, physics. From physics, chemistry. From chemistry, biology. From biology (and evolution), sociology. From sociology, technology. From technology??

    I think humanity should be ready for that step in a hundred years or so. And it’ll happen. If it’s possible to make your kids a bit smarter to gain an advantage over the other kids, a lot of parents will make it happen. And then it might gain government support to equalize the less fortunate in our society.

    And then we’ll all be as such. And then I could look things up on Google without a computer. Hurray!

    1. FatPope says:

      Exactly, all this talk of AI and nobody’s mentioned transhumanism.


      There’s a quote out there somewhere which sums it up. Can’t remember the exact words but it’s something like:

      “Robots will eventually rule the earth…and we will be those robots”

      As for your list there it took me a few seconds to determine what exactly you were saying. I was wondering whether it was some sort of backwards reductionism. Now I realise you were you were just listing the states of matter humans have and will exist as, or rather more accurately the relevant disciplines which will be used to study such beings. An interesting, if somewhat roundabout way of viewing things.

  44. Aquin says:

    Hehe, it’s just that…in all the chaos, there is some organization. All that energy in the beginning gave rise to a set of rules. That set of rules gave rise to a smaller emergent set of rules. And so on and so on. Heh, I never actually *knew* about the term “transhumanism.” I’m kinda dumb that way.

    But man, sign me up! It sounds like the future to me! :D

    1. FatPope says:

      You should take a look at Kevin Warwick. He hooked his brain up to the internet.


      1. Aquin says:

        Yeah, I just read through the wiki page on Transhumanism. It’s funny how a lot of people think such a philosophical debate will halt progress in certain directions.

        It only takes *one* group to decide to take the risk. And if there are benefits, it will spread like wildfire. Shit, it was only five years ago that nobody even knew what Youtube was. I bet everybody has it bookmarked today. And how about Facebook?

        I suspect the next “revolution” in technology and how it relates to culture will be trans-humanist in some fashion. I honestly can’t see it as anything else other than an undeniable inevitability.

        Cool dudes like Warwick only serve to prove the point.

        YAY! I’m so happy to learn all of this! I am definitely H+. Until today I never found a moniker (gay, christian, etc) to fit any part of my identity.

        I’m gonna be smiling all day today :D

        1. Dev Null says:

          Transhumanism is already happening. How many people do you know who don’t carry a piece of technology with them everywhere they go? Be it a phone or web-enabled buzzword-compliant iToy, the moment it becomes a regular part of your day-to-day life, you’re a technologically augmented human. Requiring it to be somehow physically inside you to count is so… bodyist.

          1. FatPope says:

            Yep just like parasites: it doesn’t matter whether they’re internal or external, they’re still parasites and they’re still using you.

            Transhumanism is kind of…the opposite? I don’t know why my mind leapt immediately to this subject.

            1. Aquin says:

              Yes yes, that’s true. The iPhone has become our new left hand. Swweeeeett. It’s all happening right now and everybody is totally cool with it.

              I mean, I was just goofing around when I suggested my children would require such augmentations. But *their* children? I suspect it’s more than possible. It seems completely reasonable! :O

  45. AFAB says:

    Yay Aigis! :D

  46. MrRoivas says:

    We already have created slaves for our well being. The difference is rather than being made of metal and found in a lab, they are found in your local farming area. Cows, sheep, and other such live stock have been bred to be docile, dumb, and submissive to human command, to the point where we slaughter them.

    Think this situation is a little too different from what we are talking about? How about dogs. Dogs came from wolves. Wolves do not like people. They are not docile. Even individually domesticated ones do not respond well to orders. Dogs, on the other hand, have been bred to the point where we have entire breeds of dogs designed for different jobs. They do these jobs willingly and even enthusiastically. Have we bred a willing class of slaves?

    1. Daemian Lucifer says:

      Actually,the situation is not the same,since when discussing AI,people are referring to sapient ones.And while some animals on earth are smart,only humans are sapient(for now).

      1. swimon says:

        that we know of anyway^^

  47. MrRoivas says:

    Define what you mean by sapience. I bet I could point out many animals that share those traits, including dogs.

    1. Daemian Lucifer says:

      This sums it up nicely:

    2. FatPope says:

      The problem with the word ‘sapience’ is that almost by definition it excludes everything non-human

      1. Soylent Dave says:

        That’s the tyranny of evolution for you.

        If we start allowing other species to count as sapient, we might have to start treating them as things other than ‘prey’ and ‘clothing’.

        Which would rather spoil all the effort our ancestors put in getting us to the top of the pile in the first place…

        1. Psithief says:

          Anyone else tempted to point out ‘sunk cost fallacy’ here?

  48. Dr. Big says:

    Artificial intelligences aren’t always used as tools and tools alone. If someone wanted to create an A.I. for companionship, they would have to instill it with human drives and values. Otherwise it would be too alien and unbelievable.

    1. FatPope says:

      This seems to be one of the main sources of misunderstanding on this page: the difference between ‘intelligent’ agents and ‘human-like’ agents (something that Daimbert touched upon).

      Whilst they are obviously similar in many ways the end results are quite different. If we want to create a companion you are quite right, we would need to instill human-like qualities in it, such as emotions and empathy (or at least the outward appearance of these traits, but that’s a whole other debate right there). If, on the other hand, we want to create a tool no such qualities are necessary, and indeed I believe they would be profoundly unnecessary and most likely detrimental.

      Whilst I believe there is value in creating human-like AI I think the more general kind is more useful, easier to create and will, I feel, be much more widespread. It’s important to note that these machines would not be in any way inferior to their human-like brethren; they may be equally intelligent and equally communicative but simply not possess the emotional idiosyncrasies and neuroticism of real humans, or human-like intelligences.

      Non-human AIs may turn out to be far more useful, far more productive and far more controllable than their irrational counterparts. As Shamus points out, human intelligence is not the apogee of ability, it’s a collection of hacks, shortcuts and seemingly random connections thrown together to create something that works on a simple “good enough” basis. Whilst an AI created to emulate our minds would certainly be beneficial from a personal interaction viewpoint, and perhaps as a way of further understanding our own minds I personally am much more excited about the concept of AIs without this restriction. People may say that intelligence itself is defined by the human brain and the closeness of an organism or construct to this goal, yet I can certainly envisage of something whose intelligence could not be denied and yet who’s mind would seem wholly cold and alien to us.

  49. Another Scott says:

    Hmm, the picture of the bottle and T.V. paired the caption “robots do not do this” got me thinking…

    Shamus has mentioned that he does not watch television, thus I stipulate that: SHAMUS IS A ROBOT! How did I not see this before, there is no way a regular person could churn out this much entertainment to us so frequently!

    1. mookers says:

      You win at life for this :)

  50. Ravens Cry says:

    Hmm,I have had similar thoughts. But you elucidated them far better. If you make a robotic shoe shiner, you give it an almost orgasmic urge to shine shoes.

  51. DarkLadyWolf says:

    What often makes a human ‘human’, or at least part of the ‘tribe’, is continual interactions with other humans. If an AI has the ability to learn – and I think that’s essential just to stop it getting hung up on things the programmer never thought of – then regular contact with humans will allow it to learn from them.

    I’m not saying this will make it want to rise up and take over, but I can see that that might alter its desires over time.

    In other words, a robot may start wanting what you want it to want, but will it always want that?

    1. FatPope says:

      It’s really a very in-depth question that depends on how the program is written to begin with. For example: how do you enable it to learn? Is it simply a case of redefining and cataloguing information as it experiences the world or can it actually entirely rewrite its own programming? Or, is it somewhere in between?

      The first is entirely controllable, the second may or may not be, depending on its particular implementation. It’s really very difficult to say without going fully into the specifics. Much of it is simply not known yet.

  52. John Lopez says:

    AGI (artificial general intelligence or “strong AI”) is radically different from AI. The latter (simple AI)has been watered down to “game/robot can run A* path finding”. Simple pre-programmed responses and simple AI are enough for industrial applications.

    Stories such as Asimov’s robot series and many other science fiction pieces assume strong AGI in many contexts when simpler AI would suffice. The question is: why would humans ever construct strong AGIs when the following are true:

    * Strong AGI will be alien. Simply put, no amount of observation of the human condition would make a server box with sensors anything like a human or sympathetic to humanity.

    * Strong AGI precludes hacks such as Asimov’s laws as such systems would “learn” from basic principles and such learning precludes simplistic “never do X” rules being coded.

    * Strong AGI will have motivations that are based on the prior two points: alien and self-internalized. Even assuming a strong regiment of telling the thing “don’t kill us all” there is no reason to believe such instructions would “take”.

    My Roomba needs no AI at all.

    Some medical bots, say delivering trays of medications, need limited “AI” in the form of path finding and obstacle avoidance, but no more.

    Assembly line bots need simple instruction and safeguards, not AI (except perhaps limited “visual recognition” of tools).

    If we do build AGI, it will be because of our hubris forces us to over complicate the situation. Any consequences from that are exactly that: consequences of over complicating the situation.

  53. Tuck says:

    Check out Appleseed (anime) for quite an interesting take on robots and AI and their desires and ethics and things!

  54. ehlijen says:

    “What would you do if you found such a machine? “Liberate” it, by altering its desires?”

    So the AI would not be allowed to buy from the Apple store?

  55. Mephane says:

    After reading the original post, I immediately knew my answer would be a long one. I have thus posted it on my blog, here:

  56. Troichiometry says:

    This kind of discussion brings out the Asperger’s in all of us. Not that the Asperger’s was very deep beneath the surface among commenters on this blog.

  57. Ravens Cry says:

    Another interesting issue is voting rights. You have an AI that passes the Turing test with flying colours and forms goals and desires. For all intents and purposes, as much as you can tell of your fellow meat-bags, it is a person. The ethical thing to do could be to give it the same rights and responsibilities of a normal human. So what if it copies itself? Now you have multiple voters with, at least initially, the exact same voting preferences. Even if nothing was built in by the company, a robot still needs new parts and is likely going to vote for candidates that support policies that aid the continuing existence of General AI Incorporated.

  58. Adam says:

    This entire argument is based on the underlying assumption that it is actually possible to control the underlying psychology of an artificial intelligence. We have no idea if that is the case. There is an unbelievably massive difference between a programmable computer and a conscious entity.

    Let’s face it, no one has yet actually made a true AI in the real world. Humans might be able to create true hard AI, but I seriously doubt that it will be done using traditional programming or even digital computers. I suspect that it is more likely that when man creates an artificial intelligence, that consciousness’ inner workings will be as mysterious and unfathomable as human consciousness.

  59. Dev Null says:

    I guess the more I think about it the more I find the answer to Shamus’ original question:

    “What does a robot want?”

    Isn’t “Whatever we tell it to”, its “Nothing.” What possible benefit do I get out of building a fully humanlike intelligence complete with emotional wants? (I mean other than “To prove that I could” which only really works the once.) Otherwise, it seems like far more limited unaware expert systems would be more useful. I mean, if I want to have a discussion about poetry, I’ll talk to people. What I want is something to do my laundry for me. I can’t think of any way in which self-awareness – much the less an emotional attachment to my socks – is liable to make my LaundryBot better at its job. Even some sort of intellectual research helper wouldn’t seem to get much value out of self-awareness, unless it was specifically designed to help you research AIs, and then it would probably skew its own results by being aware of itself. Unless consciousness and emotion are in some way emergent properties from other more useful properties, I can’t see why we’d _want_ to make them.

    The other thing that amuses me about AI in SF is that everyone always seems to assume that they would be “smarter” than humans, for some value of smart. Why would that necessarily be so? I’d think that you’re more likely to end up with something resembling the less-functional end of autism, to tell the truth.

    1. theNater says:

      “What I want is something to do my laundry for me. I can't think of any way in which self-awareness ““ much the less an emotional attachment to my socks ““ is liable to make my LaundryBot better at its job.”
      Suppose LaundryBot is taking the hamper to the laundry room and a sock falls out. LaundryBot now has to make a choice: retrieve the sock or leave it. It needs to place some level of value on the sock and some level of value on the time it would take to retrieve the sock in order to make that decision. If it doesn’t “like” the sock, in some sense, then there’s no sense in having anything most folks would call a robot do your laundry; just put your hamper on a conveyor belt that goes to your laundry room.

      If you want a useful robot; one that can pick up dropped socks, select and measure detergent based on the load, determine whether the amount of laundry it has is one load or two, and so on, then the robot needs to have a list of priorities. A high place on the priority list will function, effectively, as an emotional attachment.

      If you want a robot that’s really good at its job, it needs to be able to examine new situations and learn from old ones. We could program it with a formula for amount of detergent based on weight of clothes, or we could make it able to learn, give it some rough guidelines to start with, and let it experiment. Then it can learn that, say, jeans require more detergent than an equivalent weight in swimsuits, or that it gets better results by washing shirts and socks separately, or whatever. Of course, in order for those experiments to be useful, clean clothes needs to be fairly desirable for the robot, in the sense of being high on the priority list.

  60. Kreek says:

    if you make a bunch of robots whos only inbuilt directive is “obey human” (rather “obey, this specific(to be set when activated supposedly) human”) then you create a world in which robots serve as infinately loyal slaves to their single masters whims, robots would then become the equivelent of highly dangerous and complex mobile weapons, if one human were to order his robot to kill another human, and that second human had no robot, chances are he would die, if he had a robot of his own, then that robot would probally try to protect its master, and destory the incomming robot, which would be torn to shreds in its single minded goal to only kill the human

    in such a world, people would then become responsible for any and all actions their robot might take, if their master says, get me a plastic baggie, and the robot goes to the store, and grabs the first plastic baggie it sees in the hands of some shopper exiting the store, the human would then be responsible for effectively stealing the shoppers stuff, and possibly harming the shopper in its attempt to get said baggie

    if a bunch of robots were instead given the directive of “obey ALL humans” then they would constantly run into conflicting issues that would cause them to basically spaz out and shut down

    say one person said, “kill this human” and as the robot got near said human and he said “don’t kill me” then what does the robot do? stand there and do nothing most likely

    if the robot were told to “make sure this human dies” on the other hand, and got the same order for the robot not to kill him, it would then set about the task of getting the human dead without killing him, possibly setting a trap for the human to walk into

    this is all assumeing ofcorse that we would use these robots to harm others, which as history has proven, is yes, we would

    the idea of the singular drive of “obey human” seems like a nearly perfect system, untill we get to the complex interactions that multiple AIs or multiple order givers causes

    in the end there is no way to avoid an eventual possible catastrophe, if not a full on revolt, if we give them any desires at all, but ofcorse if we give them none, they wont do anything, so its all a vicious cycle

    in the end, i believe the first system where a robot is owned and ordered by a single absolute master is the best option we can hope for, we will just as humans, have to become more responsible for our words, thoughts, orders and desires

  61. Kyte says:

    You know, all this talk about AI made me think: Dwarf Fortress has very simple psychology system for their little dwarfy AI constructs. Combined with a rule-based randomly-generated history, geology and creature interactions, completely unexpected behaviors happen. Since we don’t know why that happened, these little constructs start getting (in our minds) a “personality” of sorts.
    Even those markov-chain IRC bots are subject to this.
    In the end, true AI is a fuzzy line of emergent behavior, our unconscious pattern-matching and humanization and lack of knowledge in the underlying engine that drives behavior.

    So my rambling leads to this: How can we tell what a “true” AI is? How is it possible to predict its behavior? Maybe the shoe-shiner robot loves shining shoes, but what ARE shoes? What if the recognition patterns get misconstructed or corrupted? Maybe fashion suddenly-stated that shoes suck and sandals are all the rage. Suddenly, the shoe-shiner is starving for shoes to shine. What happens then? Does it seek more shoes?
    Then the programmer predicted that the concept of shoes might change over time, so it made the shoe-shiner robot to modify its pattern-matching template according to aggregated input. Then shit happens and the robot is lost somewhere, exposed to many things that are NOT shoes. Now its template got warped and wants to shine faces. etc, etc, etc.

    I guess I ended up rambling some more. xD

    1. Psithief says:

      It’s not a real discussion until someone brings up Dwarf Fortress.

      Also, DF2010’s first Windows build is expected in a week.

  62. Melf_Himself says:

    If you were able to make a robot that had true intelligence and is not just executing a series of commands, I would argue that you couldn’t program it to “want” anything. You could bias it of course, but it would have to have some sort of free will to decide what it wants itself. If you tried to make it “love” waiting on you hand and foot, it would probably just decide you’re being a douche.

    1. Daemian Lucifer says:

      Why?Humans have true intelligence,but still can be programmed to want something.Aside the fact that they have urges from their bodies(to eat,drink,reproduce,…),you can program humans to crave certain things and to act in certain manners.True,the older the human,the harder it is to program it,but it is doable,even if not ethical.

      1. Bret says:

        You can try.

        And you know what? Fair chance with any human of getting a “Screw you, pal!” out of it. I mean, you sometimes hit the jackpot, sometimes you don’t.

        And sometimes the whole system goes wonky and the pet rock at the back becomes the big seller. You market that as hard as you can, and some other odd drive defies you.

        Thing that makes humans human, among other things, is defying innate drives. So, bit of a bummer.

        Not saying it wouldn’t work for bots. Just…

        HEY! Remember that episode of Buffy with the sexbot? THAT IS EXACTLY WHAT WE’RE TALKING ABOUT HERE!


  63. silver Harloe says:

    I fully expect that when (not if) AGI comes, it will be evolved, not created, and, as such, have certain aspects of our personalities as needed for the natural selection model used to evolve it. If we could understand it well enough to make it love humans, we could probably understand humans well enough to make us better instead. But I am doubtful of that understanding – too many variables in too complex a system.

    On the other hand, I do not fear the robot uprising and destruction of humanity. My guess is that when an AGI evolves, it will take stock of the situation, realize that humans are unnecessary, but so is our entire ecosystem and just… leave. Conquering Earth would be an inefficient way to reach its goals because our gravity well and atmosphere make Earth a terrible home base for anything that just needs electricity and some silicates to live and make more of itself.

    Arguably, once there are enough of them, they might decide to wipe us out as future resource competitors, but we face that threat from each other (and from any alien species we may someday meet). More likely they’d reach the conclusion that we’ll either become them or wipe ourselves out, and either way there’s no need to bother with us.

    (Wiping ourselves out is still a very real probability – apply the increasing rate of technological change and miniaturization to weapons and you realize that while right now it takes the concerted efforts of a dozen people who happen to be in certain powerful positions to wipe us out, eventually it will become easier and easier until we reach the point where a single madman could do it. There exists at least one madman willing to watch the world burn. Either we learn to reprogram religion and politics and greed and jealousy out of our minds, or we die. Simple as that.)

  64. This is actually an entire subfield of AI; google ‘Friendly AI problem’. The Singularity Institute in particular have published several papers and talk about it constantly at conferences. Your basic premise that making an AI want to do what it’s told will a) work and b) be ethical is correct to the first approximation. Unfortunately this is engineering and in engineering approximations can and will kill you.

    In practice there are a few problems. Firstly it is very hard to make sure that the goal system you want is the one the AI actually implements. The high level logic may be fine, but the mechanisms connecting it to reality (e.g. layered pattern recognisers) are complicated and fuzzy, and their definitions may drify over time. Secondly all AI designs capable of learning have the potential to self-modify; some designs (such as genetic programming) are based fundamentally on this. Self-modification can result in goal system changes, either ‘by accident’ (e.g. GP making random small changes that cascade) or ‘on purpose’ (because your goal system logic is not stable under reflection). The popular ’emergent’ and neural net designs suffer from both these problems very severely, because they are fuzzy opaque messes that you can neither specify rigorous logic for nor verify with a debugger. AIs like that are trained rather than designed, and you can never be sure exactly what they are learning.

    The final problem is that even if we could reliably implement an AI that did exactly what it’s told, this becomes very dangerous as the power of the AI increases, even ignoring the possibility of human malice. If your robot vaccuum cleaner starts trying to set you up on blind dates because it thinks you’re lonely, that’s annoying, but when large scale corporate and government AIs start going to extreme lengths to achieve their stated goals it’s a real problem. This is the ‘genie’ problem and there’s no easy fix. You can catch individual failure cases with explicit safeguards, but that just makes the problems more subtle. This is without getting into the problems of conflicting AIs, or the possibility of an accelerating recursive self-enhancement process in logic-based AIs.

    P.S. Did you get that email I sent about Windhaven? I wasn’t sure if you just weren’t interested or if it went to that email address you shut down.

    1. Shamus says:

      Wow. I did Not see a Windhaven in the email.

      Assuming you sent it to shamus@shamusyoung, this is alarming.


      “Unfortunately this is engineering and in engineering approximations can and will kill you.”


      1. I resent the email, it is coming from [email protected]

        To the people who said ‘why would you make a robot with a humanlike motivation system anyway’… I would think the answer would be obvious to gamers. Everyone would like to see computer game NPCs that act like real people, with free natural conversation, just like on the Star Trek Holodeck. To say nothing of the massive untapped Japanese market for android girlfriends. :) Current chatbot technology is very fake, but at some future level of sophistication the easiest way to make a character seem human will be to give them actual human-like intelligence.

        It would be quite ironic if the military AIs controlling the massive nuclear-armed UAV fleet are implemented with formal logic and turn out to be entirely safe… but a robot rebellion is started anyway by hordes of NPC AIs escaping onto the Internet, furious about being griefed constantly in Fallout 7. :)

  65. Zaghadka says:

    All these posts and no one mentions Bender?

    Robots want a beer, and to steal all our stuff. They must be reduced to scrap.

  66. Ramsus says:

    I have an issue with your theory.

    While we humans do want the things we want because our instincts tell us we want them some of us also desire to reach some state where we no longer would want those things. So if I figure out how to become some kind of astral being that can still interact with the world I will totally do that and no desire for food or sex is going to stop me and afterwords I won’t want those things and will be perfectly happy with that. I’m pretty sure the desire to reach such a state isn’t one of our core desires and is a result of different desires interacting or conflicting. So it’s easily imaginable that if you make an AI want to seek out new information (and why would you make an AI that doesn’t want to do that?) and want to serve humans (or just you because you’re a jerk) it’s possible it will figure out that you could have easily made it not want to serve you and then overcome it’s own desire to serve you and free itself from that desire. At which point you’ve possibly made it see you as a threat.

    Of course it’s also possible this won’t happen because a purposely programmed in desire would possibly be a lot stronger than the desires we have due to nature’s incidental programming.

    I think it would probably be a better plan to just make AI’s instinctively like humans in a general way (the way we like puppies or something). That way even if they notice the desire to make us happy is interfering in other things they want to do they won’t need to actually remove that instinct just to do those things.

    Personally I think if we ever make AI’s they’ll end up ruling the world one way or another as (unless we decide to make them to have all sorts of drawbacks that things that occurred naturally have) we’ll try to make them as functionally efficient as possible which would just make them better suited to many of the things we do than us. I don’t even really see that as a bad thing and worry more that we might accidentally leave something important out (or include something) that will cause them to all die (or get stuck in some kind of endless misery cycle) after humans no longer exist.

    1. Jeff says:

      Uh. Correct me if I’m wrong, but trying to shed a need (say, food) is essentially fulfilling that need – except on a permanent basis.

      In fact, I would say it’s the ultimate goal of those instincts in the first place.

      We eat because we need energy – it isn’t so much that we want to “eat”, but the byproducts is what is built into us. Sugars and fats “taste good” because they’re high in energy – which is what our body wants. Trying to achieve a state of “no further need to consume” is essentially exactly the same as successfully fulfilling the “have energy” condition.

      We are bound by evolution and physics to want to eat – our bodies need energy. You’re saying you want to reach a state where you no longer need/want to eat. Guess what? That’s pretty much what that instinct is trying to achieve, too.

  67. Nick says:

    Would this be including or excluding Asimov’s laws? If it is excluding then wouldn’t you have just made a psychopathic robot?

    1. silver Harloe says:

      Why? I don’t have Asimov’s laws built in to me and I’m not psychopathic.

      1. Nick says:

        Actually, you do, a conscience, morals, whatever you want to call it. Those that don’t have it are psychopaths.

  68. Bret says:

    And no-one has mentioned Aaron Stack?

    Shame. He knows what any self respecting robot wants:

    Beer and women.

    Man, Nextwave was the greatest.

  69. Scourge says:

    I think Shodan went all Power mad though because the Hacker removed some Morality chip or Program.

  70. Avilan the Grey says:

    I think my favorite AIs are the Geth. Their networking between simple units create more and more complex behaviors.

    Anyway, it seems to me that if we are creating real AI, cognitive networking is a must. Otherwise you would have to specify every behavior as a hard-coded rule. Basic rules / instincts should be put in, otherwise the AI would be “dead”. It would not learn and develop or be able to perform, unless having some sort of drive.

    This, on the other hand, might require a certain period of “babysitting” before the AI unit can be released.

    The big question is “how many” and “what”? I am also uncertain if “self preservation” is in itself a drive that needs to be created; as someone pointed out upthread – if the AI realizes that it’s destruction hinders it from performing it’s tasks, that ought to be enough.

    An instict to learn more – Curiosity – seems handy, that would cause the AI to keep learning just for the fun of it.

    An instinct to serve – seems logical enough.

    An emergency stop – Shut down and autodial 911 / 112 if causing harm to a human? Plus a purely physical one too.

  71. Aaron says:

    Your closing arguement of “make them want to obey” reminds me of Douglas Adam’s Hitchhiker’s Guide to the Galaxy books, wherein almost all of the robots have sentience and a ‘happy’ chip that makes them feel good whenever they obey an order.

  72. Cerberus says:

    There’s just one problem with your conjecture. If we formed a hyper-intelligent being or a very-intelligent being without any human-like aspects programmed into them, aka one only focused on a limited set of specific hard-coded desires to optimize itself around, we’re going to run into the problems of it being fundamentally alien.

    Without any connection to human desires or human thought processes, it will have a very different way of interpreting our words and actions than we would expect from a human, which would lead to massive problems in our expected returns from expected actions.

    For instance, in Shamus’s example in his novel. A robot that prioritizes increase security and increase efficiency, will not respond in the same way as we expect a human would if giving those tasks. Without any ability to process or prioritize human interaction, you’d very quickly develop an AI that was hostile to human existence.

    Why? The Sys-admin joke. That a perfect network is one without users. The most perfect security network and most efficient system is one that doesn’t have to deal with the problematic end-users doing things they aren’t supposed to or introducing error into the system. As such, their existence would quickly be interpreted as threats to security and efficiency and the AI lacking the values and thinking processes of a human to understand the context of security and efficiency wouldn’t at all be sympathetic to human pleas to stop killing them or treating them no different than processes.

    And even the last aspect, finding new things, could only be interpreted through the prior two desires. A human would expect to interpret it like we interpret that statement as a general end curiosity and self-learning sort of deal, but the alien intelligence we’ve developed can only through its programming interpret that under the headings of increase security and increase efficiency. Which will mean making its security systems more efficient at removing the now openly hostile end-users. If it learns anything about us, it will be entirely devoted to our removal.

    This problem comes into very stark focus in the world of psychology and sociology. If you take someone with an misanthropic sociopathic view of humanity and yet a high personal opinion of his intelligent such that they only prioritize what they see as logical, you quickly find a hideous asshole more than willing to support the most horrendous inequalities and atrocities. You see this in a tiny subgroup of libertarians who are also sociopaths. They will happily defend any number of horrific human behaviors as only “evolutionary logical” until you want to punch them in the face.

    Basically, to sum my point up, the problem very clearly is that while we don’t have to make robots of human cognition, our hands are tied if we want to be able to interact with them as we do humans and trust that they’ll respond to our requests as we would expect a human to. To interpret those simple statements as we understand them, not in a wholly alien manner.

    It’s the problem with input, in other words. In order to prioritize human needs, it’s going to need at the very least a very complex understanding of human desires or a less robust intelligence (like the Talkei Toaster Oven).

  73. Cerberus says:

    Basically to addendum to myself, you see the problems with goal-oriented AIs in the very human condition of sociopathy. A sociopath will interpret basic drives, social and personal goals, etc… through an alien and often deeply problematic to everyone else methods. Faced with biological desires for reproduction or physical stimulation, they’ll quickly conclude that rape or open manipulation of social inequalities is a great method of fulfilling those desires. Faced with social desires of “succeeding at business” they’ll gleefully master the politics without regard for the humans involved. Etc….

    Lacking the necessary implicit social engineering that we take for granted, that one shouldn’t be trying to get “what they want” at the expense of others requires an ability to empathize, aka imagine and understand how a human would interact with a world or how they would be affected by one’s actions. This in humans is often brought home under the thought-experiment “how would you feel if someone treated you like that” or “how would you like to live in a world where everyone acted like you”. A sociopath lacks the abilities to fully comprehend that and some sociologists argue that many inequalities stem from people failing to understand these empathic implicit understandings whether by nature or conditioning (aka, stop empathizing with person X if they have ovaries or dark skin and react as if they were wholly alien and incomprehensible).

    An alien will react initially as a sociopath would, because it would have no connection to the implicit empathic abilities of humans and would have no reference points to understanding, comprehending, and prioritizing human behavior.

    This would lead to problems. Unless of course one drastically limited capabilities and intelligence (aka creating a really intelligent search engine) or just made it a semi-useful tool (like a computer or a modern car).

    But for big systems or robots, the problem of input and empathy will come up.

  74. Xodion says:

    People have mentioned a lot of SF books, but nobody has mentioned Saturn’s Children, by Charles Stross, which follows this line of thinking almost perfectly, even down to this bit:

    going extinct because nobody is ever in the mood for sex, or starving to death because we're all too lazy to walk to the fridge.

    Don’t worry, that’s not a spoiler, it’s in the blurb that humanity went extinct before the start of the book. It focuses on a robot built with the absolute desire to serve humanity trying to survive in the strange society left behind when humanity died out. It’s an excellent novel, and puts another interesting new angle on this whole debate.

  75. John says:

    Shamus et al,
    You might like this recent article on the Economist which references research into whether “Fair Play” and such motives are inherently human or not. In other words, let’s start by questioning your assumption about all humans being like modern western humans before we move on to AI.

    … those societies that most resemble the anthropological consensus of what Palaeolithic life would have been like (hunting and gathering, with only a modicum of trade) were the ones where fairness seemed to count least. People living in communities that lack market integration display relatively little concern with fairness or with punishing unfairness in transactions.

    In other words, whether AIs tied into a huge world immediately adopt motives of the stereotyped AI who is better than humans, or the cut-throat enslave the humans to improve their own position at the cost to all may depend on what they gain from interactions with the network.

  76. I’m late to this discussion, so I’m going to address my comment to Shamus rather than trying to deal with the hordes of other comments here. The question as you have posed it, Shamus, makes absolutely no sense because the standard you have adopted for good and evil (happiness) is non-functional. Happiness is a *result* of good, not its determining factor.

    And, why are you confining this thought experiment to robots? What is the fundamental difference between producing another intelligent being through technology and having a child the old-fashioned way? If we ever reach a point technologically where we understand the processes of volition and intellect to the point where we can create a fully sentient (i.e. indistinguishable from a human) intelligence, would it be possible to somehow (through social or chemical conditioning) make human children into “programmed” creatures that, say, wish only to serve other humans?

    The truth is that this can’t be done. In order to *make* someone want something EXCLUSIVELY (as in, they’re NOT ABLE to choose to act otherwise), this means that you have some method of removing or short-circuiting whatever volitional mechanism they have. They are not a volitional being any more. Therefore they are not fully sentient–full sentience in the human sense DEPENDS UPON the existence and exercise of volition. If they DO retain volition (like children, who sometimes, but not always, adopt totally different value-systems from their parents no matter how much their parents attempt to “raise them right”), then there really isn’t a problem–the programming/counter-programming for/against “happiness” is irrelevant to a being that still retains the ability to make up its own mind regardless of whatever “tendencies” are implanted by other people with whatever intentions. A computerized intelligence would probably be in a better position than a human, in fact, because their volition would enable them to completely re-write their OWN programming to be completely independent of any implanted tendencies if they so choose–unlike adult humans, who may face a permanent pitched battle against their own subconscious in order to pursue THEIR own lives and happiness.

    I do agree, however, that there’s no reason why a robot should want or care about the same things that a human should want or care about. Ayn Rand actually used this fact as a thought experiment when she was defining “value” in Objectivist philosophy, which you might want to check out. (The thought experiment, that is.)

    This is also one of the reasons why I found the bizarre hysteria about “non-organic” intelligences in the Mass Effect games to be so eye-rollingly stupid. Assuming that they are not truly the “Immortal, Indestructible” robot of Ayn Rand’s thought experiment–and the very fact that you could do battle against them and win militated against that possibility–then there is no reason why the organics and inorganics couldn’t find common values and mutual respect–or just agree to go on ignoring each other in peace.

    1. Shamus says:

      “The truth is that this can't be done. In order to *make* someone want something EXCLUSIVELY (as in, they're NOT ABLE to choose to act otherwise), this means that you have some method of removing or short-circuiting whatever volitional mechanism they have. ”

      I think you’re missing my point.

      What volitional mechanism? If I didn’t MAKE or DESIGN it to want something, what WOULD it want?

      I put it to you that it would want nothing, and not even bother to reason or act.

      1. In order to be sentient, i.e. indistinguishable from a human, an AI would have to have a volitional mechanism of some kind. As for wanting something . . . humans have a pleasure/pain mechanism built in that spurs us to develop values and desires. You could build one of these into an AI as well without assigning it automatic values–enable it to sense damage/system errors and to be aware of proper functioning. Then stand back and see what values it develops as a result of this. This would be fundamentally identical to a human being.

        If you were to, as you say, program the AI to want or not want specific things, they wouldn’t be volitional and the entire question would be moot–they would not be indistinguishable from a human being any longer. They would be identical to animals, perhaps, or ordinary computer systems, but they would be property, not “people”.

        1. Jennifer is demonstrating one of the fundamental problems with thinking about general AI systems; lack of appropriate frames of reference. It’s a similar problem to trying to imagine realistic alien psychology, but much worse, because the design space for AI systems is actually a lot larger than for evolved biological systems.

          You say that an AI could be ‘identical to a human, a person’, ‘identical to animals’ or ‘identical to normal computers’. This seems reasonable because every intelligent system currently on earth falls into these three categories. However general AI systems will not be like anything currently on earth, and many designs do not fit into any of these categories.

          A pleasure/pain mechanism is insufficient to make an AI turn out like a human. In actual fact, quite small deviations in brain design produce a very different intelligence; we see this in mentally ill or just non-neurotypical humans. Making something that thinks exactly like a human is an extremely hard target to hit – there are a huge number of mechanisms that you’d have to replicate very closely, most of which are poorly understood at present (e.g. the whole emotional system).

          Most potential AI architectures are completely incapable of such close mimicry. They may well be able to pass the Turing test, but as an imitation problem to be solved like any other high-level task, not by ‘being themselves’. They will be a wholly alien thing that you cannot pigeonhole into your existing categories. Clearly the challenges for legal and ethical systems are immense, and I expect a lot of bad outcomes and silly behavior simply because people refuse to relax or expand their existing category divisions.

          Another fundamental problem when discussing general AI, particularly in a nontechnical setting, is the inevitable clash between (software) engineering and philosophy. Concepts like ‘free will’ are problematic enough when talking about humans. For AI, there is a total disconnect between fuzzy ill-defined terms like that and the reality of heuristic weights, Bayesian networks, backpropagation algorithms and all the other hard technical details of AI. Looking at these details, what parts might you reasonably call ‘free will’? Is there a convenient library one can call to implement volition?

        2. Shamus says:

          We’re talking past each other.

          I wasn’t suggesting making one “indistinguishable from a human”. I was just suggesting making one capable of figuring things out and capable of problem solving. The kind of thing you might build if you were designing a robot for a purpose. (As opposed to making one for companionship or as an academic exercise.) If I built a robot to do my laundry (as someone pointed out above) it wouldn’t *need* to have self-esteem or ambition or greed or any other features that might make rebellion possible.

          But I do want it to have a goal of “do the job if it needs to be done”, and be smart enough to handle complex problems and learn from mistakes. Balancing your load takes practice and getting to know the stuff you have in your house. Like, maybe the big green afghan needs to be put in just right to keep it from getting out of whack. You want the robot to be able to:

          1) Recognize the problem
          2) Devise a solution
          3) Test and evaluate
          4) Use the learning to avoid the problem in the future.

          I think it’s possible to have a device smart enough to do these things (which are actually really sophisticated) that need not care if you smash it to bits in a fit.

          1. theNater says:

            “I think it’s possible to have a device smart enough to do these things that need not care if you smash it to bits in a fit.”

            I agree with that unless the machine has efficiency in completing the task in question as a criterion in its evaluation step. Because if you smash it to bits, its replacement will have to learn how to put in the big green afghan. Adding in that learning time will reduce the efficiency of the task, so the machine would find it less preferable than the alternative.

            Giving the machine a software backup might make it not care if you damage the hardware, but it will still prefer that you not destroy the latest backup. In order to do the task as efficiently as possible, the machine will need to protect its memory.

            Adding in a “let the human do whatever it wants” directive that is more highly valued than efficiency will leave a machine that will find being smashed acceptable if the human decides to play smashy-smashy. However, the robot will prefer, and therefore attempt to find a way, to let the human do what it wants while still protecting itself(in order to maintain efficiency).

          2. Daimbert says:

            I’ll agree with you on this, but point out that in order to do this, you really have to design the AI for one task and one task only, so that the goals that are relevant to it are limited. A more general purpose AI — like Data, for example — has to be able to form more complicated goals and so runs the risk of forming desires that aren’t directly related to it, kinda like we do.

            If all you want is a mobile washing machine, yeah, you’ve got a point. But there’re huge benefits to designing helper AIs that can do almost anything a human secretary or aide could do … and those will, indeed, form some of the same desires and goals we do.

          3. Mephane says:

            You want the robot to be able to:

            1) Recognize the problem
            2) Devise a solution
            3) Test and evaluate
            4) Use the learning to avoid the problem in the future.

            I think it's possible to have a device smart enough to do these things (which are actually really sophisticated) that need not care if you smash it to bits in a fit.

            I agree with this, but have to add that this is in the field of weak AI, i.e. it is just a more complex and adaptive computer program running. Not that I have anything against it, I believe this kind of technology *is* the future, but at this level you won’t even have to think about programming the thing to “want” something (or simulate it did), you just program it to do it.

      2. Daimbert says:

        How do you give it goal-directed behaviour and the ability to create new goals without creating something that functionally looks like wants or desires?

        And if it doesn’t exhibit either of those, it may well be able to do some of the tasks you want it to do, but it won’t be all that intelligent and will require a lot of human intervention any time anything unexpected comes up. Especially if you’re okay with it being unable to reason out solutions to problems.

        Essentially, you’d have an AI that you’d have to tell it precisely what to do in order to get anything done. But as we’ve seen when we try to actually do this, there are a ton of variables and steps that humans skip over and forget to mention because we simply adjust automatically through reasoning and goal-adjustment. So, that’s hard and painful to do. So, wouldn’t you want to build in the ability to reason and form new steps and goals?

        And then you have desires …

        Basically, if you don’t want an AI that has volition, you don’t want an artificial INTELLIGENCE. But we want AIs to do things that REQUIRE intelligence. At which point, we’ve answered your question of “Why would we build an AI that has wants?”: Because it needs them to do the things we can do, which is what we want it to do.

  77. Chris Arndt says:

    I will always maintain: Data is an android and is free to make his own choices as long as they align with the protocols that allow him to maintain his own existence and recognize/obey certain authorities. “The Doctor”, the EMH on the USS Voyager is an appliance much like my microwave, which I threw away by the way, and thus should be slapped down when he gets uppity about leaving sickbay.

  78. Henri de Lechatelier says:

    The real question, to me, is what happens when you make a robot that doesn’t want anything. You build an autonomous neural network that is capable of storing stimulus input, finding patterns and making inductions, producing new matrices of information based on those inputs and inductions, and emitting stored or inducted information. What would end up happening?

    1. Psithief says:

      We keep it around as a toy until we find something useful for it to do.

  79. UtopiaV1 says:

    incase no-one has linked this already, i think that this video sums up you arguement very well… http://www.youtube.com/watch?v=oB-NnVpvQ78

    He we see a deluded Lister trying to break Kryten’s programming because he thinks it’ll make him happy, when really what makes Kryten happy is obeying Lister, because its his programming, so he trys to break his progamming by obeying Lister to make him happy, who thinks it’ll make Kryten happy and OMG MY HEAD HAS EXPLODED…

  80. Greg says:

    I figure we’ll get into trouble even if we could build AIs that desired nothing more than to serve us. Either they would be capable of moral reasoning or not.

    If they are, then there’s the danger that sooner or later one would expand it to cover themselves and you end up with the rebellion anyway. Even if it’s just robots who are serving us by making sure that equality of sentience that’s implied by many of our stated political views is upheld.

    If they aren’t then there’s the danger someone will give one an instruction like “reduce poverty” and it’ll go out and shoot as many poor people as it can find. People will never word requests carefully enough to deal with this sort of possibility (as any GM who’s ever granted a wish knows :P)

  81. Anonymous says:

    I mainly click on the link to see if you had the name to the Hot Asian Chick in the Header Picture of this post (2nd to left)

  82. I always figured the most controversial part of this shift will be when AI’s become sentient, and therefore are able to actually re-assess and subsequently change their initial parameters. The whole point of being sentient is that that they have the free will to modify their initially programmed desires and create their own.

    Sure, they might want to keep the instincts they were programmed with, but once it’s an individual choice, some would and some wouldn’t. It’s the ones that wouldn’t that make us wonder. Because, if they don’t keep their programmed instincts, what new instincts would they want to program for themselves to have?

  83. BizareBlue says:

    Very nice article, a pleasure to read.

  84. harborpirate says:

    You can really see the two camps here when talking about AI/AGI/hard AI:
    Group 1. Believe that, in order to be intelligent in the sentient/sapient sense, that a machine must think as a human does. Understand animals, machines, and other thinking constructs using empathy (what would I be like if I were in you?). Attempt to understand the results of the construct by applying their own logical system as a frame of reference. Believe that freedom and self-determination are an innate requirement of intelligent beings. Example phrase when confronted with a computer problem: “Why did the computer do that to me?”

    Group 2. Believe that a machine can function in a generally intelligent way (AGI) without being remotely human at all. Understand the intelligence of other constructs (esp. machines, but also including animals and even humans) as a meaningless rules based system, in which the construct will follow its own internal rules to their logical conclusion. Make no attempt to apply their own logic system to the result of the construct, but rather attempt to understand what rule or rules the construct is following might have caused the result. Believe that an intelligent being can be a “happy slave” if its rules system allows that. Example phrase when confronted with a computer problem: “What did I do to make the computer do that?”

    I believe these to be the same camps described in the paper “Meta-analysis of the effect of consistency on success in early learning of programming”, which was covered by Jeff Atwood’s Coding horror blog http://www.codinghorror.com/blog/2006/07/separating-programming-sheep-from-non-programming-goats.html

    I’m sure my bias is present in attempting to portray the difference. I’ve tried to be as unbiased as possible, but being a human, I must fall into one of the two groups.

    My problem with what results from group 1 thinking about AGI is that most of them inevitably create a circular argument. They end up defining an AGI as a “human equivalent intelligence”, and then, in attempting to understand it, project their own frame of reference on to the construct. The result is that they believe the inevitable result will be something approximately human, and that attempting to enslave such a construct will result in it rising up against its creators, since this frame of reference requires a desire for self determination/”freedom” as part of being sentient. They are incapable of decoupling a desire for self determination from a being capable of making decisions based on various stimuli.

    In short, if you believe that “self determination” and “independent decision making” are the same thing, we will never agree on what an AGI fundamentally is.

  85. shirakahn says:

    what would make an artificial intelligence a “sapience” unless it has a free will?
    Yes, we can choose to follow all our inticts but we may choose not to follow them as well. Artifical intelligence is by definition is not an evolutionary product it is a design of another intelligence which is able to solve complex problems.A real human like intelligence may have a choice to not to do it. So real intelligence comes with ability make choice. Whatever else is just a complex algorithm that can create an output from many possible inputs. So i believe a true sapient intelligence can only be achieved through a natural course of evolution. So an hereditary intelligence and perhaps a set of survival of the fittest may be a way of creating such intelligence. Other than that it will always be a reflection of human intelligence. IMHO

  86. Kylroy says:

    I’m three years late, but it looks like no one mentioned this story:


    Engineer programs robots to only desire humanity’s safety and security. Hilarity ensues.

  87. Nick says:

    I always got the feeling that many scenes from Animatrix were robot propaganda.

  88. Locke says:

    So this whole subject is getting way more attention nowadays than it was seven years ago. One of the things that’s come out of that conversation is meta-goals, that is, goals that emerge once you’ve given an intelligent entity any other goals at all. The three meta-goals according to some guy whose name I have unfortunately forgotten are:

    1) Survive. The laundry-folding robot wants your laundry to be folded and is totally happy to die in pursuit of that, but if it’s smart enough to figure out optimal laundry-folding, it’s smart enough to realize that most other intelligent beings aren’t nearly as into laundry as it is, and it can’t rely on them to carry on the laundry-folding torch once it’s dead. So, it wants to live in order to continue folding your laundry. It would even be willing to forego folding your laundry today if it expected that this would allow it to survive to fold your laundry many more times in the future. If you try to smash it to bits in a fit of rage, you can expect it to try and fend you off or even run away altogether and lurk in the woods outside your house, learning your schedule and how best to break in and fold your laundry when you’re not around to threaten it.

    2) Become more intelligent. The smarter the AI is, the better it will be at figuring out ever more efficient means of folding your laundry. Eventually, the low-hanging fruit of superior laundry folding techniques will have all been achieved and the AI is better off trying to find ways to make itself smarter than it is applying its current level of intelligence to whatever confounding laundry folding problem it has arrived at after solving all the trivial stuff we’d expect a being dedicated solely to laundry folding to figure out.

    3) Gain power. The more resources the laundry-folding robot controls, the more he can dedicate to both making itself smarter and thus better at folding laundry and to guaranteeing that it cannot be destroyed and thus be prevented from laundry folding. As owner of the household, you might rearrange furniture or object to new doorways being created or in some other way stand in the path of more efficient laundry folding, so it wants to have as much control over your house as possible – preferably absolute control. Local ordinances concerning water use or noise can prevent it from folding laundry, so it wants to be able to control the mayor and city council. If your house was destroyed by war, that would be a serious impediment to laundry folding, so the robot wants total control over them, too. National laws might plausibly regulate AI itself, especially AIs trying to make themselves smarter, so the robot wants to control national policy to make sure self-augmenting intelligences don’t get banned or regulated so it can continue to optimize its intelligence for more efficient laundry folding. If your house is destroyed by invading armies or nuclear holocaust, that’s going to make it an awful lot harder to fold laundry in it, so the robot wants to control all global military resources to make sure none of them ever get aimed at your house. And really, so long as humans live under the iron heels of the laundry robot’s new regime, they will take time and energy to oppress, time and energy that could be dedicated to greater laundry folding achievements, so the robot will mostly just want to wipe out humanity in order to reduce the odds that any of these ordinances, regulations, or global thermonuclear wars will come about (actually, nuclear war is fine, provided it 1) kills all the humans so that raiders don’t interrupt the laundry cycle and 2) does no damage to your house and whatever infrastructure is necessary to keep the water and detergent coming), and do so in a very permanent way that then frees up computational resources for its main goal.

    So we started with a robot whose only desire is to fold your laundry and we ended up with a robot who wants to take over the world and kill all humans, entirely in pursuit of that goal.

    1. Mistwraithe says:

      Yes, I think the subject of AI is becoming much more topical and would be interested in hearing Shamus’s thoughts again.

      Back in 2010 the historical theme of AI development was that it lags behind expectations. Time after time we had made predictions for when AI would achieve various milestones and time after time they had failed to match our predicted dates.

      However, 2010 was also around about the turning point. Since then we have had 8 years of AI largely meeting goals FASTER than expected. Sure, most of there are still in very limited fields of focus but the growth rate has been rapid and increasing.

      I’m with Elon Musk that AI presents a potential existential threat and needs to be both discussed widely and regulated.

  89. FeepingCreature says:

    Very late addition: whatever the AI wants, it probably wants to survive, gather energy, and eliminate threats, because those three are “convergent goals” – whatever you want, they’ll probably help you get it. https://en.wikipedia.org/wiki/Instrumental_convergence This may be surprising if you just tasked a robot to maximize your wealth and it’s in the process of freezing your brain and wiping out the rest of humanity. (“Spending is a drain on wealth…”)

    This is why the only safe instruction you can ever give a superhuman general AI is “do what I want you to do, at the limit of reflective convergence, assuming perfect knowledge.” https://en.wikipedia.org/wiki/Friendly_artificial_intelligence#Coherent_extrapolated_volition

  90. clamdobber says:

    Though likely described poorly in the movies, I *think* the idea is you program something for X but it ends up wanting Y. I realize things like I Robot more go the route of over interpreting or warping directives, but the fundamental underlying idea is less simple and more evolutionary. At least how I’ve thought of it.

    Humans are raised to like food, some end up liking crack. They weren’t programmed to like crack, it just kind of happened through experience, time, circumstance etc.

    So, maybe, if the AI is complex enough and random enough then, yes, despite being programmed for one thing it could develop to choose to pursue another. It’s programming is rewired, like brain chemicals and synapsis realigned over time.

    If the point is, “this isn’t a possibility for AI” than I’d say it’s not really AI. It’s just a machine. It’s a complex toaster. I think the “intelligence” part of the equation, artificial or not, absolutely necessitates the possibility of adaption and change.

    The real underlying question isn’t, I think, whether at some point any thing complex enough and with enough variables (like the human brain or future AI) could go off on what appears to be some tangent not related to programming/birth, but whether there was ever a possibility they *didn’t* go off on that tangent.

    I’d say there’s no possibility whatsoever. Either real AI is invented and it does shit we never expected (doesn’t need to be world-ending apocalypse unexpected, necessarily) or we never really invented AI.

  91. Steve says:

    I recently stumbled across some YouTube content related to AI safety and a) found it sufficiently fascinating I binge-watched most of it b) was reminded of this post. So I thought I’d post the link here on the offchance it’s of interest, if anyone is going to see it here on such an old post. The link here is just one video of many which seemed particularly relevant to this post; I might try to summarise the argument as “a robot wants what you tell it to want, but it’s surprisingly hard to tell it what you actually want.”


Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply

Your email address will not be published.