Morality Modeling

 By Shamus Apr 12, 2007 25 comments

Corvus is talking about morality, as judged by game engines.

I have long been an advocate of interpreting behavior normally described as Good or Evil as “beneficial to the tribe” and “harmful to the tribe” respectively. This means that a single action, let’s say stealing a wallet, will be interpreted differently depending on the perspective (i.e. tribal allegiances) of the person judging the action. For a law abiding citizen, the theft of a wallet would be considered harmful to their tribe. After all, the action represents the removal of a beneficial resource from the tribe by an external force. If that wallet held money which was going to be spent on purchases from the rest of the tribe, it would have a greater impact. People are far less concerned when an empty wallet gets stolen. To the thief’s immediate family, however, the theft of the wallet represents a gain. If the thief shares a cut of his take with a larger organization, they too benefit from his actions. In other words, people who stand to profit from the act do not tend to regard it as a harmful one.

If I could nitpick this a bit, I think the “tribe” idea is beginning to hit on what the real crux of good vs. evil is. If my buddy brings me $100 which he stole from someone who I dislike, I would still want no part of the $100. Stealing is wrong, even if the person I’m stealing from is from “another tribe” and is a jerk. Even if he’s a billionaire and will never miss a measly (to him) $100, I would still not want any part of it. Even if I knew I would never get caught and that the victim would never miss the cash, I wouldn’t want it. Most games would give me “light side” points for this, I suppose, although I hardly think this is a position of novel or heroic altruism.

You could turn around and argue that I’m simply expanding my personal perception of “tribe” to include people like this billionaire. But this hits on my main point, which is that good vs. evil could be defined by how big I think my tribe is. An evil man might have a tribe of one. (Himself.) A slightly more reasonable person might include their family. Most people would include their neighbors and people they know. So, the more “good” your are the more inclusive your tribe is. The good / evil slider in games is really a very crude tool for figuring out how big your personal tribe is.

Some people strive to extend their own “tribe” to include all humans. (While I agree with this notion, it turns out to be very hard to pull off in the real world, as you face many situations where your “fellow” tribe members hate and kill one another. It’s pretty hard not to take sides in this, which ends up kicking someone out of your mental tribe.) Some people (PETA) want to extend their tribe to include animals. Most of us balk at this idea in general, but our own pets get a pass for being members of our “family”, as it were. Marking tribal distinctions based in race is a taboo in our culture, although doing so based on political beliefs is not. Thus, saying “I think [pick a skin color] people are stupid and untrustworthy” can get you fired, but saying the same thing of Democrats or Republicans is hopelessly routine. In real life, I’m sure the system is more graduated than tribe / not tribe. Usually family ranks above friends, who rank above coworkers, who rank above employers, who rank above total strangers in your area, who rank above total strangers from someplace far away, who rank above conquering space aliens, Nazis, and telemarketers.

Having said this, I would not want to try to model all of this with a computer. Not on a bet. Sweet mercy. You’d need a quad-core machine with a couple of gigs of memory just to figure out if NPC A would be willing to lend a dollar to NPC B.

This reminds me a lot of my earlier discussion about Oblivion, and how introducing a system of fencing stolen property simply moved the AI stupidity from one point to another. In the end I concluded that you would need AI capable of investigating and solving crimes before you were free of nonsense behavior from NPCs.

I hasten to add that I don’t want to suggest that the plans for the Honeycomb engine are in any way stupid. Most of the writing he does about it has me nodding my head, “That’s a really great idea!” My nitpickery aside, I actually find his idea to be pretty compelling. It will be interesting to see it in action (uh, someday?) and to see what sort of reactions players have to it. The rules of a game world drive player behavior to a certain extent, and more interesting rules will lead to more interesting behavior, which – all other things being equal – will lead to a more interesting game.

20525 comments. Hurry up and add yours before it becomes passé.


  1. Corvus says:

    Hopefully we’ve got the underlying system design light enough that we’re going to be able to pull this off with a reasonable amount of server resources. We’ve put a lot of work into that and we think we can pull it off. To share a secret — it’s because we cheat and cut corners at every available opportunity. In other words, we’re not trying to build real social models as we only need a viable appearance of social models.

    You hit on a good point about tribal definitions. In my post I mention that there will hopefully be nine levels of tribal association, from immediate family to global citizenship (yes, that includes animals, trees, etc). How influential any given deed is depends on shared tribes. You and the billionaire victim would already be in several overlapping tribes, including at least the human tribe and the global tribe. Also, it’s possible to behave in a fashion which alienates all of your tribal associations… making you, in effect, a tribe of one. However, I’m also guessing that you personally would find yourself in more immediate tribes which eschewed the sorts of thieving behavior you describe in the example.

    Anyway, thanks for the kind words about the HoneyComb Engine. Much of my writing on it at the moment is very long range stuff, as the things immediately to hand are being nailed down in other mediums (like code). By the end of the year, we hope to have a publishable road map and time line. One of the things I have to keep reminding myself (when I try and get overly complicated with the design) is that we’re not trying to recreate reality, but present a flexible and compelling storytelling tool which provides our audience with the ability to shape personal stories that have lasting impact.

  2. Strangeite says:

    Wow, a post that appeals to my inner Philosopher, Ethicist and Anthropologist. As I read this post, I head the voices of various professors gone by screaming in my head that you are confusing three very separate and distinct ideas. The idea of tribes, or more particularly, the idea of dividing the world along lines that separate specific groups in to tribes and how individuals relate to these various tribes, is the realm of Anthropology. However, the concept of good, evil and morality do not exist in Anthropology, except as to give “us” meaning as to what “they” believe. Morality sits squarely in the realm of Ethics. And, Good and Evil is the realm of the philosopher, whether or not he believes in the concept is a different story, but it is within that field that it is debated. The nature of Academia is that never should these concepts co-mingle unless a specific field develops the idea into its own, and by then it doesn’t resemble the original concept in the slightest.

    I know this is a post about video games and that it is a stretch to connect it to the divisions in Academia. (On a related note, E.O. Wilson wrote an excellent book called “Consilience: The Unity of Knowledge” arguing that the above divisions in Academia are harmful to understanding) But the reason why I bring it up is because all of the posts about the nature of video games reminds me of a concept in Anthropology. The term is Reify or “to make real”. This term exists in the anthropologist’s toolbox to explain why certain “tribes” act the way they do and believe what they do. It shows the power of the human mind to make real various concepts despite evidence to the contrary. The best example I can give is the example my very first Anthropology professor gave. He stated that if you ask his 32 year old Navy fighter pilot son if Santa Claus was real, his son will unequivocally state yes and without hesitation. When he was a child, he came to his father and stated that he didn’t believe in Santa Claus anymore. The professor told his son, then Santa Claus doesn’t believe in you and therefore I guess you won’t be getting any presents come Christmas morning. Needless to say, the son immediately stated that he believed in Santa Claus and to this day becomes very agitated when others speak of him as a fictional character. In this professor’s family, Santa Claus has been reified. Obviously this is a silly example but it illustrates how people can make real very abstract concepts. I believe that these posts and dialogue about video game morality are in fact beginning the process of reifying them to many of your readers.

  3. Shamus says:

    Oh my gosh. Your stuff about nested / overlapping tribes was right there in your post. I read it, but somehow missed that it was exactly what I was talking about.

    I should add: Just the little glimpses I get of this thing suggests that it’s astoundingly huge. I have not read all of your archives so maybe you’ve already gone over this, but I’m curious where you’re going with it. I see you’re digging a huge foundation, and as I peer over the construction fence I can’t help but wonder what the building will be when you’re done.

    Your point about cutting corners is well taken. I’ve worked on physics engines at a couple of points and the central skill to making them work right is knowing when to simulate and when to approximate. I would rank that above knowledge of physics or mathematic skills. I think the worst physics engines I’ve seen are the ones written by math majors who knew volumes about physics, but they didn’t know when to stop simulating.

  4. Segev says:

    The big problem with modeling it -acurately- is the overlapping-but-not-nesting tribes. For instance, a saint might sit in that “global tribe” including all men, women, children, animals, and plants of every race, color, political affiliation, tastiness, and creed. But a PETA memeber who includes “animals” in his “tribe” might be a much worse person than the animal-hating bastard who includes all of mankind, if the PETA member includes none of mankind that doesn’t include “animals”.

    In all, though, the nesting tribes idea is a neat one. And really, even that would handle the psychopath who tries to only associate with the “outer ring” of tribes, as he’d alienate the rings “further in”.

  5. Corvus says:

    Strangeit, personally, I’m not conflating anything that doesn’t deserve to be conflated. My contention is that blanket morality is the unfortunate byproduct of creating a set of rules which protect the tribe. Also, academia has a very important place in my game design. I consider game design to be as important as storytelling, which is my focus.

    Shamus, if I had a dime for every time I fired off a response to someone’s post only to see they already covered my point…

    I like your metaphoric expression of our process and it’s very accurate. I have a world I want to present and when I looked at what it would take to make it stand tall, I realized that I’d be better off for the moment focusing on a very solid foundation and well defined building practices. The HoneyComb Engine has become a storytelling goal in and of its own right. One day, when it’s done enough I hope to show it off with a storyworld I call the Drachurae Cycle. It’s also about community building and seeing what other people will do with it. That’s what really has me excited these days.

  6. Strangeite says:

    Corvus: I certainly wasn’t claiming that you were mixing your apples and oranges, just the nature of this post brought back the brainwashing of a liberal arts education. If I had a dime everytime that happened…

    I must admit that I am not much of a video game player, and when I do, I tend to play mostly sims (Civilzation, SimCity, etc.) but your honeycomb engine is very interesting. I agree that game design is as important as storytelling because a poorly designed game hinders the story, while a properly desinged game merely hides in the shadows so the story can shine.

  7. The Gneech says:

    Actually, I’d like to see the “morality slider” gone and as much a relic of the past as “Pong”.

    Let’s just have the game and leave the morality out of it, PLEASE!

    -The Gneech

  8. Roy says:

    Um. How, Gneech? If you’ve got a game that lets you kill people, for example, you’re inevitably going to have people who start to question the moral implications of killing people.

    There will always be games that are unconcerned with the morality of the game-world, but I don’t think that it’s bad to have games start looking at things like morality or ethics. Some of the more interesting games I’ve played have played with the way they present moral situations.

    I find things like this really exciting. Sometimes I want mindless entertainment, but sometimes I want really compelling stories and thought provoking gameplay. I think that there’s plenty of room in video gaming for both ends of the spectrum.

    I believe that these posts and dialogue about video game morality are in fact beginning the process of reifying them to many of your readers.

    I’m not sure I understand what you mean. Are you suggesting that it’s impossible for a game to play with, examine, or illustrate moral issues and problems? Are you suggesting we shouldn’t be concerned with these things in the first place? Something else entirely?

  9. ohnoabear says:

    Personally, I agree with Gneech, not because I want morality and compelling storytelling out of my games, but because I want more of them. The morality slider becomes a crutch that ends up doing the thing I hate most in RPGs: force me to play a certain way.

    First, it attempts to measure something about my character that should be the sole domain of the player(at least in CRPGs). It provides a measure of how moral this character I’ve created to play with is within the confines of the game. As Shamus has said earlier, this rating doesn’t really match up to any moral systems once you stop playing the virtuous rescues-kitties-from-trees hero that can do no wrong.

    Second, this measuring invariably leads to gameplay rewards, forcing you to go a certain way if you want to see certain parts of the game, or get the most powerful character you can. I still haven’t done the zither quest in Jade Empire after three playthroughs (it requires you to drop the morality slider pretty close to the bottom), simply because my characters that tried to follow the Way of the Closed Fist as expounded in the game didn’t kill enough innocent people or steal from enough crippled grannies. I can’t imagine playing through either KoTOR as a neutral character, because you don’t get the insane bonuses that going all light or all dark provide.

    There are better ways to model morality in video games than one global stat that determines how moral your character is. Personally, I think a simplified version of the Honeycomb system would be nice, where characters are only aware of the moral decisions you’ve made relative to them or people they know. This will still provide for some morality modeling in the game, and prevent both the problems of adding yet another stat to level up and the annoying problem Morrowind had of banishing your from every city in the world once you committed a crime.

    You could also just provide rewards on a decision by decision basis, rather than making it a stat that persists. I’ve been playing Vampire the Masquerade: Bloodlines recently, and while it is a deeply flawed game in many respects, I like the fact that the morality system is pretty hands-off. There’s humanity, but it only gets decremented if you kill innocent people, and gets incremented if you take the good paths in certain side quests. For all intents and purposes, it exists only to keep players from depopulating the game world and as a remnant of the PnP system the game is based on. Other than that, all morality is handled on a quest-by-quest basis, and the only thing a good or evil decision impacts if the reward you get for the quest. It’s a pretty low-tech solution, but it lets you play the character you want to play and not suffer for it in terms of the game.

  10. Larry says:

    This reminds me of an essay about the “monkeysphere” that I read about a year ago. It can be found here: http://www.pointlesswasteoftime.com/monkeysphere.html

    In a nutshell, it points to an experiment that shows that any given monkey can only keep track of X-number of unique individuals that make up his or her group. Beyond that, they’re no longer “Us” and they’re part of “Them.” The bigger the brain, the larger your monkeysphere, but it’s still finite. The guy explains it much better and far more entertainingly than I do, so rather than me paraphrasing his entire essay, go check it out.

  11. With all these posts about “good” and “evil” in games, I have to wonder if you’ve played Dungeon Keeper? A very fun and interesting game that is played from the “evil” point of view. I haven’t read all the comments, so if this has been brought up I do apologize.

  12. KingMob says:

    What happens when you take out the traditional morality axis from a fantasy role-playing game?

    Check out Monte Cook’s Arcana Evolved (Monte Cook was the editor for the 3rd Ed. DMG, amongst other things, and now heads up his own imprint, Malhavoc Press). Rather than the mixed axis of good versus evil versus law versus chaos that we’re used to, characters may serve ideas such as ‘light’ or ‘darkness.’

    Are good and evil ignored in his system? They become the province of roleplaying rather than rules.

  13. Randolpho says:

    I have to say these recent good vs evil posts are quite interesting, Shamus. But I wonder… how can you and Corvus fit the concept of evil in the name of good into your model? You know… ends justify the means type stuff.

    Think of The Operative in Serenity. He is an ends justify the means type guy, and yet he is unique in that he openly acknowledges that what he does is evil. The vast majority of folks who commit evil acts in the name of good do not have the same internal insight; they still believe they are good, despite their evil actions.

    Another such example is the way the dark side is portrayed in Knights of the Old Republic 2. In that game, darksiders justify their evil as a sort of Darwinian advancement — that by culling the herd of the weak they better the herd in general. I believe Jade Empire espouses a similar philosophy for the Way of the Fist (I’ve been trying to avoid most of your JE posts for fear of spoilers, as I’ve not played it yet).

  14. Corvus says:

    ohnoabear: Our goals is to remove, or at least minimize or obscure, the “do act, get cookie” playloop. This means that there are not always specific rewards for performing a villainous act. What it means is that people’s and groups of people’s reactions to you will change as you make choices in the world. I hope that everyone doesn’t run out and try to push the scale to the extremes, as I think that would be less than fun. And yes, individuals and groups who are directly affected by your actions react more strongly to you. The approach we’re taking aims to sidestep the “one more stat” syndrome by providing you with meaningful “in storyworld” feedback.

    Randolpho, look at it this way — The Operative’s main tribal alliance was with a group that did not mind his villainous acts. In fact, they encouraged them as his behavior benefited them greatly. The side effect of this tribe’s goals was supposed to be that it would ultimately bring about universal peace, a goal the Operative revered. However, when it became clear to him how polluted his tribe was by its association with villainous elements, he couldn’t take it anymore and broke his alliance. It all fits quite nicely within the HoneyComb Engine’s model, actually. Perhaps I should create a tribal alliance outline for Mal over the weekend. That’d provide a nice concrete example of what I’m talking about… thanks for the idea!

  15. Shamus says:

    Randolpho: I’ve always viewed the slider as “this is how people see you” not “this is how you see yourself”. Although, the slider moves even when your crime goes unseen.

    Actually, the slider represents, “This is how the game judges you” which is why many people find the dang thing so annoying. Don’t moralize to me, game!

  16. Purple Library Guy says:

    I don’t think the “tribe” thing is really sufficient. Yes, it’s a powerful tool, especially when you include overlapping-but-not-nesting, but there’s more going on.
    Starting still in the tribe paradigm, there is conflict within tribes, even the most immediate ones like families. What’s the basis for our judgments of who’s right in those? And while normally, as a matter of habit, we do take the side of the tribe-portion closest in against the one farther out (e.g. our family vs. the city, our city vs. the province/state, our province/state vs. the country, our country vs. some other country) it’s not always true. Civil wars are notorious for pitting brother against brother, father against son and all that. People sometimes turn their relatives in to the law. And so on.

    Add in the fact that many of these “tribes” are voluntary. Say that so-and-so American is more or less siding with, say, Iraqi resistance fighters against his own country’s military. You can say that this is because of his membership in one of the “radical left” or the “anti-war protester” tribes. Well, great, but while that’s useful for modelling it doesn’t have a lot of explanatory power. Why did he join those tribes? Ask him, and he’ll probably talk about some sense of justice not relating to tribal membership.

    Again, in general, *why* does a person decide to extend their “tribe membership” to broader levels, particularly why do they make efforts to extend their notional tribe to the “all of humanity” level or beyond? Again, it’s generally some kind of ethical sense. It’s not like they’ve ever met most of those people, nor are they banding together for protection against aliens. People aren’t good because they join the “all of humanity” tribe. They join the “all of humanity” tribe because they’re good.

  17. Shamus says:

    Purple Library Guy: I would agree that the tribe system still breaks from reality, but this goes back to what I said in an earlier comment: You have to know when to stop simulating. If I modeled the individual ethics of all of my actors, someone else would come along and point out that they still were too unrealistic because none of them were hypocrites. None of them suffered fromm moodyness that made them occasionally irrational, like real people. None of them had false information where they misunderstood the tribal affiliation of another actor. This is why I said:

    I would not want to try to model all of this with a computer. Not on a bet. Sweet mercy. You’d need a quad-core machine with a couple of gigs of memory just to figure out if NPC A would be willing to lend a dollar to NPC B.

    The skill is knowing where to draw the line. It will be interesting to see HoneyComb applied, which is probably the only real way to know how pleasing the tribal system is.

  18. Deoxy says:

    I agree with Purple Library Guy that the “tribe” system has some serious breaks with reality, but I think it could be an enormous improvement over the current system(s).

    I think where the tribal system would fail (and could be improved) would be some kind of “tribal ethics” level – that is, my “tribe” thinks it’s wrong to steal FROM ANYONE, whether I consider them “my tribe” or not, while some “tribes” (such as certain “traveling people” groups, for a stereotypical-yet-still-amazingly-accurate example) view theft as essentially the fault of the person who didn’t keep a good watch on their stuff (that is, if you get caught, it’s your fault for failing to steal properly, and if you don’t, it’s their fault for failing to keep their stuff properly). This has little to do with “tribe” (though some groups’ ethic do take that into account – witness the mess in Durham, for instance, where the race of the people involved was all that some people needed to determine “guilt”).

    That is, I think the “tribal” system being discussed has some excellent merits but also some very strong limits.

  19. Dev Null says:

    who rank above conquering space aliens, Nazis, and telemarketers.

    In that order, presumably.

    Deoxy’s got a point. You’re talking about ways of judging moral-ness of a character, but how moral you are is just how much you stick to a given moral code – depends on what the code actually contains as to how that will look to people from a different morality. So the Communist (a real one, who followed the principles of communism not one of the many things in the world labelled as such) who observed you stealing something you needed wouldn’t care which tribe you or your victim were from, because his morality would make this act ok so long as you needed it more than your victim – this in spite of the fact that you AND your victim might consider your act to be wrong. Its the same way you get religious zealots doing horrible things _with the express intention of saving their victims, or even the world._ They are including their victim in their tribe, but their moral code is fucked different from mine.

  20. Corvus says:

    Deoxy, you’ll note that my post says we plan on doing exactly that! Some tribes are more or less tolerant of behavior which varies from their norm. In essence, each tribal unit has a “this is our code” and “this is our tolerance for deviant codes” variable. This allows us to craft a wide variety of social atmospheres. What I’m going to be interested to watch happen is the tone of a culture change over time based on its member’s actions.

    But as Shamus pointed out, we’re not trying to replicate reality here. I plan on leaving that to high end research institutions that can afford rooms full of computers. We’re not even really trying to make a simple model of reality with the HoneyComb Engine, when it comes right down to it. It would be far more accurate to say we’re crafting a model which honors and extends folkloric storytelling traditions.

  21. Thad says:

    If you would like to know more about morality (he says in an “after school special” voice) I would like to suggest “The Science of Good and Evil” by Michael Shermer, which looks at the evolution of morality and how “good” and “evil” are social constructions. (And does also talk about the different levels of “tribe” that go with family, nearby friends, etc…)

  22. Corvus says:

    Always eager to read more perspectives, thanks Thad!

  23. Yahzi says:

    “So, the more “good” your are the more inclusive your tribe is.”

    You just stole my whole take on D&D alignments…

    Curses! Foiled again!

    :D

  24. The original Privateer, and the follow-up Freelancer, have models of that sort. Of course they are Wing Commander/Starflight type games, but they do illustrate that kind of system.

  25. Tsetut says:

    I see this as a sort of venn diagram. Basically there would be several sets of rings. A PETA member could be on the 9th ring for animals, and include all of them, but the 1st for humans, and include only himself. He could then be on the 5th ring for plants, and will try to prevent them from being destroyed. That would allow the overlap that would otherwise be really complicated.

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *

*
*

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!