MMO Population Problems

  By Shamus   Dec 30, 2009   62 comments

Someone emailed me with a series of questions about the server limits in MMO games. The whole thing was too big to tackle in one go, but the central question was roughly:

“Why can you have millions of people viewing the same webpage, but you can’t have more than a thousand or so users in the same shard of an MMO?”

While the availability of processing power and bandwidth have skyrocketed in the last decade, that “1,000 users” number has changed very little in the same time period? (And perhaps it’s even gone down in practice.)

Welcome to the thorny problem of multiplayer worlds, which involves population density, server network throughput, client framerate, client side prediction, and a whole bunch of other variables. I’m not an expert at this by any stretch of the imagination (I usually stay away from server-side work) but I offer my non-expert overview of the problem here.

Consider you, in an MMO, standing all by yourself:

traffic_alone.jpg

As you move around the world, you send your positional data to the server. At minimum, you need to send your x, y, and z position, along with a heading to indicate which way you’re facing and some additional data to indicate how you’re moving. You might also send data indicating if you’re looking up or down, and what other things you might be doing. (Performing emotes, shooting, changing weapons, spell effects, etc etc.)

How often this is sent depends on the game in question. A fast-paced action game might have a base framerate of 30fps, which works out to you sending your positional data 30 times a second. (Or as close as your PC can come, if you’re really struggling to run the game.) An MMO might get away with a lot less. I wouldn’t be surprised if they only updated a few times a second.

In our case above, you send this positional data to the server, and in turn the server sends back its own idea of where you “really” are. If you’re lagging it’s possible that you and the server might not agree. The server is ultimately in charge, and so your game client will “correct” your position, yanking you back to where the server says you are. People call this “rubber banding”, because you appear to walk forward and then snap back to where you were a few seconds ago.

So you’re constantly sending your position, and the server is sending back your position. For the sake of simplicity, let’s just assume these two things happen at the same rate. You send one up, and get one back down. The server gets one, and sends one back out. No problem.

Now, you find yourself standing near another user:

traffic_two.jpg

You send your data, and you get back data for two. You get your position, and the other player’s position. The other player gets the same deal. One up, two down. No problem.

But the server is doing something different. It’s getting two updates, but then it must send two updates to two people. So, it gets two, but sends four. I’m sure you can see where this is going:

traffic_twenty.jpg

Twenty users. Everyone sends an update, and gets twenty in return. (Their own position, plus one for each of the other 19 people.) The server gets twenty updates, and sends four hundred – twenty updates to twenty people.

Well, obviously we’ve got a very annoying growth curve here where the outgoing updates is the square of the number of people in the environment.

We need to mitigate this problem if we want to get anywhere near making a half-decent MMO. We do this by trying to cut down on the numbers on both sides of the multiply sign:

Maximum Distance

Obviously we can implement some sort of maximum-view radius. Everyone will only see other users that are within some reasonable distance. Picking a good value is crucial, and that “good value” depends on a lot of other factors. Making this value too small means people will pop in and out of view abruptly, and two people might have trouble finding each other even in a perfectly open area. The woods might be thick with players, but if they’re spread out and I can only see people within twenty meters of me, the place can feel sort of empty. Suddenly you start to lose the “massive” in MMO.

On the other hand, people are not spread out evenly over the world. They tend to congregate in cities and around key locations. It might seem reasonable to see everyone within fifty meters when I’m out in the wilderness, but with a view distance like that I’ll be able to see almost everyone in town. If you’ve ever been to the bank or the auction house in World of Warcraft, you know that a huge number of people cluster around those areas. This optimization fails to help in dense areas, which is where most of the problem occurs.

Limit Viewable Characters

For the auction house clustering problem, you might consider putting an upper limit on how many people you can see. Perhaps if the local population gets to be extremely dense, everyone will only see the closest (say) forty people?

This will introduce a little strangeness to the world, in that you’ll have I-can-see-you-but-you-can’t-see-me issues. To help picture this, imagine a super-simple world where you only see yourself and the one person closest to you:

traffic_radius.jpg

Blue and green will each see one another and not be aware of yellow. Yellow will see blue but not green. Try adding this feature to a game with PvP and see how fast it all flies apart and gameplay turns into a morass of confusion and exploits.

Cull Extraneous Data

Some games will go to great lengths to reduce how much data is sent. For example, perhaps your vertical position is only sent when it changes. (Left 4 Dead does this.) This can let you leave out one of the x, y, z values if you don’t need it, although a feature like this would be worthless in a world without a good supply of flat surfaces.

An obvious way to cut down on data sent is to simply not send out updates for a character that’s standing still. This works really well for people who might be using the bank, auction house, or a shop (finally! a technique that helps with the in-town crowds!) but won’t be a great deal of help in the wild when people are always on the move, looking around, and firing off abilities and spell effects.

The server might decide to “throttle” updates from other users. So, the game might give me ten updates per second if I’m with just one other person, but if I’m in a huge crowd I might get updates for other people once a second. I’ll still be able to see a huge crowd without bringing the server to its knees, but everyone else will appear to have very choppy or uneven movement.

CPU Concerns

Ideally you’d want a system that scales based on what’s around the user. If they’re out in the wilderness alone, it should be aggressive at showing other players and showing them at great distances. But if they’re socializing in town on a Saturday night during a holiday event and the place is wall-to-wall with players, then the server should be working to limit how much you see. Oh! Don’t forget that the server should always try to show you people who are in your party, even if they’re a little distance away and you’re in a crowd. Also don’t forget that there’s an overhead to introducing you to someone new because it has to send a larger block of data describing their avatar. And the server also needs to propagate chat messages which, while pretty lightweight on their own, still have to be sorted by distance and zone so that everyone sees the text that they should. Don’t forget that you need to transmit data for pets and mounts. And then the killer: moving all those tens (or perhaps hundreds) of thousands of monsters around and keeping players up to date on where they are.

Okay, it’s obvious that this stuff can get incredibly complex in a hurry.

And typical of the tradeoffs that that you face in these sorts of situations: The more of these bandwidth-saving techniques we use, the more CPU we need. In a world with 1,000 or 1,500 users (a common user count for an MMO) you can end up doing an absolute ton of per-frame-per-user operations. For example, finding the closest 50 users for everyone in the world means doing a ton of distance calculations and sorting. Every one of the 1,000 users has a unique list of monsters and other users that they can see. That list needs to be updated and sorted on a regular basis. While this doesn’t need to be done every frame, it’s still a nasty multiplicative problem because for each of the 1,000 users, the other 999 need to be considered for visibility. Any number that ends up being squared is a number you want to keep a low as possible.

We have lots of great algorithms for doing these sorts of sorting and comparison operations. (A lot of Information Science theory is spent on these very subjects. Very smart people were sorting and comparing massive lists of data long before anyone ever thought about making an MMO.) But even with our fancy new whiz-bang computers and advanced sorting technology, 1,000+ people moving X times a second produce (and demand) more than enough data to tax a server. Given the multiplicative effects of crowds, a modest increase in population can lead to severe spikes in demand.

traffic_world.jpg

And in the end, do we want more than 1,000 people in the world with us at once? There is a certain “sweet spot”. You don’t want that “ghost town” feel, but you also don’t want the “smothering” effect you get in high population areas. Big cities are notorious for rudeness, small towns are famous for being friendly. This isn’t because only jerks live in cities and only nice people live in small towns. There is a psychological effect to having a press of people around us, competing for our resources and taking up our elbow room. People get impatient and edgy when they’re fighting a crowd. But if you’ve got lots of resources and lots of room, you might even find yourself eager to connect with someone else. You’re more inclined to greet them on the road and help them if you can. This behavior carries over to the virtual world.

I think that despite an MMO being “massive”, it’s actually a lot more fun and a lot more conducive to team play if you can get people to have that “small town” feel. That sense that we’re all in this together. You want enough people that players will be able to meet, interact, and form groups, but few enough that they’ll actually want to. Even if your servers have lots of power and bandwidth and your MMO could support 5,000 simultaneous users, it would probably be detrimental to gameplay to do so.

202020262 comments? This post wasn't even all that interesting.


  1. RCTrucker7 says:

    Shamus, you have a knack for breaking things down into a very simple and easily understandable format, without making the reader\listener\audience feel like they’re being “talked down” too. I really enjoy these type of tech articles you do.

  2. Jason says:

    Even if your servers have lots of power and bandwidth and your MMO could support 5,000 simultaneous users, it would probably be detrimental to gameplay to do so.

    No.. It depends on the size of the world. Azeroth is the size of a big mall, so of course 5000 would seem like too much. A world that’s closer to a real size would handle a lot more people.

    • Shamus says:

      Jason: Sure, as long as they don’t mind making a word FIVE times larger. And it would need to be really “five times as much content”, not just five times as much wilderness and monsters, because of the city clustering problem.

      That could get expensive.

  3. Abnaxis says:

    @Jason: Best-case, world size is another one of those factors that takes more computing power along the order of a squared polynomial (to control all the NPCs, etc. across a larger area, assuming the players are using the extra space, of course)–likely worse, if you can’t do like Shamus said and toss the Z-axis coordinates most of the time. More players in a bigger world becomes intractable just that much faster.

    Ergh! Beat to the punch!

    That was a very long detailed, technical explanation when the concluson is that the population caps are there is because people work better that way? :D

    Seriously, though, this is a good article–well written for the target audience when an engineer would just say “ah, it’s order x-squared with the number of players.”

  4. Arnold says:

    I’m with Jason on this one.

    Just imagine an MMO with only one, ultimate server (and ultimate client and ultimate bandwidth etc to go along with it). All of WoW’s 15 million subscribers in one world.
    Of course, this would require several things: a massive exponential increase in content–you wouldn’t want to have to wait in line to be able to find enough mobs to complete your eye-of-newt (or murloc?) thingummy quest, or to be able to buy things from a vendor. Imagine having to queue in WoW to purchase reagents or whatever–it just wouldn’t fly. Therefore, a massive increase in content (Possibly procedurally generated??? Shamus, the work you did on your procedural city might come in use again if you ever decide to work in the MMO industry… ) would be necessary.

    I think that even in places with a lot of crowds, e.g. large cities, small sub-communities will form. Therefore, I don’t think that it would be bad to have 5000 vs 1000, or even 15 million vs 1000 on one server. Back when I played WoW for a while, I got to know some people, we had a great time, but for every person I knew, there were hundreds (or thousands?) I didn’t know. Would that be any different if the theoretical ultimate server /client existed? I don’t think so.

    Edit: Shamus, you’re too fast. But still, I’m considering a hypothetical “ultimate” server. Besides, with the amount of money that I am sure that Blizzard is making, they have got to have something in the pipeline…

  5. Emkinator says:

    RCTrucker7: I really enjoy these type of tech articles you do.

    Me too.

  6. Torsten says:

    Usual solution to crowding problems is to divide the gameworld into several smaller instances or playfields. Big cities are often separated as their own playfields, much like instanced dungeons, only so that the city can hold hundreds of players simultaneously. The rest of the gameworld is a separate instance, and often also divided into smaller parts.

    There is a difference between servers for 1000 players for whole gameworld and servers for 100 000 players with several instances. A living gameworld is often easier to achieve by the latter system. Personally I choose the big servers and massively multiplayer experience that they bring over the smaller servers anyday.

  7. Volatar says:

    Very well written Shamus. That explains most MMO’s very well.

    I am however now VERY curious how the HECK Eve Online puts all their 150k+ subscribers (I think the record is 40k online at once) on ONE server. Yes, its more of a cluster than a server, but all MMO servers are that way these days.

    One shard, no instancing. How do they do it?

  8. Abnaxis says:

    I was thinking: why have caps hard-coded in at all? We have technology limitations, and that’s all well and good, but what we are really getting at is quality of service. Why not let players join and throttle updates until no one joins anymore because they don’t want to deal with the lag? Y’know, let the consumer decide instead of forcing that decision on them? If there really is a sweet spot for population desnity, let people find it on their own.

  9. guy says:

    @voltar

    I’d guess they start by breaking up the data between star systems, so that checks only occur within your system. They may even have one “shard” per system and pass people between them. From what I’ve read, the game experiences massive slowdowns when you approach 1K people doing things in a single system.

    WoW can’t do that, because zone boundaries can be seen across, and figuring out who needs to be checked for sight range probably takes more CPU power than it would save.

    The biggest part, though, is probably finding ways to simplify data sent. Does EVE have customizable ship appearances?

    @Abnaxis

    Because doing things that way is incredibly painful for the users, speaking from text MMO experience. Though not as painful as having it decide to stop talking to people in an area where too much updating is occuring.

  10. Silfir says:

    That’s because people don’t know this stuff. If the game sputters and stutters and becomes unplayable, they say “Man, this game sucks, I’m leaving”. And they’d be right too.

  11. John Lopez says:

    Eve Online works as one “instance” because each system is isolated from the updates in other systems. Furthermore, even inside a system the sensor range of ships are fairly short. Finally, because the ships dock instead of hanging around at auction houses you also have a significant percentage of your population “out of sight” at any one time.

    When ships do come together in large numbers (such as an Eve University noob swarm) you start to see the actual impact of a large number of players at once. Back in the day I played 150 in a limited area was about the limit (although I hear they improved things since).

    In other words: yes, everyone is on one “instance” of the universe… but it is rare for any number of those players to be in on place needing to see the updates described by Shamus. I also suspect that the lumbering nature of the ships and their weapons means fewer updates per second. My personal experience with Eve Online could be more called “Massively Empty space with my Mining Ship and a chat client”.

  12. Abnaxis says:

    @guy: Exactly–it sucks when servers get crowded either way, but it’s better when we let players decide what is too crowded. Further, by the way I understand online crowds work, you have two kinds of crowds–smaller crowds, where people are fighting and need updates fast, and larger crowds, where people socialize/trade and it won’t end them to have reduced performance. If you want to do the former, go to a server with fewer people on it and you won’t take a hit. If you want to do the latter, pick a crowded server where there a lot of people standing around socializing.

    Of course, idealy you would need some sort of in-game mechanisms for informing people of service quality and allowing characters to move between servers, but that really doesn’t strike me as all that difficult. Also, I don’t think people will just say “it’s slow, it sucks, I don’t want to play” because my impression of modern gamers as a whole is that everyone is getting more accustomed to looking at performance numbers like ping and latency when gaming online, so a simple readout in the corner would probably be enough to inform everyone of their QoS. And for those who aren’t accustomed, you popup a canned text message saying “this server is crowded, move if you don’t like it.” Hell, they’ve paid for it so they’re not going to quit that easy….

  13. guy says:

    @Abnaxis

    Again, speaking from experience, that just goes terribly. The readout is a new idea, but that would not make large-scale raids and PVP any less like wading through tar while watching numbers go down.

  14. Picador says:

    I know almost nothing about MMO design, but surely a lot of these server-bottleneck problems could be solved with a solid P2P architecture for such a game. All players in the same (instance/area/region/whatever) update each other’s clients, with maybe some redundant friend-of-a-friend failsafe stuff thrown in. With proper checks in place, you could even limit exploits in a PvP-style game. Then the server just needs to receive and send N updates instead of N-squared, and each client needs to send and receive N updates as well. Your client isn’t up to the task? Then only your own character’s participation in the world suffers.

  15. John Lopez says:

    @Abnaxis: I recall some early online games having such a mechanism (you could see the number of users and ping time). In the end it came down to scrolling through large lists of servers trying to find that sweet spot of populated, but not too populated and having a decent ping.

    Over time there were a few servers that were used heavily (despite the lag of being slightly too full) and the rest were idled ghost towns with small populations.

    Many of those small population servers were the hangout of some clan or another and they didn’t take kindly to strangers on “their” turf (i.e., get ready for the ganking/ griefing/ insults/ etc).

  16. Kevin says:

    Eve is up to handling 1400+ people in a single system on a single server when needed (Jita, the main auction system). That one system is on one server, but not everyone is in space, so the updates for location are limited to those in space…and that’s generally at gates or stations where there are 30 to 50 people gathered.

    However, that particular system also causes the highest database hits. It is the central market and there can be millions of hits on the database per minute (every buy and sell can result in all clients in the REGION getting updated). I’d guess the real number is closer to 50 million hits/updates to the database and clients per minute during a busy time.

    Other systems share servers but are generally less populated…however other market systems are beginning to be moved to dedicated servers as well.

    CCP (owners of Eve) want to know when major fights are being planned so that they can allocate server resources to prevent slowdowns. Given the PvP nature of the game, this is something the players should want to do as well. ;)

    One last thing, Eve’s client and server use the exact same code for figuring out where a ship should be. When they get out of sync, there is a slight problem, but since it’s the exact same engine, that only happens when things are…weird. They had to (ultimately) launch 40 of the largest ships at the same time to get the issue to reliably reproduce. (As proof that they have a sense of humor, they then created a conga line of them….)

    If you are interested, go out and watch the videos at YouTube for the Eve FanFest. CCP is pretty open about how they do things and there are a lot of round table and open discussions about certain aspects of the “behind the scenes” stuff. It’s good viewing.

  17. John Lopez says:

    @Picador: Peer to peer updates only works if your game doesn’t require trusting the validity of such updates. “Proper checks in place” made me laugh: this has been tried and tried again, but enumerating badness never works (it is like trying to write an universal antivirus). Eve’s method of server + client updates cross checking is a good compromise (just kick bad clients) but still requires the server to do the work.

    That said, for friendly PvE it could be a good solution as long as those “proper checks” on the *server* prevented griefers from dominating the landscape.

  18. 2tm says:

    @Abnaxis – While a lot of people are getting accustomed to knowing, at least generally, how performance works, very few people care enough to subscribe to a game to need to do the “oh, the game I’m paying 20$ a month for is too busy right now, I guess I’ll come back in half an hour and see if things are better” game all the time, which is what would happen. Most people would say “This game is too much trouble and always running slow because it’s constantly near capacity, I’m going to go play a game where someone else deals with the performance issues and I can just enjoy myself without stressing about server lag” and as soon as you have a “most people” statement that negatively reflects on the subscriber base of an undertaking as large as making a decent MMO you’ve lost marketability and thus profit. As much as we like to think that people will work it out on their own humans are influenced greatly by first impressions and a game that is constantly lagging or causes an even mildly significant amount of stress just to find someplace you WANT to play you’ve lost any serious dedicated interest they might have had going into the venture.

  19. Zerai says:

    @picador: The problem with a p2p system is the lack of control, this is, when I attack a monster, i send the attack to the server, it calculates the damage and tells me how much i have done (be it PVP or PVE) if it was another player PC, where is the random damage calculated? if in my PC, i can tell it to do max always, if the enemy, he can always receive least damage (or always miss if there’s dodge) and there’s probably a lot of similar possibilities

  20. Peter H. Coffin says:

    Along with location-based clustering of servers, it’s also often possible to use the client to fake a lot of location data. Do all those people in the city actually have to be real players? No. They don’t. Pastiche some names and semi-random gear together and the client can make up a few thousand “players” to populate a scene that the actual client’s never going to interact with.

  21. @Abnaxis There is also the issue of servers having limited memory and CPU. If you keep accepting players, sooner or later you’ll end up with a machine that doesn’t have enough memory left for normal operation and becomes unresponsive. Then, if it doesn’t crash by itself, it requires a forced restart. Either way, you’ve just pissed off a lot of players, and created a ton of unneeded work for yourself.

  22. Pez says:

    @Abnaxis: How many people are you going to piss off in the process? The general perception is going to be “this game runs like crap” not “this server is too full, let’s try somewhere else.”

    Especially for the players who have lots of time invested… I would think they’d be more likely to quit altogether than start over on a new server.

  23. wererogue says:

    Really well written – I enjoyed that a lot.

    tl;dr – a website is like someone handing you a sheet of paper, and then moving on to the next person, who is probably willing to wait for you to be done with the people before them. An MMO is someone having the same conversation with 2000 people who need to know what everybody said *RIGHT NOW*.

  24. SatansBestBuddy says:

    The largest number of people I’ve ever seen at one time was 50.

    It’s kinda weird realizing that those people made up 5% of the total population of the server, like the world suddenly got a little smaller…

  25. Jattenalle says:

    Since everyone seem to have forgotten it: EVE Online

    Besides, any decent piece of modern hardware is more than capable of doing a few hundred thousand distance and frustum calculations per second.
    Adding in some simple grid sorting (A player at Y coordinate 1000 will never have to be compared against a player at Y coordinate 100 if the maximum view range is 10 for example) and you can easily support thousands of players.
    With the current rise of GPGPU hardware (For cheap) being able to run tens of thousands of threads on floating point optimized chips gives you even more room for players.

    It all comes down to what you (Shamus) pointed out in the end though: Does the game world support that many players?

    And most of the time the answer is a resounding NO :)

  26. Eldiran says:

    Fascinating stuff, Shamus. Good article!

  27. pwiggi says:

    @Jattenalle: They didn’t forget it; while you were posting there were several comments about EVE :)

    There are some other single-shard games out there: Vendetta Online is one, but it has a very small player base so there are rarely any problems.

    There is also Infinity: The Quest for Earth, a procedurally generated MMO in development. I believe it is going to be single-shard as well.

    Eve, Vendetta, and Infinity all have the same advantage, though: they are *vast*. Tons of space means that people won’t consolidate in one location too terribly often, lessening the population problem.

    Jumpgate (both JGC and the upcoming Jumpgate: Evolution) are/will be single-shard as well. I’m betting they use the same “trick” as the ones mentioned above, though; lots of empty space.

  28. midget0nstilts says:

    Good points. From a network engineering perspective, there’s no real reason why we couldn’t have as many users as we want. With some clever use of load balancing and other optimization techniques (including the ones you’ve mentioned), it could work. However, I think the real reason, as you allude to in one of the comments is cost. Why make the world ten times bigger when you could have the same world on ten different servers? That would entail more art, map and possibly voice assets, not to mention more quests, more dialog, etc. There would also have to be disproportionately more hardware and bandwidth for the reasons you outline above.

    On a side note, an interesting study in population dynamics on online games would of course be the plague outbreak that happened on WoW.

  29. icairus says:

    As interesting as this article is, I knew what the essential collection of problems. I’m still left asking, though… basically the first question from your article:
    While the availability of processing power and bandwidth have skyrocketed in the last decade, that “1,000 users” number has changed very little in the same time period? (And perhaps it’s even gone down in practice.)

    The processing power and bandwidth available on the server-side have increased dramatically as well, and though I’m sure the draw distances have gone up, I doubt that’s taken up the full change.

  30. Jack of Spades says:

    Second Life also maintains all ~40-80k players on one “shard”, managing it by managing population density — no more than ~40-100 avatars per 256m x 256m region (you can see across borders) and by managing maximum sight distance. Of course, it’s a rather different beast in many ways from other MMO-type environments; all the content must be streamed from the server, since it’s all user-generated. That becomes the real bottleneck.

  31. Carra says:

    Time for some back of the napkin math.

    Let’s assume there are 100 players around you.

    You only need to know (x,y,z,id avatar). This can be done with an int16, int16, int16 and int32.

    So that’s 100 x 80 or 8000 bits, 1000 bytes. So why not send one message with 1000 bytes. It’s a very small package to send. And it scales linearly with the number of players around you.

    Of course loading the level will take a bit longer as you need to know all the ids of the items a characters is wearing to properly show him.

    • Garden Ninja says:

      (I know this is an old post, so you probably won’t see it, but…)

      It doesn’t scale linearly, though. I’m not sure that 80 bit is correct, but either way, for the purpose of your example, it’s of fixed size. Let’s call that number B, for now. And let’s call the number of users in the area U.

      Yes, the server will have to send a message of size B*U to the client. The part you are forgetting, is that it has to send that data to every client, and it has to do it ~30 times per second. That puts the formula at B*U^2, which is the same squaring problem that Shamus brought up.

      EDIT: …and, I just noticed that the next comment makes the same point. Oops.

  32. Brian says:

    Carra: you have missed the point.

    If you have 100 users, you certainly can build just one message with everyone’s position. But you have to send that out 100 times, because every user needs an update. Using your size estimate, the message is 1000 bytes, so you’re sending a total of (1000 x 100) = 100,000 bytes of data.

    When the user count doubles, to two hundred, the message gets twice as big. But you also have to send it out to twice as many people! You wind up sending (2,000 x 200) = 400,000 bytes– four times as much. It scales quadratically, not linearly.

    For a thousand users, you’re looking at (1000 x 1000 x 10) = 10 million bytes of packets to transmit.

    And note that these numbers are just for one update! You need to transmit that amount of data many times per second, to maintain the illusion of smooth movement.

    This is why MMO server farms have huge, fat, gargantunormous network connections.

  33. SoldierHawk says:

    Damn Shamus. I don’t quite understand enough about that to add anything constructive, but I will say that was quite the education. Thanks for sharing this with us!

  34. noneofcon says:

    The current record for the number of players online at the same time in EVE is 54,181.

    The EVE server, “Tranquility” can be thought of like an onion, with various layers.

    The first layer is the load balancers, which direct players to the proxy servers. The proxy servers are made up of 8 3.3gHz wolfdale blades, which hangles connections, routing and data intergery.

    Next is the “Sol” layer, which are about 60 wolfdale blades each running at 3.3ghz with 16GB of RAM. This is where all the game calculations take place (combat, mining, and everything else).

    The last layer is the database. Its made up of an IBM x3850 M2 server with 2 2.6ghz 6 core Xeons connected to 2x RamSan-400s (not sure what size) and 1 or 2 2TB RamSan-500s. (The RamSan 400s can do 400k I/O operations per second while the RamSan 500s can only handle 100k I/O operations per second.)

    Yet, even with all this big fancy numbers, people still have trouble with lag in EVE. Why? The EVE client and server are written in stackless python, and it only supports one thread per system or grid (a grid being an section of space, usually about 500km cube or so. See below for more details.)

    The star system Jita used to have heavy lag because being the central trading hub, it saw a great deal of I/O operations from trading and people moving various objects around. It was later moved to its own blade and can now handle up to 800+ people in system mostly lag-free.

    The other major area of lag is 0.0 fleet fights. Say you have a gate camp of 250 to 300 ships in space around a stargate (used to get from system to system) and attacking anyone who comes through. Another group of players want to break the gate camp and brings their own fleet of similar size or larger. So you might have 300+ people trying to load the same grid at the same time, which includes everything in that space. The game has gotten better over time, but usually just by throwing more hardware at the problem, as software changes would require a major code rewrite.

    Sources:
    http://wiki.eveonline.com/wiki/Tranquility
    http://www.eveonline.com/devblog.asp?a=blog&bid=632
    http://www.eveonline.com/devblog.asp?a=blog&bid=663

    *Grids have no set shape or size. The beginning of this can help explain it better than I can: http://go-dl1.eve-files.com/media/0905/gridfumanual2%5B2%5D.pdf

  35. Volatar says:

    I love how I very early on swung the conversation over to Eve. :)

  36. Picador says:

    @John Lopez, Verai:

    My (small) MMO experience being 100% limited to co-op play, the problems you’re describing are on the periphery of what I assumed these systems are supposed to achieve. The idea of “cheating” at an online video game strikes me as so unimaginably pathetic that I assumed it was a fringe phenomenon. And surely, even if such “cheating” were possible, nobody would go to the trouble of reverse-engineering and modifying a compiled binary client for the sole purpose of “winning” some video game on the Internet? Surely?

    Clearly, I don’t understand the nature of the beast. But then, the only times I’ve even been tempted to play a game online have been co-op situations, where I have some friends and we want to play through some game together. The idea of being in an online world populated by homophobic, cosplaying 15-year-olds is terrifying. (True story: the one time a friend and I accidentally left our L4D game open to strangers, we were joined by some hyperventilating teenager named “whoretits”.) In short: I will never be able to solve this problem, because I cannot identify the victory conditions.

  37. glassdirigible says:

    @Arnold: “Imagine having to queue in WoW to purchase reagents or whatever–it just wouldn’t fly.”
    It happened when they had the arena PTRs. More than one person cannot purchase the same item simultaneously for whatever reason. I also had it happen more than a few times buying reagents in a major city (it was one of the only people in the game that sold that particular quality in a convenient place).

    In larger populations there is also the issue of so many people vying for one’s attention so requests that are not pertaining to your interest in a large chat channel can be irritating. Especially if the person in question doesn’t communicate like an adult. I remember on more than one occasion giving snarky answers to people asking somewhat reasonable questions with abysmal grammar. If someone asked the same question with halfway decent grammar I’d immediately shoot off a response with as much information as they needed and I’d try to make sure they understood.

  38. guy says:

    @picador

    People totally do that sort of thing all the time.

  39. Vipermagi says:

    Guild Wars is quite straightforward in the bandwidth saving regard. Areas are always instanced, and the maximum party size for a normal area is 8 (12 in 2 Elite areas, 6/4 in early areas). In a town, only 100 people fit (could be more), and then a new ‘district’ is created.
    The party search system spans across all districts, so it’s not too limiting (although player competency can be).

    The largest amount of players you can have on one screen during an instance, is 24. This is either Alliance Battles, or Hall of Heroes. These game modes are designed to not cluster everyone, but without a good connection you can still see your ping double.
    In PvE I’ve managed to gather up well over 200 foes. Despite no monster having teleporting abilities, they all seemed to teleport around somehow ;) Disconnected attempting to leave the Challenge mission.
    —–

    Nice read overall :)

  40. Telas says:

    This is similar to the “if technology is so much better, why is gas mileage on cars not improving very quickly?” question.

    Answer: Because people want more than just gas mileage. The Mazdaspeed 3 has more horsepower than Magnum PI’s Ferrari 308.

    Same in MMOs. In addition to wanting bigger worlds, people want many other things that require more bandwidth, processing, etc.

    Add my voice to the “Thanks, Shamus!” chorus.

  41. Nidokoenig says:

    One thing I thought while reading this, though I don’t actually play MMOs, was that enemies with some sort of phase spell that they cast on you.
    This essentially puts you/your party/anyone in the blast area in an instance to fight the demon and whatever friends he’s stashed in his pocket universe for anything between twenty seconds and a couple of minutes, during which time only you and the guys who phased along with you need to know about each other. Make them soul-sucking and/or player race-eating nasties and you have an excuse for them to be drawn to densely populated areas.
    This would work especially well if you want to give your game an aura of malevolence and emphasise a fight for survival in a ravaged world, which I realise isn’t everyone’s cup of tea. Certainly a lot of people wouldn’t like having to get their binoculars out and scan a trading outpost to see if it was empty enough to safely run in and sell their gear.

    Alternatively, you could have different planes of the same world that players can move through somewhat freely. Similar to the Light and Dark Aether in Metroid Prime 2, except travel between those was restricted to gates.
    You get players spreading naturally across however many planes you make and only have to send data regarding people on the same plane, or who are grouped if there’s an element of scouting for enemies or other resources, and you can limit that to communication.
    The different planes can have the same shape and object placement, but use different textures for any elemental rock/paper/scissors mischief you have going on in the various planes and different actual objects(e.g. a tree could be flourishing in one area, dead in another, or on fire/seaweed/topped with a cloud/a stalagmite). You get a larger world for comparatively little effort.
    You could even try to get away with palette-swapped enemies, just a basic structure for enemies with little embellishments like auras, clouds of smoke, scales or whatever.
    If one plane passes a certain level of crowding, you can start zerging the people in it to get them to phase to another plane or die or send in phase demons to scatter them evenly. As the game grows, you can improve the servers or add more elemental planes.

    Making trading and shopping more based on, say, a guild of telepaths with small-scale teleportation powers(i.e. just enough for what you’re trading) could also help. You don’t need one big hub for shopping and an auction house, you just need a whole bunch of little shacks with one guy in them spread all over, so fewer people in the same room for the same function. More convenient for both players and the server. You could even make the guild a hive mind and save on new dialogue.

  42. Carra says:

    @Brian. I noticed that. For the end user it scales linearly, for the server it scales quadratically.

  43. Jeff says:

    I thought hardware limitations (in terms of data generation/transfer) would come into it too. Let’s say a website is 1 MB. With 1 million people reading the site, that’s 1 million MB sent out, once. If the base game spits out 1 KB/s of data, with 1 million players, that’s 1 million KB/s, continuously. That’s even ignoring that each player generates data – even if each one only sends 1 KB/m, that’s an additional 17 KB/s which then has to be sent to each player – 18 million KB/s constantly.

    Just as a thought exercise, it’s mind boggling that anyone could compare accessing a continuous data stream with accessing what is essentially a single static piece of data.

  44. Duffy says:

    As to the EVE comments:

    Technically each system is treated as an instance and as they see regular population densities change CCP assigns more and more resources to particular systems, but even then they still occasionally have to lock you out of entering a system (getting rarer). The Fleet Battle Notification system helps to get around these issues in the pvp environment. EVE’s “instances” create a continuous traversable world that shares a huge database server. That database is more impressive then their instance structure.

    If you look at older MMOs, such as EQ, they shared this design: each zone was a different instance. Hell, every WoW server actually constitutes of an Eastern Kingdoms instance, Kalimdor instance, Outland and the new capitals instance, and a Northrend instance. Those make up the traversable world, all the dungeon instances are on servers shared amongst your battlegroup (shared server cluster).

  45. Abnaxis says:

    @Felix Pleșoianu: Technically, what you say is true. In computers, there is always a hard ceiling on everything. There are only so many bytes and eventually you run out. However, I would be very surpised and disappointed if it takes more than (say) a kilobyte to represent all the data that goes into a player in the game–where they are, what they’re wearing, and what they’re doing, everything. A gigabyte would therefore be able to handle a million players without overflow. Considering the fact that these machines are not computers so much as business investments bought for thousands of dollars, I would hope we could manage to spare 1 GB for player data. The CPU, OTOH doesn’t “crash” when we ask too much, it just makes us wait while it works. So a lot of players shouldn’t force a restart, so long as we design it right and code intelligently (which really isn’t a given, I suppose).

    Okay now. Few responses to my idea, now I respond in kind. Prepare for the Wall ‘o’ Text!

    First of all, the problem as I see it: Players have expectations. They expect a fun game, which is massively multiplayer (i.e. lots of players sharing the world with them), which runs smoothly and simply. They each want these things to varying degrees–some people couldn’t care less about population density as long as they get to play in a huge world, while others want to be social butterflies, while still others want to lay the beat down on their fellow avatars until they are crowned king. CPU time required per update increases very quickly as more players occupy a server. With these limited resources, an ideal solution is to distribute the resources in a way that optimizes each individual users’ expectations–you don’t need a high FPS to chat, but you do to fight. Conversely, you don’t try to take on a thousand players at once in PvP, though a population that big is benefitial for a marketplace.

    I’m not talking about taking the cap off servers as they are now. What I am talking about would have to be done with a new game, because the way I understand most MMO servers to be set-up it just wouldn’t fit. To work, the system needs to do the following:

    1) It needs to be transparent. Users need to know in plain language how the system works and why they might be seeing performance hits, without having to troll around forums online or dig through a manual. This is probably the most difficult, as I have yet to see a game do this, ever. As soon as something becomes technical, deveopers expect users to do research to make the game work rather than bother trying to put it in layman’s terms within easy reach. Nevertheless, it should hypothetically be possible.

    However, there is a silver lining to this problem. Namely, when people start seeing a performance hit, they don’t just drop the game and forget about it. Yes, it tarnishes their first impression, but the first thing they are going to do is ask someone else what the problem is. Since the performance hit comes from there being too many people on the server, chances are there is someone around who both actually read the information given to them and is willing to help someone ignorant. I’m not naive enough to think everyone is reasonably going to go about fixing their own problems and searching for a soluton, but c’mon, when you plunk 50+ dollars down for a game followed by 15+ dollars a month for a subscription, Joe Consumer isn’t going to just toss his hands in the air and give up if the games lags a little, especially if we practically hit him on the head with information telling him why it is that way.

    2) Transitioning between servers needs to be seamless. If a server becomes crowded or if a player decided their done trading and ready to go do some PvE, they need to be able to switch to another server without losing quest progress and without having to hike to wherever they were just standing. Just a quick loading screen and they can get back to playing.

    This requirement is why what I propose would never work on most MMO systems without completely rebuilding them from the ground up. I think the real key to this step is removing the link between characters and servers. When I make a character, it needs to be stored in a separate database, to be imported to whatever server I choose to play on, not stored on the server to be lost if the server crashes. There is no reason why players ever need to be concerned with what actual physical system they are on, unless they have friends they want to meet, which brings me to #3…

    3) There needs to be a way for people to interact across servers. It needs to be obvious that when you don’t see your friend, it is because they are on a different server. You need to still be able to communicate and easliy determine how you can both get in the same place. This is already done on a few games, and many of those same techniques can be applied here.

    4) There needs to be a way to make sure those big, shiny servers we buy aren’t wasted. They are investments, which need to pay for themselves. This just basically means we need to be creative in figuring out how to offload work from busy servers to idle ones, but with instances and PvP arenas this realy shouldn’t be that difficult.

    To me, all of these seem doable with a bit of code and some (gasp!) writing, but they simply haven’t been done.

  46. MrNiceGuy says:

    Hey,

    Go see Wizard 101 (Our kids play this one).

    They solve a LOT of these issues in ways I didn’t see before.

    * All MOBS are on tracks (predefined paths) so the information can be as little as one byte per movement (Each track only has one kind of MOB). This eliminates X,Y,Z, Facting, activity, etc.
    * Combat takes place in circles, where participants cannot move. Now complex combat animations can be coded into very fixed (small) data packets. When combat is engaged all mobs and players are drawn to their combat slot in a straight line. That is, no tactical movement in combat.
    * Zones are small (not always a good thing) so number of players per zone is much more evenly distributed.

    * As far as I can tell, they have COMPLETELY eliminated any A* path finding needs. A* is the CPU killer in almost any game. They did this while keeping the MMO enjoyable.

    They still have a few lag issues in the main shopping / town area but not nearly as many as other MMO’s seem to experience.

    It’s also the one MMO that me, my wife, and kids can all play together.

    Oh: And you can move from one sever to another simply by clicking the server name. Chats are cross-server and you can teleport directly (server, instance) directly to a friend. Unless the server is 100% full (then they need to port to you).

  47. mrniceguy says:

    Hey,

    Studying that MMO more I realized they design their worlds so a Z position IS NEVER NECESSARY! There is no point on any of their maps where a player can be at the same X,Y and different Z positions. This also explains why you cannot swim under water in the game (and flying is always at the same height).

    Its a very good example of optimizing your game to fit the technical limitations rather than trying to shoe-horn something into the problem space.

    Even if you guys don’t like the game, you should study the mechanics. They are pretty darn good.

  48. Atarlost says:

    Would it be possible to use architecture to segment towns into mutually nonvisible zones that can be split among servers the same way EVE’s star systems are? Yes, loading screens are bad, but I’d rather tolerate loading screens going from the blacksmiths quarter to the alchemists quarter than lag all the time.

  49. Blurr says:

    Your statement that a client running a game at 30 fps sends data 30 times a second to the server is a false assumption. At least in CS:S, servers have a setting named the “tick rate” which is precisely how many times the server accepts input from a user per second. Most servers I play on have the tick rate set at either 66 or 100.

    • Shamus says:

      Blurr: Tick rate varies from game to game. Q3A is a 60fps game. Other games work at 30fps. I went for the low side since we’re discussing MMO’s, and I’m SURE they don’t run at 60fps.

  50. Blurr says:

    Shamus: I’m not sure you’re understanding. FPS, as you know, is the rate at which the player’s screen is refreshed. The tick rate is completely independent of the FPS (it is also constant), and does not only vary from game to game, but from server to server in the same game.

    Valve explains Tickrate as: “During each tick, the server processes incoming user commands, runs a physical simulation step, checks the game rules, and updates all object states.”

    I’m just arguing semantics here, your estimate does in fact seem completely reasonable. :)

  51. Lkz says:

    IIRC, tick rate is basically “server FPS”, or something.

  52. LachlanL says:

    @Blurr: (From the OP) “An MMO might get away with a lot less. I wouldn’t be surprised if they only updated a few times a second. ”

    I’m pretty sure he understands that MMOs don’t necessarily update as fast as the render refresh rate.

  53. […] on MMO problems, Shamus Young explains why you can’t just have thousands of people in an MMO space. You probably know this, but if you don’t, you certainly will after reading […]

  54. Adam says:

    My limited experience of Lineage II illustrated the problem perfectly. One city had what seemed like hundreds people sat around acting as vendors, but trying to walk from one to the other was like stop frame animation.

  55. Shoku says:

    Here’s a solution for cities that the people with some programming experience should appreciate: cheat. In a city crowded with the whole upper level population of a game like WoW you can’t tell where anyone is anyway and even finding people you know is something you’ve got to orchestrate through chat and landmarks-
    So why bother showing them?

    Ghost town eh? Well that’s actually what I’m proposing in a double entendre sort of way. When the city gets crowded you could literally just fill it with a haze of faceless silhouettes jumbling around the player.
    If it’s really crowded you could even have the fog slow movement and maybe a slightly more opaque one step out of the crowd and bump the character to encourage people to spread out by just not wanting to enter those coordinates.

    Since you’ve got to use the chat system to find anyone in a crowded area anyway this could be exactly the way you flag other characters for display out of the crowd, though you’d probably want to make guild members and friends automatically stand out since we can often pick people we know out of crowds.

    And you could provide a bunch of chairs and benches to let people stick out from the crowd like how ships dock in eve instead of all moving right up to the auction npc (I’m piecing that together from what I’ve heard so maybe it’s totally wrong.)

    I think if you dial the values right and get a decent artist for the crowd-smog effect it would work really well and please everyone (or as close to that as you can ever get.)


    mrniceguy: most of the npcs in WoW are essentially on tracks as they just have waypoints they go to without any obstructions in the way. Because of this they don’t walk around players or any of that. You can use about the same level of computing power to have npcs path through way-zones if you do it right.
    *Well, maybe zones isn’t the best word here since I’m talking about rectangles on the ground rather than having npcs walk all over the world.

  56. Erik Randolph says:

    I’m going through your archive of game design articles, and this one makes me wonder how MAG is resolving these issues. Being an FPS, accurate position and movement information is vital, so I can’t imagine there are many shortcuts they can take, and yet there’s hundreds of players active in the same game at the same time.

    EDIT: And oop, this is the last one in the category at this time, I didn’t notice. All the same, maybe I ought to look for some info on how they handle it.

  57. […] MMO, even though the communities aren’t always what one might call “massive”.  (Shamus wins again with a great article on that one…)  At some point, you’re going to run into someone who isn’t doing it right, and […]

  58. Great article! This was just the sort of thing I was looking for since I’m currently working on an MMO and trying to figure out server optimizations.

3 Trackbacks

  1. By The Sunday Papers | Rock, Paper, Shotgun on January 10, 2010 at 12:02 pm

    […] on MMO problems, Shamus Young explains why you can’t just have thousands of people in an MMO space. You probably know this, but if you don’t, you certainly will after reading […]

  2. By LAG LAG LAG! owe my! | MMO Download on March 27, 2010 at 2:21 am

    […] MMO Pupulation problems By Shamus, http://www.shamusyoung.com/twentysidedtale/?p=6185 […]

  3. […] MMO, even though the communities aren’t always what one might call “massive”.  (Shamus wins again with a great article on that one…)  At some point, you’re going to run into someone who isn’t doing it right, and […]

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *

*
*

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!