Project Good Robot 10: Software Engineering

By Shamus Posted Friday Sep 6, 2013

Filed under: Good Robot 58 comments

Sometimes people ask me what the difference is between a programmer and a software engineer, and my usual flippant answer is that software engineers are what programmers call themselves when they want to sound important. But that’s not entirely fair and often the distinction is worth making.

Metaphorically, software engineering is designing a bridge that won’t fall over, and programming is the act of rolling up your sleeves and laying bricks or nailing boards together to make the bridge a reality. It’s easy to see the difference between the two disciplines when we talk about it in terms of construction, because you’ve usually got two very different kinds of people doing those jobs. The designer does her thing, then the builder does his, and the next thing you know you’ve got a bridge. But when you’re building software the same person is most likely doing both jobs, and they’re most likely doing them at the same time.

Perhaps it sounds foolhardy to design a bridge while you build it. Joel Spolsky will tell you that you should design the bridge before anyone starts putting bricks on top of each other. And for the most part that’s true. Without good engineering you’ll either end up with ungainly code, or you end up doing lots of rewrites.

But you can’t write a spec if you don’t know what what you want to build or how you want to build it. This isn’t a bridge-building project, this is a skunkworks. We’re looking for fun gameplay and you can’t really appraise the fun of something until you’ve got a working prototype.

So we’re coding with no spec. This leads to interesting problems.

Most bugs are boring, but this one was funny: I was adding a feature to make robots fire in volleys instead of single shots.  I messed up the cooldown timing and the bot began shooting at me every single frame – 60 shots a second.
Most bugs are boring, but this one was funny: I was adding a feature to make robots fire in volleys instead of single shots. I messed up the cooldown timing and the bot began shooting at me every single frame – 60 shots a second.

Hm. Let’s see. I’m adding this code that will create this animated character for the player. All the controls are in player.cpp, so it makes sense to put this character-drawing stuff in there with it. I mean, one controls the position of the character and the other controls the appearance of the character. Those are basically related ideas, right?

Now what about this code that tracks player stats like shields, energy, XP, level, etc? Well, previously the only player data was the player position and momentum in player.cpp. This level-tracking stuff is arguably just more player data, so it belongs in player.cpp.

For testing purposes, I need to be able to see the player health. I’ll just have it print the raw values right to the screen. Just some quick-and-dirty rendering so I can get back to testing.

Now I’m adding code to restrict what the player can see. Well, that ties into player position data, so I guess that should go into player.cpp.

I need to see more than just player health. I need to see my supply of missiles and energy. And you know what would help? If I replaced these ugly numbers with proper status bars so I can get a feel for the values without needing to take my eyes off the action.

I want to draw a blue sphere around the player when their shields go up or down. Oh, I can just attach that to the rendering of the player character here in player.cpp.

The animated player character needs to generate particle effects as it flies around. That’s directly tied to the character, so that belongs in player.cpp.

I need to handle input from the keyboard or the Xbox controller. Well, all the controls are already in player.cpp, so this is a natural extension of that.

Taken alone, any one of these decisions is pretty reasonable. But taken together, they are madness. One source file has inputs, game mechanics, particle effects, animation code, HUD rendering, line-of-sight checking, messing with the stencil buffer, polygon rendering, collision detection, mouse controls, and half a dozen other things. Going back to our bridge analogy: We’ve successfully built a structure that can let travelers cross the valley. But this is not a bridge, it’s a giant shapeless mound of bricks. It does what it needs to do, but with no sense of order, clarity, or efficiency.

As suggested in the comments, I restored the flashlight mode, but now the flashlight is purely cosmetic / psychological. You can see all around you, but where you’re aiming is brighter. I like it.
As suggested in the comments, I restored the flashlight mode, but now the flashlight is purely cosmetic / psychological. You can see all around you, but where you’re aiming is brighter. I like it.

So now that we’ve made a mess, let’s do some proper software engineering and give this thing some proper shape. (One advantage we have over bridge-builders is that we can sometimes get away with this sort of slapdash “build first, design second” mentality.)

When you’re designing code you have to start thinking about interfaces. Not like, text boxes or buttons. I’m talking about programming interfaces. The whole point of cutting up your program into distinct sections is to manage complexity and try to control side-effects.

Side-effects is a tough problem to explain. Let me take a stab at it:

Let’s say I’m doing processing on a missile object in the game. I move it, check for collisions, and find it has just smacked into a bomb robot. So I pause processing this missile for a second to send a “you take damage” message to the robot. The robot’s health goes below zero. This type of robot is supposed to explode when it does, so it spawns an explosion. The explosion initializes itself by looking at everything within its blast radius and sending it a damage packet. Let’s see… there’s the robot that spawned us, but he’s already dead. Then we have the player, so let’s send them a damage packet. Oh! and there’s this missile in the blast radius. I’ll send it a damage packet. The missile, upon getting the damage, sees that its hitpoints are below zero and so it destroys itself and deletes itself from memory.

Ok, great. What were we doing before all that? Oh, right. We were updating that missile. But isn’t that the missile we just deleted from memory? But that means-

Would you like to check online for a solution to the crash bug you just created? No, thanks Microsoft. I’ve looked, and “online” is the worst place to look for those kinds of solutions.
Would you like to check online for a solution to the crash bug you just created? No, thanks Microsoft. I’ve looked, and “online” is the worst place to look for those kinds of solutions.

To be clear: This is a contrived and simplified example that will hopefully help you grasp this problem. I didn’t actually make this mistake. This would be a crash that would happen only when the player used a missile to kill a robot that exploded on death. Depending on how the game works, that might be hard to identify and track down. Players would just report “a crash”. After a while someone might notice that the crash only happens when using missiles, but only sometimes. (If the blast radius of the missile is larger than the blast radius of the robot, then this crash will only happen sometimes.) Eventually you’ll be able to replicate the bug and follow this snarl of logic down to where things go wrong.

When you’ve got a program with 40 different systems running (not a made up number in this case) the number of ways that different systems can interact becomes kind of terrifying. The most expedient way to solve the above problem would be for the missile to set a flag for itself saying, “I’m in the middle of processing, so don’t delete me.” That would fix this particular crash bug, but it wouldn’t fix the larger problem that these interfaces are badly designed and these systems don’t interact in a safe way.

If you design the interface properly, then when you’re writing the explosion-making code you might notice that you can’t directly destroy other objects. All you can do is send them damage packets, which they will process in their own time, thus preventing large-scale interactions.

You can take this idea even further, where individual objects are not allowed to change anything but themselves. This is what John Carmack was talking about when he got into functional programming in his Quakecon keynote. In a functional programming paradigm, an explosion couldn’t deal damage to other objects. Some super-object that owns both of them would need to offer a snapshot of the explosion to the missile saying, “Here is an explosion. Use it to damage yourself and let me know what happens to you.”

Some people suggested that it should be pure darkness where your light isn’t shining. Here is what that looks like. It might be fun for a brief gameplay section, but I think this sort of thing would wear out its welcome quickly.
Some people suggested that it should be pure darkness where your light isn’t shining. Here is what that looks like. It might be fun for a brief gameplay section, but I think this sort of thing would wear out its welcome quickly.

It sounds pretty restrictive, and it is. But languages have gradually been getting more strict for decades. As we make larger and more complex software, we need ways to mitigate that complexity and make it comprehensible. In the old days we had imperative programming. (Also called procedural programming.) I did imperative programming for over a decade before I even knew it was called that. There wasn’t a commonly used name for it because all languages were basically imperative. (At least in the areas where I operated. I never got near programming theorists or academics. I was just a kid figuring stuff out on his own, without the help of an internet.)

The imperative approach to the above problem would have program execution jumping recklessly from missile-process to robot-processing to explosion-processing, where at any moment any system could make changes to any other. Object-oriented design built some walls around those systems and forces the programmer to deliberately place turnstile doorway for data moving in and out. An impatient programmer might be writing a bit of code to make an explosion do its thing and get annoyed that they can’t just directly destroy some other object. Hey! Why can’t I call missile->Destroy ()? I know explosions ALWAYS do more than the base hitpoints of a missile, so there’s no need to waste time calling missile->Damage (N). It will be way more efficient* to just add a way to destroy the missile directly.”

* “Efficient” in this case means, “I have no idea how it might impact performance, but it’s way more convenient.

The lava level! The water level! The… lots-of-green-plants level! The purple one because shut up it’s cool!
The lava level! The water level! The… lots-of-green-plants level! The purple one because shut up it’s cool!

Object-oriented design lives or dies based on how smart your interface is. You can build a turnstile at every point in the wall, at which point the wall is no longer in your way but also no longer protecting you from dangerous levels of complexity. You’re effectively back to imperative programming, but with really screwy syntax. If you happen to get paid by lines of code, you might do okay for yourself.

Functional languages make the wall impenetrable, and will only let you build ONE door. I wouldn’t want to program anything complicated like that, but it’s an interesting approach to what is clearly a growing problem: Eventually we won’t be smart enough to work on our own software.

 


From The Archives:
 

58 thoughts on “Project Good Robot 10: Software Engineering

  1. Piflik says:

    Man…I really hate Software Engineering…I just had an entry level lecture on that very topic at University, and it is the most boring and dry thing I came across in my life. Stupid Waterfall-Model and Class-Diagrams. I can get behind Flow-Charts, they at least have some information in them, but the rest is just tedious busywork. (I guess I might change my mind, when I start working on really big programs, though)

    But these images really look great. I like the energy/shield bars at the top. And the ‘flamethrower’ is nifty XD (done the same thing many many times myself…net resetting a spawn timer after a shot…real fun)

    1. Primogenitor says:

      Class diagrams can be great when you need to understand a bunch of someone else code – at least if you’re the sort of person with a visual brain, and the class diagram is written well, and it’s the sort of program that is well represented by a class diagram (though other bits of UML can help for other types of program).

    2. Alan says:

      It’s not all like that! Software engineering at it’s core is just, “Have a plan before you start coding.” You can get away with a lot less formality. It’s entirely reasonable to describe a class’s API by just writing it as code or pseudo-code.

      A lot of the annoying stuff is a coping mechanism for very large programs with large teams which include programmers who are just punching the clock. The bad news is that describes most programming jobs. The good news is that better jobs exist, and if you’re a good programmer you can probably get them. The catch is that “good programmer” means more than “writes good code.” A lot of programmers overlook how essential good communication skills are if you’re going to make a living at it. You need to be able to communicate well with both technical people and non-technical people, two very different skills. You need to be able to write design documents and code that other programmers can easily understand. You need to be able to write documentation that non-technical people can easily understand.

      1. Volfram says:

        Software engineering at it's core is just, “Have a plan before you start coding.”Âť

        This is effectively why I will periodically stop coding and go for a walk. I never felt like I could get away with it at my last job, but since I don’t have any managers breathing down my neck right now, there’s nothing stopping me from saying “I don’t feel like doing this now.” Or more accurately, “I could totally power through this now, but if I do that I’m going to have to rewrite it three times and it’s still going to be terrible.”

        Walking to the grocery store gives me a chance to mess around with potential solutions in my head without writing a single line of code. By the time I return, I usually have some much-needed food and a good idea of what I need to write next.

        1. Elec0 says:

          This is an excellent idea. Not necessarily the walking to the store, but taking a break from coding and coming back. It’s the same with any complex problem, if you step away sometimes it’s easy to see mistakes you’ve made.

          1. Nick Powell says:

            “But I’m so close! I just need a couple more minutes and I’ll have this problem fixed!”

        2. Erik says:

          Yep, i do this quite often since i work from home 2 days a week. Just stepping away and doing some mindless chores (dishes, vacuming, that sort of thing) which dont require a lot of brainpower help me focus and organize the thoughts in my head.

        3. William Newman says:

          “Software engineering at its core is just, ‘Have a plan before you start coding.'”

          Actually it includes some other stuff too. Make your software testable. (And beyond that, make the individual subsystems of your software testable also.) To the extent practical, make your software so that small changes in the functionality of the program only require fairly small changes in the program code, not major rewrites. Factor out shared functionality so that it is only implemented once. Make sure everything which is part of the program is properly recorded. (E.g., for the love of all that is holy, put not just (the equivalent of Unix’s) .c and .h files under source code control, but also the equivalent of Makefiles and compiler options and library requirements.) And probably at least half a dozen other considerations that aren’t occurring to me off the top of my head.

          Those considerations don’t follow automatically from having a plan: they *should* be planned for, but sometimes they aren’t, sometimes because people accidentally screw up the plan, and sometimes because people don’t understand s/w engineering well enough to realize that these things are important for a good plan. I’ve seen big projects which messed them up despite putting a considerable amount of effort into planning.

    3. Canthros says:

      Flowcharts are the devil. (IMHO. FWIW, UML’s not much better.) Mostly because they’re huge and only useful at the loosest interpretation of “here are some labeled boxes and arrows”. I find pseudocode both smaller and clearer. (That said, flowcharts and UML diagrams are handy for talking to people that aren’t coders, like business people.) Class diagrams are actually fantastically useful for illustrating the relationships between types in your system, which I find doesn’t work well in raw text, at all.

      The waterfall model produces reams and reams of documentation as an intentional side effect. This is a major annoyance for developers and engineers, but much of that documentation is for consumption by users and clients. Quite a lot of it is for CYA purposes. Almost none of it will be practically useful once the project releases/is complete.

      There are better SDLC methods than waterfall, but it’s the one most often (mis)implemented at places making software. (In practice, almost nobody actually does waterfall: it’s awkward and cumbersome and puts a lot of responsibility on the client to understand what their engineers are telling them. At the same time, most places think they do waterfall, and their process may superficially resemble it.)

      Don’t get too hung up on formalism, really. Most of the stuff you’ll learn in a CS degree will be useful as an abstract foundation for later learning, but may not be directly applicable as-is.

      1. Zukhramm says:

        That looseness though, is why I like them. Or “them”, I’m not really using any formal system other than “boxes and arrows”. It seems to me pseudo-code would be hard to get an overview of and would focus too much on what the code does, which I at that early point don’t really care about.

    4. Duffy says:

      Disclosure: My degree was in SE and I favor the organizational structure of it versus the looser ideals of traditional CS majors. At my school a lot of the actual coding knowledge was the same and both majors had cross classes with the other, CS favored Linux and C++ while SE favored MS Visual Studio and C#.

      Anyways, it generally comes down to a matter of scale. Pseudo-code is great if we’re outlining how to handle a particular code loop and it’s functions, but a diagram is way better when trying to design out the interface or flow of an object from subsystem to subsystem of a complex web app. It also depends if you’re working with anyone. If you don’t need to communicate or organize multiple people a lot of it can be discarded and you can focus on good coding habits.

      I’m a little jaded as I primarily deal with SQL Databases and good SE principles can make or break a database, whereas front end code can often ‘tolerate’ looser design principles.

      Now, when you’re talking millions of lines of code using half a dozen languages and a dozen different technologies all written and permutated on by a rotating roster of 20-30 developers, those principles and diagrams come in handy.

      1. methermeneus says:

        I can’t remember where I read it (might have been something relatively silly like UML 2.0 for Dummies, but it might have been an article, or maybe a Cracked.com article), but because the human brain can only hold ~4-7 ideas in front at once, it’s not flowcharts that are evil, but giant complicated flowcharts.

        A much better approach (and, oddly enough, something that fits perfectly into OOD) is to have multiple levels of flowcharts. You have one that gives a general overview of the program’s systems with, say, 5ish points. Then you have five flowcharts, each of which shows the main parts of those parts, each with no more than 10 nodes. For really complex programs, you’ll probably wind up with a whole lot of flowcharts, but in those cases each programmer will probably only have to keep track of one or two at a time, which is much more manageable, and the project manager only has to know the top one and the ones right under it unless there’s a specific problem, in which case he just has to check which flow charts the programmers with the problem are working from.

        Of course, this is all theoretical for me, so maybe someone with practical experience will be able to contradict me or explain better. (Most of what little programming I do is more skunkworks-y like the projects Shamus posts, except not as good.)

    5. Volfram says:

      I sat through maybe two of those “design procedure” lectures, and learned that they’re mostly just buzzwords.

      Other Games Development team: “We’ll be using the Agile development procedure!”
      me: “OK, great! What’s that mean?”
      oGDt: “um…”

    6. dan says:

      “I just had an entry level lecture on that very topic at University, and it is the most boring and dry thing I came across in my life.”
      In my experience these kinds of statements are indicators that the issue is 100% a problem with the lecturer and 0% a problem with the topic.
      These diagrams the Softies use are the same as the free-body diagrams of Mechies and the circuit diagrams of the EEs and the P&IDs of the Chemies: draw a picture of what’s going on so you can understand it with your mind.

  2. Canthros says:

    Functional programming isolates side-effects, eschews writable globals, etc. It seems pretty cumbersome with complex types (as in an OO language), but using it on the internal implementation of a type can make maintenance much, much less painful (because it means less tracing to see where a thing was changed). Not without its tradeoffs, but I wish more (i.e. any) of my coworkers had been exposed to LISP or Scheme or something during their education as nascent programmers!

  3. Alan says:

    Functional programming is awesome, and every programmer should do more of it. If you’re a programmer, I recommend Carmack’s essay
    “Functional Programming in C++”.

    If you’re not a programmer: Functional programming is awesome and can make software that is provably safer than imperative or object-oriented programming. Safer in this case meaning “doesn’t crash” and “can’t be hacked.” But it sometimes slows the program down and support for lots of neat stuff like 3d graphics is usually better in imperative and object-oriented languages. To this I add: imperative programs look like recipes and are arguably more intuitive, while functional programming is not so obvious. So compromises get made. Carmack is arguing for pushing to be more functional, but not to be purely functional.

    1. Duffy says:

      I was all ready to start waving my pitchfork, but then I read the article. No one told me I was already doing functional programming, how rude! It seems like he’s really just describing taking O-O encapsulation to it’s logical end point. I was under the impression that this was just the ‘right’ way to do things.

      1. Kian says:

        He does say in the essay that “functional programming” in C++ is what everyone is taught is the “right” way to code, and no one follows in the real world.

        1. Zukhramm says:

          I wish I was taught this. At best, one time a lecturer spent five seconds saying “immutability is nice” before continuing to make the same rogue state mutable monster Java code as before.

  4. Gabriel Mobius says:

    I’m really enjoying the new interface, it looks super slick. I also notice an xp bar there at the bottom, which fills me with a sort of manic glee. I can’t wait to see what the finished product is, at this point I definitely think it’s safe to say you could sell this. I’d certainly buy it.

  5. Brandon says:

    I am currently working on a piece of legacy software that was clearly designed with some plan in mind, but not enough of a good one. It’s a really slick program in places, but in some places the design is an absolute disaster.

    Of course, it was written back when software engineering wasn’t really a thing, yet. I’m actually surprised it’s as well designed as it is.

    I really want to start working on my own code soon though. Bug fixes are teeeedious.

    1. Lisa says:

      Don’t worry. Once you start on your own code, bug fixes will be your life! ;)

  6. Software development has moved on from Speces and Waterfalls. One of the big deals is Extreme Programming (http://en.wikipedia.org/wiki/Extreme_programming) The basic gist of is that you start with a short list of what a customer wants, code it, release, evaluate it and repeat.

    The techniques focus on a quick release cycle and extensive unit testing so that you know there is a problem if you make a change to the code. Unit testing is tedious to add to an existing problem but if you do from the get go it really works and works well.

    Plus they work on what is called refactoring which are methods of that have been shown to be reliable when rewriting code.

    And then there are some more esoteric stuff like pair programming which uses the fact that having two programming working at one station often results in less bugs due to having a pair of eyes.

    However Unit Testing is the key element in all this. The idea is to do what it take to cut down on your timing of your release cycle and make changes in a way that doesn’t introduce bugs in existing code.

    Some projects have been able to redo their entire design without missing a beat.

    I have adopted various elements of this in the CAD/CAM program I maintain and develop and really works well.

    And under this the software engineer become more of a team leader that makes sure that everything gets done. For example that the programmers don’t skip unit testing. That customer feedback gets passed along properly and so on. Plus also initiating the point at which a design needs to be refactored. Basically coordinating the efforts of a bunch of programmers.

    1. Endominus says:

      That works alright for small projects, but if you’re making production code or are working in large groups, that would be a nightmare. Interdependency in modules requires a consistent interface for them, and if there isn’t a design in place, a module that gets refactored may cause every piece of code that calls it to get refactored as well, causing large knock-on effects. I’ve been in projects where we were rewriting code every week due to lack of planning or project creep.

      What you seem to be proposing is test-driven design in an Agile environment. I don’t see why creating a design document or full list of requirements is detrimental to that. Certainly, it allows for less flexibility on the part of the design team, but it also stops the client from deciding, three months into your Windows scheduling application, that he’d rather have it on his iPhone instead.

      I was not aware that pair programming was an esoteric concept. It was the first thing emphasized in my college programming courses. I had thought it was essentially standard practice, even if no one liked doing it. It can be a good way for a junior developer to learn from a senior, but it is less efficient if both programmers are proficient in their field; pair programming is like have a permanent real-time code review going on, which can get expensive.

  7. Paul Spooner says:

    Oh man, I’ve been here so many times. The problem is compounded when you’re really “in the groove” and chasing features and don’t want to put on the breaks, step back, and make a solid architecture. In your bridge building analogy, this is like building a bridge in the middle of combat. Sure, you could draw it up and make it pretty, but time is of the essence! Much easier to just drive your buldozer through the brick warehouse and push it into the river. Instant bridge!

    Until six months later when the river is in flood stage.
    I guess the moral is, don’t do combat programming?

    1. Brandon says:

      Combat programming sounds so cyberpunk, I love it.

      I want to be a WarDev.

    2. WJS says:

      Hmm. I wonder if extending the analogy to how combat bridgebuilding really works would give any insight; you build your bridge beforehand, then move it into place when you need it. The analogue to that would be keeping lots of libraries around, I guess?

  8. Mark says:

    Object-oriented programming and functional programming are both useful for game programming, but what the big boys use is entity-component systems. These can be implemented straightforwardly in an OO language, but they’re about the farthest you can get from OO design, structurally speaking. I’ll explain below:

    Everything in the game is an Entity. An Entity is nothing more than a collection of all the pertinent data relating to a single thing. This “thing” can be a single bullet, an enemy, a sound effect, each player, a camera or other element of the HUD, the level itself (though, for optimization’s sake, it’s often better to treat the environment as a special case), or anything. The structure of an ECS maps very cleanly onto a relational database, and in this metaphor, an entity corresponds to a database row.

    An entity is an agnostic, possibly even typeless container, with a GUID and all the data about the simulated thing it represents – if it has a location in the simulation, it has world coordinates; if it’s drawn on the screen, it has a texture or model, and the state of its animation; if it has AI, then it has a field containing its “knowledge;” if a player controls it, then it knows the GUID of the abstracted input device that player is using.

    Each of those data fields is a “component,” and they correspond to columns in a relational database. Neither entities nor components have behavior – all they are is data. All the behavior is located in functions, which may or may not be pure according to your needs, called Systems.

    A System can be something huge like physics, collision, and rendering, or something small like jumping and shooting, but the main idea is that each System does one thing. Each System finds all the Entities which have the Components that are of interest to that System – the renderer, for instance, only needs entities that have all the components related to rendering, and the physics system has no need for the HUD entities – and uses that data to actually do that thing (the renderer system is the function that leads to all the calls to Draw). There are benefits to making it so that each System corresponds to exactly one Component, but that’s not always practical. Systems must be able to manipulate Entities arbitrarily, because all game logic lives in systems.

    In theory, you just have your main game loop invoke all the Systems in the correct order. In practice, it gets a bit hairy, because sometimes Systems are interdependent on each other; the usual solution is to have some sort of message-passing functionality so that Systems can signal special cases that others might need to be aware of, which introduces a bit of overhead. There’s literature on this subject.

    There are a mind-boggling number of ways to implement this in C++. The most structured way to do it would be to make one class for each possible Component, with protected members representing the data associated with that Component, a private boolean indicating whether the Entity has this Component (whether or not this field in the database is null), and public get-, set-, has-, and remove- methods for the data. Each System needs to be aware of which Entities it should attempt to act on. You might make it so the set- and remove- methods also adds (or removes) the Entity in question from a collection representing all the Entities that the relevant System needs to know about. Or, less efficiently if you have a large number of heterogeneous Entities but more efficiently if Entities have long lifespans, you can just have a big list of Entities and have each System act only on the ones that have- all the Components needed.

    Each Entity needs to have the possibility to have each Component; in a dynamically-typed language, this would be trivial; in C++ you can obviously just make the Entity class have fields for each Component, or another popular choice is to have the Entity class multiply inherit from all the Component classes (you may derive some benefit from using CRTP in that case).

    (Or, for very simple games, you could store all game data in an embedded SQLite database and dispense with all this object-oriented malarkey; each System queries the database for everything it needs. Obviously this is untenable if you have too much going on each frame, but data flow becomes a complete non-issue.)

    One very useful property of ECS is that it gets you parallelism for free, because if any two Systems have no dependence on each other, you can execute them in separate threads. Multiplayer, too, is cheap (just have multiple Entities representing players), networking is less of a nightmare (effectively, some Systems run on the client rather than the server), and this particular architecture is what makes MMORPGs possible at all. The fact that the entire game state is stored in a single metastructure that maps directly to a relational database also makes serialization and deserialization very straightforward. There is significant cognitive, engineering, and performance overhead to this structure, plus you have to rethink how you approach unit testing, but it enables – and enforces – a separation of concerns that makes it much easier to think about what your game is actually doing, and the bigger the game, the more benefit you’ll derive from it.

    I’m just an amateur on these matters, but there are useful resources on this big ol’ internet of ours if you want to learn more. Notably, Unity implements a very mature ECS, and a bunch of tools for interacting with it.

    1. Paul Spooner says:

      Thanks for posting this. I’ve suspected for a while that the Entity Component System approach was better, but didn’t know what to call it. It seems like a pretty versatile way to develop software, though with even a higher overhead than OO. That’s really the trade-off though, isn’t it? More and more overhead and structure to simplify complexity. You could do the same thing with parametric programming if you were some kind of genius, and it would run faster too! But then no one could understand it.
      Eventually of course, we’ll get to the point where it’s nearly all overhead. Perhaps we’re there already.

      1. Mark says:

        A game is a complex piece of software. That complexity has to live somewhere. The trick is structuring the program in such a way that the complexity can be reasoned about more easily. An appropriate separation of concerns is vital. ECS isn’t suitable for every purpose, but it’s an excellent approach for large amounts of heterogeneous data with high levels of interdependency.

      2. Zukhramm says:

        The overhead shouldn’t be that much larger, I think. Most of it would be finding what entities your systems should work on, and that could be reduced for certain systems, and maybe some set operations.

        And I realize now I’m looping twice over every set of entities. I should fix that.

    2. MadTinkerer says:

      The Source Engine and Game Maker also use entities for pretty much everything. (In GM, entities are called objects, but the documentation explains that term is used for the sake of user friendliness and that it isn’t meant in a strict OOP sense.)

      Random question & answer site that happens to have aggregated a TON of relevant links: http://stackoverflow.com/questions/1901251/component-based-game-engine-design

      Wiki with practical examples: http://entity-systems.wikidot.com/

      The Game Programming Gems series of books have a lot of articles on entity systems. Game Programming Gems 6 shows how to use what the author calls Game Object Component System (same thing) to go so far as to define a basic outline of an entity in the engine and then you can potentially load all entities from data. In the example, it’s XML that’s used, but the implication is that essentially any and all of the “game parts” apart from the essentials needed to parse the data can be defined outside of the engine and edited by the developers and potentially modders as well.

      1. Zukhramm says:

        GameMaker might use it inside their engine, but there seems to be no way for a user to actually access that, so there doesn’t seem to be much point to it.

    3. Atarlost says:

      Sounds almost like the components should be classes that the entities inherit if not for the ugliness of having a bunch of loops that iterate over all the game’s objects and call the main methods. (eg. every physics object would inherit a class physics_object that contains the physics engine)

      Except it’s difficult or impossible to create classes with arbitrary inheritance trees at runtime and clunky to have the loop to iterate over everything and invoke the physics code be outside the physics code.

      1. Kyte says:

        Don’t do inheritance trees, make each component a class that implements an interface for each system it interacts with. So the Player object could implement IDrawable (exposes sprites, animation frames, timing, whatever), IControllable (exposes X/Y coordinates, etc), IDamageable (exposes HP, armor, etc), ICollidable (exposes X/Y coordinates, momentum, etc) and so on.
        Then each system can grab all the objects that implement their respective interface and manipulate them as needed.

        (Although in my example the physics engine and the control system would fight each other. Ideally you’d make the control system apply a force that the physics engine would later add to the rest of the forces to translate into movement and finally into XYZ coordinates.
        So IControllable would expose PlayerForce as read/write and ICollidable would expose PlayerForce as read-only and such)

        1. Zukhramm says:

          We don’t even have any actual entity objects. Components are individual classes that can be retrieved from a map by an entity id. Unlike individual interfaces, components can be added, removed or swapped around while the game runs.

  9. giantraven says:

    Are there any plans for a post discussing what kind of music/sound design will be going into this? I imagine you could create some really interesting procedural stuff that would be great to read about.

  10. Michael says:

    Am I the only one who thinks that the shooting-every-frame shot looks like a pretty cool flamethrower. Give it low damage per missile, shortish range, and throw it in.

  11. Zukhramm says:

    With something that’s purely functional (and this includes Carmack’s Haskell Quake) are actually not allowed to change anything, including themselves. Keeping the door analogy, they only have windows, you can look at the insides, but you can’t touch them.

    1. Bryan says:

      They only have windows, but the windows can … uh … create a new, slightly different copy of the room?

      I think this metaphor is breaking down a bit. :-)

      But this is basically how anything ever gets done in a functional system. A new copy of the thing gets created, and used instead of the previous copy.

      1. WillRiker says:

        This isn’t strictly correct: you cannot do I/O without side effects because I/O is, by definition, a side effect. When you’re writing any useful program in a functional programming language, you have to be aware of where your I/O is happening and how the non-functional I/O parts of your program interact with the pure functional parts. Haskell in particular has a fairly clever way of encapsulating non-pure-functional code.

  12. krellen says:

    By about the fourth item on the list, I said to myself “man, player.cpp is getting kind of crowded. Might want to break that up a bit.” Of course, any person looking at the large picture lined up sequentially like that probably reached the same conclusion; programmers don’t do that sort of thing on purpose (very often), and with the many levels of removal between each of those steps it was probably really easy to miss at the time.

    Which, of course, is exactly why you’d want to have a program engineer to submit the piecework you do to; they don’t get bogged down with the nitty-gritty of the problem your solving and can notice this meta-problems before they escalate out of control.

  13. Sean Conner says:

    Shamus, you might want to read Purely Functional Retrogames, a series of articles about writing PacMac in a functional style. In fact, a lot of what he says is really interesting, as a former 8-bit game programmer turned functional programmer.

    1. Benjamin says:

      Very interesting blog, thanks for posting it.

  14. RCN says:

    Shamus, this game is actually looking quite good. In fact, it already looks better than many comercial shmups I’ve played. Have you considered getting it on Steam Greenlight?

    1. Bropocalypse says:

      Yeah, I could see droppin’ a fiver on this.

  15. Phantos says:

    The lava level! The water level! The… lots-of-green-plants level! The purple one because shut up it's cool!

    Don’t forget the hot-springs episode.

    1. Bryan says:

      The sewer level! We have to use that “make everything brown” shader that everyone else seems to be using, in *some* level in this game!

      Er, wait a minute…

  16. Bropocalypse says:

    I’ve run into the “rewriting code multiples times” thing on my current project… not because I haven’t tried to design it first, but because the specific part of the project I’m working on (pathfinding) is something previously alien to me. I wasn’t really sure what I was designing, exactly. I knew I had to start over when I was trying to add code to functions that I couldn’t remember exactly what they were for.

    1. Volfram says:

      I actually just rewrote a large group of if statements in such a way that they’re marginally slower but take up half as much space and no longer duplicate code.

      I decided the reduced difficulty to maintain made up for the 12 extra checks that would need doing every time the block was run(several hundred times every time I load a level).

  17. Neko says:

    Edit: Fiddling with code formatting.

    Yeah, it can be hard to pin down these nebulous software development roles. As a subject, Software Engineering bored the hell out of me at Uni. It was all flowcharts and entity relationship diagrams and that jazz. Databases were thrown in there at one point because hey, databases.

    The actual Software Engineering as a thing we need to do before putting code to screen is, well, something I just figured was part of the Good Programmer role. But I must admit, I didn’t really appreciate the benefits of e.g. const-correctness until working on a very large project with other people, much later.

    For the still-sane:-

    const is a keyword in C++ and friends that (amongst other things) lets the programmer declare that this thing here won’t be modified. While this doesn’t really matter for the simple stuff like integers, which are passed by value, a lot of the really interesting stuff happens by passing references to objects around. If I declared my function as

    Frobbable *
    frobnicate(int a, string b, Frobbable *thing_to_frob);

    then it might not be immediately obvious what happens to the Frobbable thing I pass in as the third argument; does it get modified, and the return value of the function is merely for convenience? Or does the function just make a new copy and return that? Changing the signature to

    Frobbable *
    frobnicate(int a, string b, const Frobbable *thing_to_frob);

    is a guarantee from the author of the frobnicate function that no harm will come to the original thing_to_frob, leading us to believe that any changed versions will come back via the return value.

    (Note, I wouldn’t be passing raw pointers around, this is just an example, most likely it’d be better to use smart pointers declared as Frobbable::ptr_type and Frobbable::const_ptr_type, or turn the whole class into a pImpl-idiom thing, possibly with copy-on-write semantics… and then you see just how far the rabbit-hole goes)

    The other thing adding const does for us is ensure that when we’re writing the frobnicate function, we really don’t accidentally modify the object when we know we shouldn’t be. If the Frobnicate class declares some methods, some of which change the internal state of a Frobnicate object and some of which don’t, we can (and should) add const to the method to indicate that fact:-

    class Frobnicate
    {
    public:
    string get_name() const; // doesn't change our internal state.
    void set_name(string new_name); // might.

    private:
    string d_name;
    }

    Now if we accidentally were to use the set_name method inside our frobnicate function, the compiler will throw a fit at us, yelling at us that we promised we weren’t going to change thing_to_frob, we promised, why would you do that. When we have a const Frobnicate object, the only methods we can call on it are those that have themselves been declared const.

    This is where the real madness sets in. If you haven’t been designing with const in mind from the beginning, it is an absolute nightmare to go back to all those methods and check if they should be const or not, just to gain this benefit of maybe having the compiler catch a few bugs for you. It’s not possible to merely make a few objects const, because then the methods you use on those objects need to be const too, and anything they interface with will also need to be const, and so on. In a small project, I must admit I don’t bother with it. Let everything be mutable, we’re probably going to rip this code up tomorrow anyway. But in anything large, it can be very useful to prevent you from introducing side effects to your code. Suddenly you realise that yes, you really shouldn’t be modifying this object at this time, this needs to be a wholly separate code path, this is why you’ve been getting that strange glitch.

  18. Ozy says:

    If I’m reading that error message image correctly, it seems that you have implemented a “crash” command. I’ve only ever programmed in Fortran but I think I may have found your problem there.

    1. Volfram says:

      True, it is much easier to demonstrate what a crash looks like by intentionally driving into a tree than by waiting around for some idiot to flip his car on the highway.

      Safer, too.

  19. I read everything that you write, but don’t really comment much – I just thought I’d post to say how much I am loving this series. Your programming blogs are my favourite thing about this site, and this series is particularly magnificent. Thanks so much for sharing your thoughts!

  20. Rick says:

    These screen shots are showing a big jump in polish, they’re looking great!

  21. Steven Stewart-Gallus says:

    The post showed bugs showing up during refactoring and I wonder if you think static analysis tools, and linters are practical for solving these kinds of problems, or not useful? What’s your opinion on static analysis tools, and code linters like cppcheck, and clang’s scan-build?

  22. Zak McKracken says:

    The “pure darkness” screenshot looks actually really nice, but being a fan of “dark” levels, might I just add that it would probably get boring less quickly if your own and enemies’ shots could light up the scenery in the same way your flashlight does. This means more light on average but more contrast in lighting over time. And quick changes in mood during battle or between encounters. … or maybe that would be too much? Not sure.

    Anyway, thanks for the software engineering stuff. I’m just about to start a larger coding project for work, and engineers are horrible coders, and now I have a link or two to educate myself before going and producing tons of elaborate spaghetti code.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply to Paul Spooner Cancel reply

Your email address will not be published.