Good Robot #43: Un-UnSolved Mysteries

By Shamus
on Mar 8, 2016
Filed under:
Good Robot

So now we’re to the part of the project where we have to stop adding Fun New Things and fix the dumb crap and insane bugs we accidentally created earlier. This is my least favorite part of the project. Or any project. The equivalent for an author is once they’re done writing a book and they have to go back and proofread. Bo-ring!

Here are some of the baffling conundrums we’ve unraveled over the last few weeks.

Bug #1: Mysterious Level Exit

This is what the level exit doors look like.

This is what the level exit doors look like.

Description:

You’re flying around the level, minding your own business murdering robots, when suddenly the Good Robot acts like it just went through a level-exit doorway. The robot flies to the edge of the room, the screen fades out, and you’re suddenly on the next level, despite the fact that you weren’t anywhere near a doorway. You couldn’t even SEE a doorway.

Why this problem SUCKED:

This bug was a phantom. It was the Loch Ness Monster. It was Bigfoot. Nobody could replicate it, nobody could predict it. It would happen once in a hundred game sessions, which meant it never happened when you were looking for it. It would pop up when you were in the middle of something else, and it was so abrupt that by the time you realized it was happening you’d already been pulled through to the next level and couldn’t remember anything specific about the game state pre-transition. What room was I in? What was going on? Was there a door in the same room? Were any robots near it? Did I stupidly blunder through some gap in the level geometry without noticing?

What happened:

This bug highlights the dangers of mixing “design through coding” with “part-time coding”. It’s fine if you grow your project organically instead of working from a spec, particularly when trying to nail down elusive ideas like “fun” and “comprehensible”. If playtesting reveals that guns don’t feel good to shoot, you don’t send a terse message to the design team telling them to write you a new specification, because:

1) Writing specs is time consuming.
2) YOU are the design team, so you’d be sending the email to yourself.

You just sit down and mess around with the mechanics until things feel right, and you send the resulting changes to the rest of the team to get their feedback. The good thing is that this makes your design is very adaptable and fluid. The bad thing is that the code is going to be kind of rough and ad-hoc.

You’ll end up implementing a feature a little at a time, and the final structure will not be as organized or as logical compared to a system where you knew exactly what you wanted beforehand. This problem will be compounded if you’re dividing your time between coding and other thingsLike for example: Writing a blog, a weekly column, hosting a podcast, participating in a Let’s Play, and writing a novel-sized retrospective on Mass Effect. and the code structure keeps getting flushed out of your memory by other short-term tasks.

The game was originally one long, continuous level with no breaks or loading screens. That was super-monotonous, so we cut the gameworld into clearly-defined gameplay sections. When you get to the end of one section (called a zone) you’ll find between 1 and 3 exit doorways to choose from.

Originally, you just bumped into a doorway to open it. It would stand open for a second, but if you flew away then it would slide closed again. If you crossed the threshold of the door (which is a simple line on a 2D plane) and the door was open, then clearly you had just passed through the door. The game grabs control of your character and has them fly through the tunnel, leaving the camera behind while the screen fades out.

The threshold of the door is invisible, but you can think of it as an infinite line like so…

An artists dramatic depiction of the line segment you must cross to exit the level.

An artists dramatic depiction of the line segment you must cross to exit the level.

Note that when I say “crossed the threshold of the door”, I mean you passed from one side of the door to the other. Doors can face any of the four directions, depending on how the level is structured and where the final room ends up. The important thing is that doors always appear on the outer wall of the level.

Fine. That worked well enough. After a week or so we got some feedback: Testers pointed out that it felt kind of unsatisfying to have bullets ignore doors, and it felt dumb to slam face-first into a door to open it, bounce off, recover, and then pass through. It only took a split second, but the action was inherently annoying. It really did seem like you ought to be able to shoot a door to open it and then zip through unhindered. So that was added to my to-do list.

Then some weeks later I worked my way down to that point in my to-do list and made doors pop open when hit with player bullets. By that time I’d long since forgotten how the original door logic worked. I still had the old logic that crossing the threshold of an open doorway would pull you through, but now you could open them with bullets.

The logic was still, “If this door is open and the player crosses the threshold, then they must have gone through the door.”

An artist`s dramatic depiction of the line segment you must cross to exit the level.

An artist`s dramatic depiction of the line segment you must cross to exit the level.

This is incorrect logic, but this alone is not enough to trigger the bug. What needs to happen is this:

  1. If the player is using one of the small number of bouncy laser weapons in the game, and…
  2. those lasers happen to sail off-screen and into another room without running past their maximum number of bounces, and…
  3. they happen to end up in the last room where they hit a door, and…
  4. you happen to cross the threshold of the now-open door before it closes again.

I wouldn’t have designed the system with this goofy logic at the outset, but because the two halves were written weeks apart – with many other concerns and distractions in between – I ended up with this bug.

Bug #2: Help I Spawned Outside the Level!

Description:

The game should end if the player dies. However, if they have purchased a warranty at a vending machine, they will respawn in the first room of the current zone after death. (This consumes the warranty.) Very rarely, the player would instead respawn outside the level.

Why this problem sucked:

Actually, this one wasn’t too bad. It was rare, since it could only manifest when the player died while under warranty. But it was easy to find because my first guess was the correct one.

What happened:

Every zone begins with a circular room. In the middle of the room is a platform. This platform is intended to hold things like vending machines and the respawn station.

The green machine is the respawn station. The orange one is a hat vending machine. No I am not kidding.

The green machine is the respawn station. The orange one is a hat vending machine. No I am not kidding.

Those machines need to go on a flat surface. They look terrible if they’re floating out in the air. So when the zone is generated, it looks for a nice level spot in the first room, and when it finds one it puts the respawn machine there.

However, under certain circumstances that platform might not be level. The code intended to connect this circular room to the next room was supposed to do so by digging a tunnel. If it needed to dig straight up – that is, if the first room was directly below the second – then that digging process would take a bite out of the platform, making it uneven. There was still a patch of flat space there, but the machine-placement logic felt it wasn’t quite big enough. So the respawn machine wouldn’t get placed.

When you respawn, the game would look for the glowing green respawn machine. If that machine was missing, you’d end up placed at the world origin, which is always above and to the left of the gameplay area.

I dislike having the entry room be identical for every zone, and if we had more time I’d come up with a variety of layouts. But it’s too late in the project, and as this bug shows, messing with the room logic can create hard-to-find edge-cases. This close to release, we don’t want to risk creating any hard-to-find bugs.

Bug #3: Ridiculous Score Bug

Description:

For some reason, some players end the game with a score that’s about two orders of magnitude too high. Their score will be around two million instead of in the ten thousands range.

Why this problem SUCKED:

Once again, this bug played hide-and-seek. After you play the game a few hundred times and once you change the scoring system a half dozen times, the score sort of loses all meaning. “Twenty thousand on level four? I can’t remember if that’s high or not.” So I’ve stopped looking at the score, even though it’s always right there in the corner of the screenWe’ve had a lot of debates over whether we need a score system, or how much people will care, or how much of focus it should be, so don’t feel the need to ask those questions in the comments. We’ve gnawed on that argument for hours.. So then you’ll randomly look at your score and see it’s clearly out of reasonable bounds, but with no idea of when it went wrong or what might have triggered it. We’ve already sent preview copies to some game journos, and we can see on the leaderboards that a few of them (but not all of them!) have run into these bogus scores.

What happened:

Here is a case where you can’t find a bug because it’s disguising itself as a different kind of bug. When we saw the problem, Arvind and I both thought the same thing: “Uninitialized variable!”

Consider:

1
2
3
4
5
6
7
8
9
10
int FruitAdd ()
{
  int apple;
  int orange;
  int gary;
 
  apple = 10;
  gary = apple + orange;
  return gary;
}

On line 7 we assign the value of 10 to the variable “apple”. Then on the next line we assign the value of apple+orange to the variable “gary”. Except, we never put a value into “orange”! In programming terms, this means that orange has a “garbage” value. It could be anything. Orange refers to some specific piece of memory. At some point in the past, some other variable was maybe using that bit of memory to store some totally unrelated information. Then that variable’s life ended and the memory was made free again. Then it was given to orange.

To be clear, the compiler will warn you about this if you do something as blatant as my code snippet above. But there are other, subtler ways for this to happen that the compiler can’t catch. So we both assumed score was being assigned some garbage number of points. Sometimes this would be harmless, but other times a roll of the dice would result in this garbage value being very high, and thus this bug.

But, no. That’s NOT what this bug is.

It was our artist Ross that discovered the bug. Which is good, since it was Ross that caused the bug. Well, he didn’t so much “cause” it as “blunder into it”, since this problem is actually caused by a design flaw that intersected with some undocumented features.

Cause 1:

The game didn’t always have a score. In the early days of the project, your only goal was survival. When we added the score, I needed a way to quickly ascertain how well it was working and how it “felt”, and I didn’t want to have to edit dozens and dozens of robots to do it. You can blow a lot of hours messing with point values, tweaking them until they feel right. And that’s not my job.

So as a way of bootstrapping the score system, I had it use hitpoints instead. If the artist hasn’t yet assigned a score value, then it would use the robot’s hitpoints for score. This seemed like a good starting point for score values anyway, since tougher robots would be worth more. This would quickly get a rough version of the score system working so everyone could try it out and discuss our thoughts at the next meeting.

The idea was that I could leave the robot definition file alone, and Ross would go through and assign score values later. (Which he did.)

Cause 2:

We’ve got several “mother” type robots in the game that can spawn little minions. (And if we want, those minions can in turn spawn their own child minions, etc.) Ross needed to create a boss robot with invulnerable minionsReally? I didn’t even know this was a thing until now.. The minions won’t die until their mother dies.

We also have machines in the game. Like robots, machines have hitpoints. If hitpoints are zero, then bullets pass right through the machine. If hitpoints are negative, then bullets will hit the machine but it will be immune to damage. If hitpoints are positive, then the machine will blow up once it’s received enough damage. There are various situations in the game where all three of these make sense, and this system works pretty well.

But robots don’t use this system. You can’t make a robot immune to damage by giving it negative hitpoints. So when Ross found the reasonable thing didn’t work, he had to improvise. So he set the minion hitpoints to a ridiculously huge number.

Cause 3:

It’s standard practice in our game that when designing minions that can be endlessly spawned by a boss, they should be worth zero XP and zero score, so that a player can’t just sit there and farm them foreverImagine if the best way to get the high score in Pac-Man was to simply stay on level 1 and eat the same ten dots over and over until you passed out from exhaustion. That pretty much ruins high-level play by turning it into an endurance test.. That would be very boring and you shouldn’t reward boring strategies because then players feel obligated to play the game in a way the ruins the fun. So these minions – even though they were invulnerable – were worth zero score and XP.

All together now:

The minions have some huge number of hitpoints and no score. Well, the program can’t tell the difference between “Ross never assigned me a score” and “this robot is worth zero points”. So when it sees a score value of zero it switches to using hitpoints for score. Ross probably (quite reasonably) assumed that since you can’t kill these minions, you won’t ever get points for them. But they die with their mother, and their points are awarded when that happens. This mother was a mini-boss in an optional level that you may or may not encounter, depending on what doors you choose at the end of each level.

So only some players would encounter this boss. And only some of those players would choose to engage this mini-boss. But if they did, then they would be given some ridiculous number of points for each of the minions, and their score would skyrocket.

So it wasn’t an uninitialized variable bug. It was several technically correct rules producing unexpected behavior.

Enjoyed this post? Please share!

Footnotes:

[1] Like for example: Writing a blog, a weekly column, hosting a podcast, participating in a Let’s Play, and writing a novel-sized retrospective on Mass Effect.

[2] We’ve had a lot of debates over whether we need a score system, or how much people will care, or how much of focus it should be, so don’t feel the need to ask those questions in the comments. We’ve gnawed on that argument for hours.

[3] Really? I didn’t even know this was a thing until now.

[4] Imagine if the best way to get the high score in Pac-Man was to simply stay on level 1 and eat the same ten dots over and over until you passed out from exhaustion. That pretty much ruins high-level play by turning it into an endurance test.


is a programmer, an author, and nearly a composer. He works on this site full time. If you’d like to support him, you can do so via Patreon or PayPal.


A Hundred!8108 comments. Quick! Add another to see if this message changes!

From the Archives:

  1. Chris says:

    Out of curiosity on bug #1, why use an infinite line rather than something like a collision rectangle?

    Edit: Oooh, base case!

    • Echo Tango says:

      Even a non-infinite line should do the trick. No need to make it 2D yet. :)

    • Bropocalypse says:

      I think at the time of coding, there was no need for it to NOT be infinite.
      Strictly speaking, I suspect that it was merely “if the player is beyond this point on this axis, the level’s over.” There was no accounting for the fact that the exit would later be moved away from the edge of the map.
      Counter-intuitively, sometimes something that appears ‘infinite’ is simply ‘less strictly defined.’

  2. Content Consumer says:

    It was several technically correct rules producing unexpected behavior.

    Now this is my favorite kind of bug. At least, when it results in something good instead of something bad… systems colliding to provide unexpected behavior is one of the great things about organically-grown games. Heck, even when it does result in something negative, it’s often fun just to find.

    • Matt Downie says:

      I read a good article about Spelunky bugs recently. Like, if you got punished by Kali you’d get a ball and chain attached to your leg, and the ball was supposed to destroy walls when it got stuck, and it could sometimes do this to walls that weren’t supposed to be destructible, and this allowed you to get somewhere you couldn’t normally. The creators don’t fix bugs like that because they don’t break the game, they just give skilled players another trick they can learn.

  3. Ninety-Three says:

    Imagine if the best way to get the high score in Pac-Man was to simply stay on level 1 and eat the same ten dots over and over until you passed out from exhaustion. That pretty much ruins high-level play by turning it into an endurance test.

    It sure is a good thing high-level Pac-Man play isn’t an endurance test.

  4. Abnaxis says:

    Commented three times from my PC to no avail, so this might be the fourth time I make the same comment (from phone this time)

    I read your preview for Good Robot linked in Twitter.

    Someone should tell the reviewer that Rutskarn’s real name isn’t actually Rutskarn.

  5. King Marth says:

    Bugs like #1 and #3 are incredibly satisfying to explain, precisely because they make perfect sense when you look at the actual rules which are themselves reasonable in isolation. The explanation for bug #2 took a second read to really get what was going on, and since it basically is an uninitialized variable problem (respawn location’s default value was a nonsensical, albeit consistent one) it isn’t quite as fun to talk about, though that consistency is also why it’s much less irritating to actually find.

    I am curious about how much time was spent on building debug tools, like ways to force specific level layouts or telemetry (like pushing a key when one of these rare bugs happens that saves a text dump of the current and previous level layouts). The usual automation tradeoff applies here: Will the time spent building this tool make up for the saved time on finding these things? Will the tool work in enough scenarios to actually be reusable? I know Warframe had some success with a special screenshot key that stores metadata about level layout and player location, for reporting map holes and other bugs.

    • Volfram says:

      As soon as he mentioned the spawn platform I assumed it was a variable initialization problem, though in part because I’ve run into those in the past myself. (“Oh that’s funny. Because this school doesn’t have location data filled out, it appears in the Atlantic, just south of Morocco. At 0,0 lat/lon.”)

      My guess, however, was that the spawn platform was ending up at 0,0 and the robot was spawning there, instead of the robot itself just spawning at 0,0.

  6. Ingvar says:

    Now I am interested to see if my absolute favourite class of “unintended behaviour” exists in the Good Robot config language.

    Let me take a step back. At a previous job, we were (essentially) running everything “in the cloud” (not as you’d expect us to, by running Amazon EC2 instances, but by having our own internal cloud; and not by configuring machines, but by defining “jobs”, and saying “I want N of these”). Writing job configurations is boring, so some rather enterprising individuals wrote a config language to do this. So far, so good.

    However, it turns out that when you’re defining a whole bunch of jobs, that need to have some thing in common and some not (let’s call this “a service”), having things like templates is really handy. But that means that you need to have a way of overriding things in templates. And for reasons, it’s vastly easier implementing this as “lazy evaluation” (basically, this means that until you use a value, it’s not computed). And sometimes you need to enable or disable chunks of these templates by conditionals.

    Now, it turns out that “instantiate a template with overrides” is isomorphic (um, “essentially the same as”) and if you have function calls and conditionals, you can implement surprisingly complicated stuff. Like something that computes a Mandelbrot fractal, encodes it as a PNG, base64-encodes that and emits an IMG tag with inline image data (yes, that is actually allowed by the HTML spec). Or, as I once proved by doing so, a small lisp interpreter.

    Yes, my favourite class of “unintended behaviour” is “accidental Turing Completeness”.

    • From Shamus’ brief description of it, it doesn’t sound like the GR config language has loops or branching conditionals, which is required for Turing-completeness.

      >(yes, that is actually allowed by the HTML spec)

      Really, with only HTML, not Javascript?

      • Bryan says:

        Yeah. data:/ URIs. That’s why the base64 step. :)

      • silver Harloe says:

        This exists, and is actually kind of handy for teeeny tiny images – since those would otherwise cause your browser to download another image (but it will only download like 4-5 items at a time, so the fewer extra items you make it load, the faster your page response) – and otherwise take up too much disk for their contribution (a modern HugeAss disk has like a 4096 block size – meaning if you have a 100 byte file, it takes 4k of disk space – 3996 bytes are wasted)

      • Ninety-Three says:

        Are all enemies in GR automatically allied with each other? If you can set up enemy AI such that Type A Robots shoot at Type B robots, but not Type C robots, I’m pretty sure you could set up a series of robots spawning other robots that fight each-other to act as conditional events. “Type N robots alive” would be the allocated memory, which is checked by spawning a type M robot which will die if there are type Ns alive to attack it.

        If robots are all on the same team, I could imagine a way to achieve the same “Type As will kill type Bs but not type Cs” if there is friendly fire and healing. Just do an N damage explosion with infinite range, followed by an N healing explosion with infinite range, destroying everything with health <= N.

    • Will says:

      For anyone who hasn’t studied theoretical computer science in a fair bit of detail, Turing completeness¹ is actually really easy to stumble on accidentally. The quintessential example of an incredibly simple construct which is actually Turing complete is the two-counter machine: a device with some finitely-complex control, and two counters which can each increment, decrement or be reset to zero. It turns out, with a sufficiently clever encoding scheme and control program, such a device can do anything a Turing machine or computer program can.

      And if something that simple is Turing complete, it’s easy to imagine that lots of other simple things are also Turing complete. It’s actually fairly common.

      ¹ Basically, being computationally powerful enough to compute anything a Turing machine can. This more-or-less includes real computers², along with a couple of other standard theoretical constructs like the Lambda calculus, and a bunch of things that are not intentionally Turing complete, like the C++ templating language or Magic: the Gathering. Turing completeness is not always a desirable property, because the Halting problem makes it impossible to automatically prove a bunch of useful things about the system in question (like “it will run within a certain amount of time”).

      ² Modulo some assumptions about memory: technically, Turing completeness requires an infinite amount of memory, but if we construct an imaginary equivalent to a real computer that has infinite memory, it’s Turing complete.

      • Tektotherriggen says:

        Pedantic question regarding “normal PCs” being upgraded to true Turing Completeness. Would we also need to have an infinite address space (in the sense of 32-bit PCs only being able to handle up to 4GB of RAM, would be need an infinite-bit version of x86)?

        Or, if we “just” allow infinite amounts of RAM and/or hard disk space, is it possible for a standard PC architecture to make use of all that (perhaps using a compiler trick to make all the memory addresses relative, rather than absolute)?

        • Decius says:

          Whenever running on actual hardware, Turing Machines are assumed to never outstrip the physical capability of the hardware. Trivially, no physical machine can duplicate the behavior of a Turing Machine that writes a number of 1s to the tape greater than the number of quarks in the observable universe and then halts.

          • Tektotherriggen says:

            Oh, sure; but I was thinking of the thought experiment where you could conceive of adding infinite RAM to a normal x86 PC – could the PC even theoretically use that much storage. Ignore the fact that that much silicon would immediately collapse into a black hole; that there would be an infinite light-travel delay time from one end of the infinite chip to the other; and other mere laws of physics like that. Is the instruction set up to the task?

            • The Advisor says:

              It wouldn’t be. The instruction set’s design is limited by what would be practical to implement in hardware.

              In order to try and keep things somewhat sane most if not all hardware uses memory addresses with a constant size. Even if certain instructions allow you to specify a relative memory address (like jumping backwards 18 bytes in code to make a loop) that just causes it to do basic addition/subtraction on the current relevant memory addresses. If you try to go above the highest possible address (~4GB on x86, ~16EB on x64) it would either cause an error, wrap around to 0, or possibly (although not likely) stay at the max value.

              An instruction set, and by extension the hardware that runs it, would need to have this infinite addressable memory stuff kept in mind during design and would need to have an insane if not outright impossible addressing system because infinite memory means that addresses need to be infinitely large as well. Unless the hardware was also infinite it would need to handle these addresses in constant size pieces which would cause processing to take an infinite amount of CPU cycles and time anyways even with physics defying instant response from memory. Suffice to say infinity does not mix well with stuff outside of mathematics as usual.

              Sorry if that rambled around a bit, or wasn’t very clear. I’m pretty tired at the moment.

              • Mephane says:

                Theoretically, one could encode the address in an infinitely extensible form. An address could for example be an array of bytes terminating with a zero-byte, so the further you go from the start of the tape, the longer the addresses get.

                • The Advisor says:

                  I actually thought that too initially, but infinity kind of makes optimizing the smaller addresses pointless since an infinite number of addresses would be so large that they may as well be considered infinite. You would effectively be optimizing for an immeasurably tiny part of the total address range.

                  I’ve noticed that the properties of something that interacts with infinity has the nasty habit of themselves becoming infinite as well (at least some of the properties anyways), which makes these thought experiments a bit unfulfilling since the answer tends to be infinity regardless of the actual question.

              • Tektotherriggen says:

                Don’t be sorry – that’s exactly the sort of answer I wanted, thanks :-) .

    • matthewhoffman says:

      I… I think I read the internal slide deck about the “Mandelbrot PNG through config language” project at work a few hours ago. That’s quite a coincidence.

      • Bryan says:

        I’m pretty sure I know exactly which language this is. If I’m right, I use it every day.

        …I’m also not a huge fan of it. :-) It’s pretty good when your jobs and services are small. But if you try to do something complicated, or you try to apply normal programming practice (like only defining things in one place) too much, you end up writing a mess that nobody can understand.

        Then there was the other configuration language, for our monitoring tools, in which one … enterprising individual … managed to write an implementation of Conway’s game of life.

        • Ingvar says:

          Can I just say “go perfectlittlehorror”? :)

          Also, yes, you really want to avoid the “multiple inheritance” situation (if that’s your wish, use the config language named like a flute), because if you use it, strange things will happen. The extraction dudes have a pretty good set of “best practices” in a document, somewhere, you should totes have a look at that.

          • Ingvar says:

            Oh, there’s also a collection of various references at “go the-tentacles” (including, but not limited to, the rather excellent slide deck by Misha, a former TL of the language, showing ALL SORTS of interesting weird shit, including the undocumented and somewhat useless side-effects in what purports to be purely a purely functional language).

    • Chauzuvoy says:

      Thus proving Greenspun’s tenth rule once again.

      • Ingvar says:

        I’d say that it’s not so much a proof of, as a variation of. The fact that it is an incomplete implementation of “more scheme than Common Lisp” isn’t accidental.

  7. Tim Keating says:

    Re bug #1, I see why you would have done it that way, given that it is so expensive to detect a collision between two rectangles…

    • Shamus says:

      Context for non-coders: Keating is being sarcastic.

      The thing is, it had ALREADY done rectangle collision when you opened the door by bumping into it. So from the perspective of the first round of changes, I would had been evaluating something I already knew to be true. It wasn’t a speed optimization, it was avoiding “pointless” clutter.

      • Kyte says:

        Dunno, the original implementation felt iffy to me in first place. Without a timeout, then you have the threshold permanently sitting there once the door’s opened. If some quirk of the level generator had let you cross it on some other part of the map you’d’ve ended up with a similar situation.

        Also it feels kinda wrong to not check twice when there’s two distinct collisions (the first to open, the second to cross). If both checks were made in the same frame then yeah it’d be doing the same thing twice but you have to preserve state across frames and that makes it two distinct operations in my mind.

  8. Warbright says:

    As a non-coder…I’m uncertain how/if you solved these bugs. It seems like maybe #2 was more of a problem to fix than leave alone. I’m just curious if the fixes are all obvious to coders, as I wouldn’t really understand the technical aspect of correcting them.

    • Falterfire says:

      Frequently (but not always) with bugfixing the challenge isn’t fixing the bug, but figuring out the root cause. Most of these bugs sound like things that are probably reasonably easy to actually fix once you’ve figured out what storm of circumstances is causing them to happen in the first place.

      Of course, that’s just a ‘it may be easy to fix’ – One of the things you learn with coding is that if you’re not actually staring at the code, making a statement more predictive than “there is a way to fix this given enough time” is unwise since there are multiple ways to accomplish any given task and you don’t know which one was used.

      • Trix2000 says:

        Yeah, reliably reproducing bugs can often take up the lion’s share of debugging. Or at the very least, that’s usually the hardest part.

        The worst bugs are the ones that disappear when you actually look into them, never to return again. Because even if it appears the problem has been solved, your mind is continually thinking “When is it gonna come back? It HAS to come back.”

        But once you can reproduce the exact circumstances in which a bug happens, the causes and results of said circumstance are generally pretty obvious.

        • Mephane says:

          And then there are bugs which happen sporadically, no one ever gets to reproduce predictably, the cause never gets found, but at least you find a way to automatically detect it afterwards and repair the mess.

          Or the variant “when X happens Y breaks”, where you have no idea why X is happening but you can prevent Y from breaking at least.

          • Phill says:

            One of my best achievements was tracking down a really rare crash bug that only affected PS3 (IIRC) and was never seen on XBox 360 or PC. It happened maybe 1 time in 10000 or less in the menu system, and the core dumps were never terribly informative. One of those “it doesn’t matter that much, we can’t meaningfully reproduce it and have no idea what is causing it” cases, which got fixed in the traditional way: I stumbled across the answer whilst looking at something else entirely.

      • Warbright says:

        Thanks guys for the responses. Makes it sound similar to my own living (medicine).

        • Ingvar says:

          The main difference is that it’s much easier to rip code apart, instrument it to insane levels, put it back together and then start digging through the data. Humans, I am given to understand, do not really tend to work very well once you’ve done that.

  9. Droid says:

    Just read all of this recently and I’m now super-hyped for this game! That review you linked on Twitter, somehow reminded me of a game series I enjoyed a lot in my childhood: “Ratchet and Clank”. Sure, it is 3D instead of 2D and on different planets and whatnot, but I think this theme of one vs. many, dodging or shooting down incoming fire while also dishing out damage to your enemies in the process, and especially a big company that sells you lots of fine tools for your task via vending machines are big similarities…

    Okay, I really have to get rid of all that nostalgia or I will never do this game justice.

    • Gilfareth says:

      Y’know, I hadn’t caught that comparison but now that you make it I find myself salivating over the game’s upcoming release even more. I love the Ratchet and Clank series and, being a short-sighted human, just the idea of Good Robot playing remotely like one of my favorite series has me more interested in it than I already was.

      Which was already quite a lot, I assure you.

      (please just release this game already ;~;)

      • Droid says:

        I recently replayed the Ratchet and Clank games I played in my childhood (1, 2 and 3) and boy are they hard sometimes… I mean, not soul-crushingly hard, but rather a “Get good faster!” kind of hard, like Dark Souls. I still miss Ratchet Gladiator, which for some reason I enjoyed most of all the ones I played, but, unfortunately, there is no PS3 version to be found.

  10. Daemian Lucifer says:

    So now we’re to the part of the project where we have to stop adding Fun New Things and fix the dumb crap and insane bugs we accidentally created earlier.

    Dont be silly.No one cares about bug fixing.What you should do now is make as many dlcs as possible and cram them in the first month after release,so that you can maximize the profits.

  11. Decius says:

    When you’re bug hunting, don’t you typically have screen-recording software up, so that you can go back and figure “When did my score explode” or “What happened before I went into the next room”?

  12. Volfram says:

    The uninitialized variables theory in #3 confused me for a minute because in D (which I will keep plugging until Shamus tries it out, because it addresses almost all of his complaints about C and C++), integer variables are initialized to 0 on declaration, and floating-point objects are initialized to NaN. (I quickly learned that if meshes aren’t displaying, something probably wasn’t initialized and all of the vertexes are being fed to OpenGL as NaN, with undefined behavior. In practice, OGL generally interprets them as zeros.) This means the scores would have been lower than expected.

    It doesn’t mean you shouldn’t still always initialize your variables, even when you want them to be 0 or NaN, but it helps reduce undefined behavior and makes tracking bugs down easier.

    • Xeorm says:

      Which is where you run into the classical problem that any change will affect someone’s workflow negatively. Initializing numbers with garbage can be useful, because it’s immediately obvious (usually) that such a number is garbage. Comparably, zero looks like a pretty clean number, and may fool people into thinking that it’s the right number, when really it’s also a garbage value.

      • Kian says:

        Gotta agree with Xeorm. Why are floats initialized to NaN and ints to 0? That’s horribly inconsistent! As Peter says below (paraphrasing) zero is not null.

        This is why I dislike the desire to prevent errors by having the language “fix” them instead of highlighting them to the programmer. The problem with an uninitialized variable isn’t that it contains random garbage. The problem is it does not contain a meaningful value. When you “initialize” it to zero, you fix the symptom, since it no longer has *random* garbage. You have done nothing to fix the underlying problem, however, since zero is still not a meaningful value. On the contrary, you have now hidden what was a clear error in something that might or might not be an error.

        • Echo Tango says:

          instead of highlighting them to the programmer

          This is one of the first things I noticed when I first started writing Go code.
          You immediately get errors, that you haven’t initialized your variables on line X, Y and Z. :)

        • Matt Downie says:

          As often as not, zero is a good default value. It works fine in C#, and if you don’t want it to default to zero, you set it to something else when you define it.
          e.g.
          public class Enemy
          {
          int mScoreValue = -1;
          int mMaxMinions; //Defaults to zero
          float mMinionCreationRate; //Also defaults to zero
          int mMaxHitPoints = 100;
          int mCurrentHitPoints = mMaxHitPoints;
          string mName = “Unknown”;

          public int GetScoreValue()
          {
          if (mScoreValue < 0)
          {
          Debug.LogWarning("Score value not set in enemy "+mName+" defaulting to HP "+mMaxHitPoints);
          return mMaxHitPoints;
          }
          return mScoreValue;
          }
          //etc
          }
          And all without needing a .h file.

          Last time I worked in C++, we wrote our own memory allocation system that set memory to 0xCECECECE when allocated, and 0xCDCDCDCD when freed – this at least gave us some clue as to what we were looking at when we got a crash, and made the crashes more consistent than random garbage values would.

        • Volfram says:

          Mostly because floating point values have several flags that integers don’t, partly for compatability reasons. As Matt Downie pointed out, though, 0 is a pretty definitive value if you’re expecting a positive one. The major downside is that it won’t propagate through your code like a NaN will, alerting testers that a floating point variable somewhere in the chain wasn’t initialized.

          Also, to be clear, I am in no way suggesting Shamus should be switching Good Robot from C++ to D. That would add at least 2 years to the development time, and I’d like to play it some time today. I’m just saying that I think if Shamus were to, say, try using D for his next experimental project, we’d be treated to a series of posts about how nice it is to, among other things, have strings handled internally to the programming language before even importing any libraries.

          Half of the comments I’ve seen Shamus make about programming are complaints about shortcomings in C/++ which D fixes. Fully half, if not more.

        • guy says:

          Initializing to zero has the advantage of consistency; a random garbage value can be anything and may sometimes be the correct one, thus producing a heisenbug. I generally prefer to have the compiler throw an error instead and manually set things to zero if I want them to be zero, but that’s not always practical. The big one I remember from Java is arrays; it has to allocate the memory for all the elements when it creates the array even if you don’t want to fill it in on that exact line. Though I honestly forget if Java actually initializes integer array contents to zero automatically; as a matter of coding practice when I want an array filled with zeros I fill the array with zeros.

          • Volfram says:

            Best practices vs. language conveniences. I typically provide explicit initialization for all of my variables, even the integers that I want initialized to zero. When in doubt, don’t leave decision making up to the compiler.

            But yes, I think the basic theory is “Oh, this zero shouldn’t be a zero” is easier to parse than “I think this value might not be what it’s supposed to be.”

    • Mephane says:

      I am a fan of of the “Treat Warnings As Errors” setting in Visual Studio. Build will be fail if you even leave a single integer uninitialized.

  13. Peter H. Coffin says:

    Well, the program can’t tell the difference between “Ross never assigned me a score” and “this robot is worth zero points”.

    And here is where we have an entirely illustrative difference between “zero” and “null”. And why having both is handy. And Shamus is thinking of about eight ways he could have done the structures for the robots read in from the file slightly differently that would have avoided the entire problem and telling himself it really is too late to go back and change it.

    • Naota says:

      This is actually the core of the bug – I explain it a bit more below, but my core wrong assumption was that the scripting parser treated undefined variables as 0, as is common in the projects I’ve worked on so far, but not vice versa. The former makes perfect sense to prevent scripts from easily crashing the project if the scripter makes a typo or leaves out a definition, but the latter is never (to my knowledge) done on purpose.

      What made this bug so damnably hard to unravel was finding out exactly where this huge injection of score was coming from, and in such cleanly even numbers. You kill tens of thousands of robots over the game, and somewhere between the start (where you’re looking the hardest) and the end (where you realize you’ve stopped looking at your score gains subconsciously) a single instance of robots dying spat out this oddly specific result.

      • Abnaxis says:

        Correct me if I’m being dense, but do you not manually spawn the enemies you edit and shoot them to test your new scripts? Is checking the score not part of that testing?

        • Naota says:

          Manual spawning and the like aren’t part of the game engine as it stands, so the way to test robots is to put them in a zone’s list of enemies, spawn into that zone via the console, and play it until you encounter them “naturally”. Needless to say, there are lots of enemies in a zone and ample opportunity to miss the score counter jumping on a specific one when you’re testing for other behaviors.

          It’s a good point though: if these new bots needed more than one or two tests to verify that they (otherwise) worked as intended it would have been more obvious. Shamus’s scripting setup makes it so easy to create new robots that the only real testing they need is a handful of encounters. Fine-tuning their balance is done all together with the other bots, in passes as part of my playtesting runs.

          • Shamus says:

            NO! Please tell me you haven’t been working for all these months and you didn’t know you could spawn robots from the console! Damn it. I remember I wrote up a document detailing all the console commands at some point. I hope I didn’t leave this one out. :(

            Anyway, it’s pretty simple. Just open up the console and:

            spawn robotname [number]

            If you omit number it just spawns one. Just for fun: Try spawning 100 robots sometime. It is stupid / hilarious.

            • Naota says:

              Oh man, after checking out the full command list it looks like I somehow guessed the majority of them through sheer inference without ever seeing the document. Well, at least that’s another good game development story to tell!

              On the plus side… free mandatory playtesting hours?

            • Ninety-Three says:

              Shamus, are these console commands going to make it into the launch game? I was always a fan of messing around with cheats in games (special shoutout to the extremely powerful console of Jedi Outcast) and I’m disappointed by how modern games seem to have been moving away from this kind of thing.

          • Abnaxis says:

            Yeah, upon further reflection after asking my question I remembered all the messes I’ve made trying to make scripting systems do things the original creators never intended. I can totally understand the issue. It’s interesting, because Shamus has said before that he obsesses over details where someone using his scripting will do something he never intended, but something still managed to squeak by. Makes me feel less bad for similar mistakes I have made when I try to make a square peg fit in a round hole.

            As an aside, what was the actual fix for #3? Assign the minions a score of 1? Have a programmer fix the code so it treats NULL different from 0? Change the default behavior so it doesn’t use HP any more?

            • Shamus says:

              Multi-fix:

              * Robots can now be invincible by assigning them negative HP. Furthermore, it won’t show damage counters when hitting invincible bots and it will make a different sound when you shoot them. Also, invincible robots won’t shake, flinch, blink, or let off sparks like robots normally do when damaged.
              * The default-to-hitpoints if score is zero “feature” was removed.

            • Ingvar says:

              Wild guess, the robot parser does something like:

              if command.has_prefix(“score:”) {
              score_set = true;
              robot_class.score = get_score(command)
              }

              if !score_set {
              robot_class.score = robot_class.hp
              }

  14. Neko says:

    It was several technically correct rules producing unexpected behavior.

    The best kind of correct.

  15. kdansky says:

    What’s the point of a score?

  16. Jarenth says:

    Dunno what you’re talking about, Shamus: I send terse, passive-aggressive emails to my career planning and personal development team all the time.

    Check out this interesting job listing, you procrastinating asshole. And don’t forget to sign up for that course, or you’re directly responsible for ruining my life.

    I never get any replies, though.

  17. Naota says:

    If you want some more background on Bug #3, the story actually goes a bit further:

    The boss of the 5th level of Good Robot generates invulnerable “child” turrets that orbit its body as a means of attack, which die reactively when their mother does. This actually presents a few problems:

    The first issue is the invulnerability. As a scripter for projects this size I find it’s sometimes necessary to do things in ways which put the least burden on the programmers – at least while they’re still developing core features. A quick and dirty hack to accomplish something simple nobody will ever see is sometimes better than another three entries on the busiest developer’s task list – especially when he’s busy in addition sleepless, sick, and writing about a game that features Kai Leng. In short, the quickest, dirtiest hack I could conceive of was robots with health so high it would be unreasonable to ever kill them. And hell – even if they did die, the robot was worth 0 points, right?

    (Interesting side-note: there are many other robots that spawn children worth 0 points, but they all have HP values in the 5-15 range – exactly the same amount as normal ‘bots give in score points on death. It’s nearly impossible to tell the bug is happening from killing them.)

    The second issue is one of game design. There is a single type of robot that can’t be killed in the entire game, and it’s an accessory to a boss. It has no visible HP bar, and is attached to something that takes a very long time to die. This is a recipe for wasted effort, confusion, and just generally poor game feel. How do we tell the player that it’s immune to damage?

    Initially, we made these enemies white to match the game’s visual language (for instance, red=homing and yellow=projectiles you can shoot down), but the players still had no frame of reference for what white objects meant on their first encounter. For quite a while the turrets shifted through various forms of confusing. We needed this mechanic to be commonly seen and explored before the boss, so that by the time players entered the room they would understand the implicit meaning of white, orbiting turrets.

    The rest follows naturally: I took the boss turrets, changed them into small firing platforms, and attached them to a new type of robot I placed at the start of the level. These tiny minions immediately get the message across after a few seconds of fire don’t kill them, and visibly connect to the thing you need to shoot instead.

    And so a devilish bug is spawned: invulnerable robots cause no issues because realistically they never die; minions are (supposed to be) worth nothing; but create invulnerable minions…

    • Volfram says:

      It looks like it’s actually a good thing you decided to break the rules like that, thereby creating an obvious bug encompassing a couple of much more subtle ones which may never have been detected otherwise.

  18. Zak McKracken says:

    Soo, wouldn’t Bug #1 also have produced scenarios where you shoot Door 1, then shoot Door 2 (so both are open), then reconsider which one you want, and then the game genuinely has no idea which one the player just picked (|i.e. will probably take the last one being shot at)?

    You don’t actually say how it’s been solved, but I assume you do have rectangle collision in there now?

    • Shamus says:

      First question: Yes, that would have been possible. But if someone did that, it would be a coin-flip which door “won” and triggered the level-change. (The winning door would have been whichever one was created first, which there’s no way to know that without looking at the state of memory.) To see the bug under those conditions:

      1) You’d have to open two doors on the same wall at the same time. For some reason.
      2) Go though one of them before either one closes. (Tricky, since doors snap shut very quickly. In fact, pulling this off might require some dedication to get the timing just right.)
      3) You’d need to lose (win?) the coin-toss between the two doors, so that the door you chose wasn’t the door that was triggered.
      4) The door you chose wasn’t a mystery door, which by definition you don’t know where it leads.
      5) This would need to take place late enough in development that there WAS a door icon, and after we’d designed the icons and settled on their meanings.
      6) You’d have to remember what you chose, know what that specific door icon meant, and then notice that where you ended up doesn’t match.

      But you are correct, the bug could have been discovered this way.

      To answer the second question:

      Yes, now it uses a bounding rectangle that covers the passage behind the door.

      • Yerushalmi says:

        So how *did* you figure out what was causing bug #1? It seems like one of those things that require a moment of pure inspiration to hit upon and I would *love* to know the context.

        • Abnaxis says:

          If I was the one writing the code, I probably would have started with the level-transition code and worked my way back from there, since the issue is “level transitions happen at inappropriate times.”

          I kind of wonder if it would have been possible to leave to “open door” code as it was, but move the rectangle so that it floats in front of the door instead of in the same space as the door, to get an “automatic sliding door” kind of effect.

      • Zak McKracken says:

        ohhkay!

        alright, didn’t know that the doors snap shut again. That alone makes my scenario much less likely to happen. Thanks for the explanation.

  19. To my surprise, I had a dream about Good Robot last night.

    I will say you need to address the balance issues in the robot that splits in two, whose children split in two, until all available space is filled. When the process starts just offscreen, you can often end up already too far behind to deal with it with your default starting weapons by the time you notice it. It shifts the entire focus of the game from “shoot all the bad robots” to “watch out for this particular robot and make sure you kill it on sight at all costs, and incidentally, do all the other things too when you have spare time”.

    Also it strains suspension-of-disbelief that the robot that tries to absorb Good Robot by surrounding it would be able to continues successfully doing so while Good Robot is unloading its full arsenal directly down the absorbing robot’s, errr, throat for lack of a better term. I think for gameplay balance, it really needs to slow the process down or even stop it from happening entirely. If you’re basically going to force a game over whenever you end up within a certain distance of this robot you might as well save the player time and just make it an instant death rather than a long doomed struggle.

    Thanks. If you can just address these issues before release I’ll definitely purchase it.

    • Droid says:

      Exponential growth really is a big problem. How about only making the first robot able to split, and if that one dies the next one, so that there is always a way the robot can still attack you, but none that gets too out-of-hand by just not noticing it quickly enough? Of course, the time it needs to split would have to go up, but since there is only one more with N robots, instead of N more, it’s still manageable as long as the reproduction time is not shorter than the time you need to kill one off.

  20. Love all these posts, and they make me want to play the game more and more.

  21. kasper says:

    Hey shamus,
    I came out of the shadows where I have been lurking forever as an avid fan (seriously, whenever i forget to check this site for a week my girlfriend asks me why I’ve been glued to my screen for 2 hours reading text and comment threads) to tell you two things:

    1. I will buy good robot on launch day and i will get as many people as I think will like the game to buy it on launch day too. I never knew how important launch day was on steam so thank you for enlightening me.

    2. In the spirit of the comment thread: minor nitpick over spelling
    In the dev blog link of pyrodactil’s good robot page I found this sentence: “It’s slipped my mind at them moment.”
    Probably should be THE moment. Better correct it before the 5th of April or you might get a 2% decreased sales figure due to grammar nazi boycot!

    Points are in reverse order of importance, obviously.

    Keep up the good work and know that I am a great fan of almost all the content this site has delivered over the years.

    Kasper

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *

*
*

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>