on Aug 10, 2011
You should know the drill by now. Movie first, followed by my response with links to the part of the speech in question.
Please, please, would someone from Quakecon log in to YouTube and CHANGE THE THUMBNAIL FOR THIS MOVIE? Thank you. Geeze.
“Every game has taken longer than the one before, and that cannot continue.”
This is, of course, one of the things I’ve been griping about incessantly for years now. I think everyone should have come to this conclusion about five years ago. Right now, Carmack is the only one saying it. (Or at least, the only one formulating a plan. So for EA’s brilliant plan is to continue to Spend Too Much, but make up for it by Charging Too Much. But that’s a rant for another time.) Other industry leaders will eventually follow Carmack, but I’m confident that the rash of studio closings over the last couple of years is a result of an entire industry that refused to see the wall they were about to run into. Even now, a few dozen studios and a few thousand jobs later, we’re still not seeing a real change in behavior that would bring these development times under control.
At this sluggish pace I imagine a few more development houses will be driven into the ground before industry leaders DO SOMETHING about it. And of course, if Microsoft or Sony hit us with another console generation, it might delay that realization even further. Because nobody will want to show up to the ball wearing last year’s graphics engine. Perish the thought.
I would love if the next generation of hardware wasn’t used to make anything look better, but simply used to make everything easier to develop. Imagine if Rage didn’t need that complicated four-tier staging system to bring textures from optical media to memory. However, until industry leaders repent, I’m sure any new hardware they get will be put into MOAR GRAFIX, and not used to give the artists and programmers a break.
“16k of ROM.”
Like he said, the high-end Atari 2600 games fit everything into 16K of memory, and it took some fancy technological tricks to make even that possible. That means that River Raid, that innovative game by plucky underdog development house Activision, was so small that its memory banks couldn’t even hold the HTML source for the web page you’re reading right now. (Not even close.)
“We don’t have detail maps […]”
I give a basic visual demonstration of detail textures here.
So, even though Blu-Ray discs can hold a lot more data, id isn’t going to be filling up the Rage BR disc with higher-detail data. Partly because they didn’t have time to re-code all those terabytes of data. Carmack didn’t say so, but I also suspect that this would be pointless – higher detail textures would take even longer to come of the disc, and that’s already their big bottleneck.
Oops. Sorry Sony. Well, your hardware is more expensive, you came to the market late, and fewer people want to develop on it, but at least you have all that extra storage space! I mean, nobody can be bothered to fill that space because games are already too damn expensive as it is, but it’s there, right?
Again, I’m hard on Sony because this console generation should have been theirs. The PS2 is probably the greatest console ever made (going by longevity, quality, and library size) and a proper follow-up would have been enough to continue their domination. But they tried to win by cock-blocking their competition, instead of trying to release a superior product. The result was bad for everyone.
(Aside: We really need a less ribald idiom for “cock-blocking”: To frustrate or hinder a rival, even to one’s own disadvantage. To childishly destroy to prevent the success of another out of envy or spite. A kind of griefing. I know English is a second language for some of you, so I’ll ask: Do any other languages have one we can
Carmack is talking about “static software analysis tools”. That’s a program that looks at your source code, analyzes it, and lets you know about possible dangers, inefficiencies, or bad practices. Just recently I was messing with cppcheck, which was suggested by reader Kdansky. This is a free, open-source version of the stuff Carmack is talking about. It’s actually kind of irritating to use, like having the world’s most pedantic nitpicker go over your code and say the same thing 11 dozen times. But it’s also good for you, for all the reasons Carmack mentions.
People praise Carmack and give credit for his stable code to his high intelligence, and I’m sure that factors into it. But I’m sure there are a handful of other coders in the industry with IQ’s in the same ballpark. Geniuses aren’t that rare. No, I think the big reason his code is so excellent is simply because he cares. Other developers have the ability to make stable, efficient, maintainable game engines, but they need to have a passion for it. It’s the sort of “are you the sort of person who will vacuum behind the couch” (metaphorically speaking) or would you rather move on from vacuuming to do the “fun” stuff?
I suppose it also helps that Carmack has a great deal of political power in the company, which gives him the muscle to keep games in development until he’s happy with the technology, not until some non-gaming manager decides, “Bah. Close enough. Let’s release it.” This passion for quality has given them great financial success, which in turn allowed them to continue to spend the time needed to build well-engineered games.
I always thought I was only attached to Microsoft Visual Studio because it was the programming environment I’d used the most, but as I’ve been learning over the past year, Microsoft really does make outstanding development tools. Their dev tools are very different from their other consumer-level software like word processing, video editing, media playing, and operating systems. In sixteen years of use, I’ve never had Visual Studio crash on its own, I’ve never had my project files corrupted, and I’ve never had it give me baffling or incorrect error messages*. I doubt anyone using Microsoft Word for the same length of time could make a similar claim. Sadly, I’m using the Hippie Freeloader edition of Visual Studio, so I don’t have access to the Microsoft tools he’s talking about.
* The compiler itself gives mysterious error messages all the time, but that’s more an issue with C++ than with Microsoft. I don’t think MS is uniquely bad in this regard.
Carmack is talking about a particular kind of bug that creeps into software, particularly 3D games. It works like this. Say you’ve got a few lines of code for moving the player around:
1 2 3
new_position.x = player_position.x + movement.x; new_position.y = player_position.y + movement.y; new_position.z = player_position.z + movement.z;
Most programmers will simply type the first line, then copy & paste it twice, changing the .x bits to .y and .z. I once had a boss who passed along some wisdom from one of his instructors in college, “Most software bugs come from copy & pasting code.” It sounded strange at the time, but over the years I’ve come to see how true it is. In the above example, the math involved is simple enough that you’re unlikely to make a blunder, but in a larger and more complex operation the odds of forgetting to change an .x or of transposing a .y and a .z increase.
The problem with copy-paste code is that it is, by its very nature, code that already compiles. If I accidentally fat-finger type “neq_position” instead of “new_position”, it won’t compile and I’ll fix the problem right away. But by copying selections of working code you can omit error checking, leave variables unchanged, accidentally perpetrate variable name collisions (where the same variable is used to two totally different things) or leave out proper memory management steps – all of which are things you’re unlikely to do if you’re forced to type out each line.
I don’t think the solution is to simply never copy & paste code. I think the solution is to recognize how many bugs arise from bad copy-paste, and to realize you’re doing something dangerous. A healthy sense of paranoia can probably help you to catch a lot of blunders before you move on, leaving a landmine for your future self.
Talking about game crashes, he mentions a hypothetical programmer encountering a situation, “Oh, but I didn’t expect that we would be changing weapons while falling to my death while being crushed by a vehicle that’s targeting me.”
He was using it as an example, but I’ve often found that movement code can be so much more complex that you would normally expect. Yes, the deep bowels of the engine can have areas of terrifying mathematical complexity, but most of it is fairly linear. Player movement is highly conditional.
The jump button makes you jump. Unless you’re falling, or recovering from a fall of sufficient height to inflict damage. But if you’re running, then the jump button needs to delay a step, or wait for a button release so you can charge the jump. Except if you’re sliding down a slope, in which case it needs to be immediate, unless you’re not quite up to speed and you’ve only slid a few feet. But if you’re in the process of charging a jump and a situation arises where you would no longer be able to perform a jump such as sliding or encountering an edge, then the jump should begin right away or the controls will feel “unresponsive” to the player. Although, this immediate jump shouldn’t be allowed if the player has just sustained splash damage, since that means an enemy is trying to knock them off the ledge, and the player shouldn’t be able to exploit around that by charging a jump just before impact. However, the jump (if allowed, and not-charged) needs to be reduced in height when the player is actively carrying one of the “heavy” class weapons. If the jump takes place during a weapon-switch, the switch should complete instantly to avoid animation problems, but the player should have a cooldown placed on their weapon to prevent them from using it until the point where the weapon-switch animation would have completed. Also, if swimming in water deeper than waist-height, the jump button should be used to simply “swim up”, and if swimming up at the edge of a deep pool of water, the player should be allowed a jump of double height so they can leap up onto dry land…
GAH! Now we’ve got two pages of code to handle ONE BUTTON. The indentation on that block of code is going to be insane. Input and movement code gets like this, and a huge tangle of interfacing conditions can often be far more daunting than a big ‘ol page of math.
Worse, it’s a devil of a problem to actually test all of this stuff.
“What I want to be working towards is a case where we can say, the game cannot… uh… cannot have an exception.” I’ll bet the reason that he faltered there is that he was deciding if he wanted to use the word “exception”, or the more colloquial, non-techie term “crash”. The two are not quite the same thing, and when discussing complex subjects like this there’s always the question of whether you want to be precise (because your fellow programmers love to correct you) or if you want to be understood by the masses. (Because most of them have no idea what an exception is.)
What he’s proposing is a lofty goal. He wants to take Rage – the biggest id Software project to date, which involves many programmers and millions of lines of code – and make it unbreakable. There may be bugs. Maybe switching weapons, while reloading, while falling, while picking up a new weapon on the way down, will still lead to some strange behavior. But it shouldn’t ever crash, lockup, or become corrupted in such a way that the program needs to be restarted.
Again, I really hope the id Software technology and culture can rub off on Bethesda. It would mean nothing but good things for the rest of us.
“The Doom 3 source code will be released this year.”
I’ve been waiting for this for a long time. When Doom 3 was new, I really thought the unified lighting would open up this new world of procedural (or at least dynamically generated) games. No more building fixed light maps in a lengthy compile. I played around with the editor when it first came out. I’d wanted to make something like System Shock 2 gameplay with randomized layouts, but I found I didn’t have the fine-grain control I needed to do a real proof-of-concept. I decided to shelve the project and revisit it when the Doom 3 source went public. (I didn’t realize how long it would take. I suspect Rage’s lengthy development delayed the release of the source.)
Sadly, now that it’s coming out I don’t have time to do anything with it. Between the book, Project Frontier, this website, and some of the other contract work I’m doing, the hours just aren’t there. Hopefully someone else will step in and make it happen. I’m sure I can’t be the only person with something like this in mind. It should be easier now than it was in 2005, since you don’t need to be nearly as picky about overdraw. There are lots of CPU and GPU cycles to blow on un-optimized level geometry.
And we come to the end. I tried to make that as accessible as possible. I apologize if I got too jargon-heavy there at the end. Hopefully you were able to enjoy his presentation at least a fraction as much as I did.
Shamus Young is an old-school OpenGL programmer, author, and composer. He runs this site and if anything is broken you should probably blame him.