{"id":35937,"date":"2016-12-22T06:00:42","date_gmt":"2016-12-22T11:00:42","guid":{"rendered":"http:\/\/www.shamusyoung.com\/twentysidedtale\/?p=35937"},"modified":"2016-12-23T10:37:39","modified_gmt":"2016-12-23T15:37:39","slug":"crash-dot-com-part-7-the-source","status":"publish","type":"post","link":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=35937","title":{"rendered":"Crash Dot Com Part 7: The Source"},"content":{"rendered":"<p>It&#8217;s the year 2000, and I&#8217;m still working on the virtual mall. The original plan called for lining up deals with distributors, or retailers, or existing online storefronts. I think the idea was that John Business would rent virtual storefronts to people who wanted to sell goods in our mall. I&#8217;m guessing those deals have been hard to come by, since it&#8217;s been ages since I&#8217;ve been given pictures of new stores and products to translate into 3D. At the start of the project I made a couple of demonstration stores, and I haven&#8217;t been given any real ones since then.<\/p>\n<p>The lack of new content has given us some breathing room in the timetable. It&#8217;s been weeks since Roger &#8211; my boss and also a pretty good friend &#8211; stopped by my cubicle with a question or request from the people representing John Business, which means I have time to spend on other projects. <\/p>\n<h3>Security Through Obscurity<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/splash_coding.jpg' width=100% alt='Is this enough obscurity? I can make it more obscure if you like.' title='Is this enough obscurity? I can make it more obscure if you like.'\/><\/div><div class='mouseover-alt'>Is this enough obscurity? I can make it more obscure if you like.<\/div><\/p>\n<p>Roger really likes my work and has often wanted to get me the source code for our software. The coders guard the source jealously. Roger &#8211; being an executive of the company &#8211; technically has access to the source, but only in the sense that he has access to the machine where the source is hosted. He could drive an hour to downtown Boston to our hosting company, go through the layers of biometric security<span class='snote' title='1'>Which all seems pretty Star Trek here in 2000.<\/span> and look directly at the machine in question, but he has no idea where the files are stored, what they look like, or how to access them. <\/p>\n<p>Whenever he asks, the coders always reply with, &#8220;Why would you need that?&#8221; <\/p>\n<p><!--more-->In their defense, we still partly rely on a dash of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Security_through_obscurity\">security through obscurity<\/a> for our software. We already have a problem with kids hacking the client<span class='snote' title='2'>Note that in this case &#8220;client&#8221; means the software used by the kids at home. We&#8217;re not talking about a business client in the sense of a person who is a customer. Stupid confusing computer terminology.<\/span> to cheat at the user-made games. Rules like gravity and surface collision are enforced entirely on the client side, and so enterprising young hackers are able to fly and noclip<span class='snote' title='3'>Fly through walls.<\/span> at will. This issue is <em>really<\/em> important to all of those teenagers using our software as a platform for game development. They hate designing puzzles, mazes, and other challenges, only to have hackers circumvent the rules. Hackers are usually more interested in attention than &#8220;winning&#8221; the game, so when they cheat they often look for ways to ruin the game for other people. They cause confusion and complaints and generally make the software a little less fun for everyone else.<\/p>\n<p>There are layers of security to detect cheating, and counter-hacks to hide it, and security checks to counter the hacks, and hacks to counter the checks. This is a long-running game of cat-and-mouse and the developers <em>hate it<\/em> because they feel (correctly) that it&#8217;s a losing battle and a waste of their time. The coders know that obscurity makes for lousy security, but for various technical reasons it&#8217;s the only option available to us. A proper fix would take <em>years<\/em> and multiply the hardware requirements of our lightweight server by orders of magnitude. <\/p>\n<h3>The Server is Blind and Dumb<\/h3>\n<p><a href='http:\/\/host.activeworlds.com\/mauz\/public\/awpics.html'><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/cdc_aw1.jpg' width=100% alt='This is an actual screenshot from our software, sometime in 2000. If you want to see more, check out <a href=&apos;http:\/\/host.activeworlds.com\/mauz\/public\/awpics.html&apos;>this massive archive<\/a> maintained by user Mauz.' title='This is an actual screenshot from our software, sometime in 2000. If you want to see more, check out <a href=&apos;http:\/\/host.activeworlds.com\/mauz\/public\/awpics.html&apos;>this massive archive<\/a> maintained by user Mauz.'\/><\/div><\/a><div class='mouseover-alt'>This is an actual screenshot from our software, sometime in 2000. If you want to see more, check out <a href='http:\/\/host.activeworlds.com\/mauz\/public\/awpics.html'>this massive archive<\/a> maintained by user Mauz.<\/div><\/p>\n<p>Right now the server is pretty dumb. The advantage is that you can host thousands of worlds on a single Linux box. Hosting a world is a lot like hosting a single webpage. Adding more doesn&#8217;t really increase the server load. It&#8217;s number of users, not the number of worlds, that creates load problems. And if you&#8217;re lucky enough to have so many people that a single machine can&#8217;t handle them, then you&#8217;re making enough money to spin up a second machine to help balance the load. <\/p>\n<p>The server is technically &#8220;blind&#8221;. It knows where the objects are and where the users are, and it broadcasts this information to everyone. It knows there&#8217;s an object called &#8220;light_pole&#8221; at such-and-such a location, rotated to such-and-such a position, and that it&#8217;s owned by user so-and-so. But it has no clue how big a light_pole is, what it&#8217;s shaped like, what parts of it might be solid, what parts might be visible, or what all of these scripts attached to it might do. It also doesn&#8217;t know what users &#8220;look like&#8221;. It knows user Shamus is at location A and heading B using character model C, but it has no idea how big C is or what it&#8217;s shaped like. <\/p>\n<p>As a result, when you&#8217;re playing the game, your own computer is in charge of making sure you don&#8217;t cheat. This is the same problem developers run into with DRM: The user has complete control over their computer, and so if you try to restrict them using that same computer then they will be able to circumvent your will. The only way to put the rules beyond the user&#8217;s ability to tamper is to have them enforced by the server.<\/p>\n<p>If you wanted the server to enforce rules like collision, then it would need to be able to download and analyze all of those thousands of 3D models. That would mean keeping all of that geometry in memory, and performing expensive collision-checking on all of it. That task is actually pretty heavy duty. When an area is really complicated, your mid-range PC can actually slow down just checking collision for your character alone. What if the server had to do collision checks for every user against every nearby object for every one of the hundreds of worlds it was hosting? Centralized processing solutions for distributed users is something that <a href=\"https:\/\/en.wikipedia.org\/wiki\/SimCity_(2013_video_game)\">does not scale well<\/a>, particularly on the computers of 1999. <\/p>\n<p><a href='http:\/\/www.slideshare.net\/slantsixgames\/collision-detection-an-overview'><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/cdc_collision.jpg' width=100% alt='Collision is a fiendishly complex and CPU-intensive problem. This is one slide from a presentation on some of the theory behind it. Click for the whole set.' title='Collision is a fiendishly complex and CPU-intensive problem. This is one slide from a presentation on some of the theory behind it. Click for the whole set.'\/><\/div><\/a><div class='mouseover-alt'>Collision is a fiendishly complex and CPU-intensive problem. This is one slide from a presentation on some of the theory behind it. Click for the whole set.<\/div><\/p>\n<p>This is made harder by the fact that our engine accepts a lot of user-made content that can&#8217;t use normal optimizing tricks. Some models might contain huge numbers of polygons. These models can&#8217;t be guaranteed to be enclosed solids, which means there are certain geometric shortcuts that are available to typical videogames but not to us. These models are often built by inexperienced users who don&#8217;t understand common optimizing techniques. Worse, the whole system to parse 3D models is handled by our graphics engine, which can&#8217;t be ported to Linux, which is the preferred platform for hosting our server. <\/p>\n<p>Every time someone talks about stopping cheating, the coders begin listing all the costs associated with doing things the &#8220;Right Way&#8221; and the bosses capitulate before the coders are halfway done. It&#8217;s not an impossible problem to solve, but it&#8217;s so cost prohibitive that it may as well be.<\/p>\n<p>The point is that making the client a pain in the ass to tamper with is the best anyone can do to combat cheating. The coders just want to avoid things that would oblige them to spend even more time fighting this losing battle, such as having the source code leak out into the wild.<\/p>\n<p>Roger doesn&#8217;t know source code from Vogon poetry, but he&#8217;s adamant that he wants a copy of the source anyway. &#8220;What if both of you get hit by a bus?&#8221; he tells the coders. &#8220;We need access in case something happens to you.&#8221;<\/p>\n<h3>I&#8217;m In.<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/cdc_source.jpg' width=100% alt='This is a dramatic reenactment of real events.' title='This is a dramatic reenactment of real events.'\/><\/div><div class='mouseover-alt'>This is a dramatic reenactment of real events.<\/div><\/p>\n<p>One day Roger walks by my cubicle and drops a freshly-burned CD on my desk. It&#8217;s the source code for our &#8220;3D Browser&#8221; &#8211; the client. He doesn&#8217;t have any requests and there&#8217;s no reason I need this, but he knows exactly what he&#8217;s doing. Over the next few days, he conspicuously doesn&#8217;t have any work for me to do.<\/p>\n<p>Once I get it to compile I go to work on it. <\/p>\n<p>One of the problems with our software is that it was originally conceived and written in 1994. That&#8217;s the same year Doom II was released. Those early versions of the software were built on graphical systems that are now woefully out of date. Depending on how you count, the world of consumer graphics has been revolutionized three or four times since then. Most game developers are free to throw out their graphics engine when they feel it&#8217;s gotten too old, because they build each new game from scratch. But our world has been running almost non-stop since it opened to the public in <a href=\"https:\/\/en.wikipedia.org\/wiki\/Active_Worlds\">June of 1995<\/a>. There&#8217;s never been a point where we could throw out the old content and make a clean break. Most of our content is user-created these days, so there&#8217;s no way to wipe the slate clean without causing a revolt.<\/p>\n<p>We&#8217;re still using primitive 1994 lighting and rendering techniques. We can only have one global directional light source for an entire world, and it has to be full brightness and pure white. There&#8217;s no <a href=\"https:\/\/en.wikipedia.org\/wiki\/Specular_highlight\">specular lighting<\/a> or <a href=\"https:\/\/en.wikipedia.org\/wiki\/Reflection_mapping\">environment maps<\/a>. No fog. No control over the clipping plane. No <a href=\"https:\/\/en.wikipedia.org\/wiki\/Blend_modes#Addition\">additive blending<\/a> to create effects like fire. No <a href=\"https:\/\/en.wikipedia.org\/wiki\/Mipmap\">mip maps<\/a>. No shadows whatsoever, not even ultra-primitive blob shadows like games have been using for years.<\/p>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/cdc_tomb_raider.jpg' width=100% alt='Here&apos;s a shot from one of the early Tomb Raider games. Note the round shadow under Lara and the other characters. This is crucial for giving a sense of depth and scale and making the character feel properly &quot;grounded&quot;. These are blob shadows. It seems like a small thing, but it makes a huge difference.' title='Here&apos;s a shot from one of the early Tomb Raider games. Note the round shadow under Lara and the other characters. This is crucial for giving a sense of depth and scale and making the character feel properly &quot;grounded&quot;. These are blob shadows. It seems like a small thing, but it makes a huge difference.'\/><\/div><div class='mouseover-alt'>Here&apos;s a shot from one of the early Tomb Raider games. Note the round shadow under Lara and the other characters. This is crucial for giving a sense of depth and scale and making the character feel properly &quot;grounded&quot;. These are blob shadows. It seems like a small thing, but it makes a huge difference.<\/div><\/p>\n<p>The thing is, our software should technically be capable of a lot of these things. Last year our programmers spent about nine months upgrading us to the latest version of Renderware, our graphics engine<span class='snote' title='4'>This is the same graphics engine as Grand Theft Auto III, which will come out next year.<\/span>. It was our first upgrade since 1995. It was such a huge leap that huge parts of the software had to be re-written to accommodate the newer ways of doing things. This meant we spent about nine months stagnating, not giving users any new features to play with. Once the upgrade was over, there was a huge backlog of features and fixes that everyone needed. It&#8217;s been a year, and we&#8217;re <em>still<\/em> dealing with the fallout of that nine-month bottleneck.<\/p>\n<p>Worse, all of that work was just to get it working and backwards-compatible with all of those years of content. The program looks basically the same as it did five years ago<span class='snote' title='5'>The only visible difference I can remember is that if you have a graphics card you&#8217;ll now automatically get texture filtering, where stretched textures will be smoothed out rather than giving you ridiculously oversized pixels. Which is nice.<\/span>. The only change is that it can now use a graphics accelerator if the user has one. <\/p>\n<p>To rub salt in the wound, <a href=\"https:\/\/en.wikipedia.org\/wiki\/RenderWare\">Renderware<\/a> was just bought out<span class='snote' title='6'>Today it&#8217;s owned by EA.<\/span> and we will never get another update for it. We just spent most of last year upgrading our program into a graphical dead end. If we&#8217;d only known, we could have gone with another engine<span class='snote' title='7'>Or if anyone had asked me, I would have advocated for dumping the middleware and using OpenGL directly. The way we used it, Renderware wound up being little more than an ungainly wrapper for OpenGL.<\/span>. This will come back to bite us over the next decade.<\/p>\n<p>In the meantime, our graphics engine supposedly offers us a bunch of new effects that we&#8217;re not using. <\/p>\n<h3>The Fog<\/h3>\n<p><a href='https:\/\/www.penny-arcade.com\/comic\/1999\/02\/26'><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/cdc_fog.jpg' width=100% alt='This is from the original Unreal. In the early days, fog effects got a bad reputation because they were often used to create smothering draw distances. (Turok was notorious for this.) But when used properly and with restraint, it was a powerful effect for very little processing cost.' title='This is from the original Unreal. In the early days, fog effects got a bad reputation because they were often used to create smothering draw distances. (Turok was notorious for this.) But when used properly and with restraint, it was a powerful effect for very little processing cost.'\/><\/div><\/a><div class='mouseover-alt'>This is from the original Unreal. In the early days, fog effects got a bad reputation because they were often used to create smothering draw distances. (Turok was notorious for this.) But when used properly and with restraint, it was a powerful effect for very little processing cost.<\/div><\/p>\n<p>Now that I have the client source, I mess around with some of these new graphical features. I add fog. I add some lighting effects. I create a demo to show off what it might look like if we gave users control over the global light source. I add some basic modern touches like footstep sounds. After a few days, it makes for a dynamite demo. Mixed with some solid art, our client starts looking more like a modern-ish game and less like a leftover from 1996. <\/p>\n<p>Fog is the big attention-getter. We&#8217;ve always struggled with draw distance. If you allow the user to see too far, then all of that distant detail will drive their framerate into single digits. You can make the draw distance smaller, but then you have problems with pop-in, where large objects suddenly appear nearby as they enter your view radius. Here in the year 2000, the preferred way of coping with this problem is to use fog. Our software didn&#8217;t use fog, which means we had to split the difference between terrible framerate and terrible pop-in. <\/p>\n<p>With fog, you can hide all of that ugly pop-in behind thick fog and the lack of visibility feels like an atmospheric effect rather than an ugly limitation. It&#8217;s not a perfect solution and it isn&#8217;t applicable in all cases, but it really is a massive improvement over what we&#8217;ve been doing.<\/p>\n<p>This demo causes quite a stir in the company. The special build I made gets passed around and everyone has a moment of surprise when they see the client running with my enhanced effects in place. This produced the effect that Roger desired, which is that all the various factions in the company unite and start asking, &#8220;Hey, why don&#8217;t we add all this cool stuff?&#8221;<\/p>\n<p>Roland, our lead programmer, is put on the spot. This makes him look bad. I&#8217;ve had the code for three days and I&#8217;ve just blown everyone away with a graphics demo. The assumption is that these features are trivial and that it&#8217;s just three days of programming to make it all work.<\/p>\n<p>Technically the contents of The List&trade; have already been decided and the new build is still a few months away, but in response to what I&#8217;ve done we have some impromptu feature meetings that quickly escalate into regular feature meetings, and from there to an overhaul to The List&trade; that will incorporate my changes. Roland is put out by this. He spends much of the meeting on the defensive, patiently explaining that he&#8217;s not against any of these new features, but he insists that they will take time to add. I pretend to stay aloof during this discussion but secretly I&#8217;m cheering the whole process on. <\/p>\n<h3>Today, I&#8217;m the Bad Guy<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/shamus_2000_office.jpg' width=100% alt='Me, in my days of villainy.' title='Me, in my days of villainy.'\/><\/div><div class='mouseover-alt'>Me, in my days of villainy.<\/div><\/p>\n<p>This is pretty unfair to Roland. See, I&#8217;ve done the easy part. It&#8217;s only a few lines of code to enable fog. But you don&#8217;t want all worlds to be foggy all the time with the same color of fog at the same density. The work isn&#8217;t just to enable fog, but to turn it into a useful feature. This means adding a whole new section to the interface for controlling all the properties of fog. These properties must be sent to the server, stored, and then re-broadcast to all users. This means changes to the interface, the client, the server, and a few of the utility programs people use to manage their worlds. It means testing fog in different environments and solving various visual artifacts. <\/p>\n<p>Worse, the introduction of fog introduces a bunch of new questions that need to be answered. For example, the software will display a user&#8217;s name over their head, just like in an MMO. But now we come to the question: Should this name be obscured by fog, or should it always be rendered at full visibility? You can devise various situations where you might really want one or the other, depending on how the world is used. The reflexive answer from the bosses is always &#8220;make it an option&#8221;, but that means more changes to the interface, the client, the server, the tools. Keep in mind that this is just one question of many, and if we answer every question with &#8220;make it an option&#8221; we can easily make a huge load of work for ourselves. Having lots of options is good, but only if the options are useful, cover the most likely use cases, and are properly documented. It&#8217;s easy to overwhelm the user with useless dialog boxes. <\/p>\n<p>Does fog work on all systems? In the same way? What about people without graphics cards? Do some systems have glitches? How do we handle the fixed image drawn on the horizon<span class='snote' title='8'>Think of those hills you&#8217;d see in the distance in DOOM, that would scroll sideways as you turned your head. This is basically the same thing. It looked AMAZING in 1993 but looks pretty dated today.<\/span>? Should that be fogged? Or drawn in full brightness? Or hidden? Or make it yet another option?<\/p>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/shamus_2000_office3.jpg' width=100% alt='This was actually taken in 2005, but it&apos;s the office where I worked.' title='This was actually taken in 2005, but it&apos;s the office where I worked.'\/><\/div><div class='mouseover-alt'>This was actually taken in 2005, but it&apos;s the office where I worked.<\/div><\/p>\n<p>It takes ten minutes to enable fog, but it takes days of experimentation and test builds to turn it into a properly useful <strong>feature<\/strong>. I knocked everyone&#8217;s socks off with this demo, and now people are expecting miracles from Roland because I made it look easy.<\/p>\n<p>I admit I&#8217;m being a weasel right now. In my defense, I&#8217;ve long been advocating for graphical upgrades, but those features never made it onto The List&trade; because nobody really appreciated what an enormous difference they would make until they saw the demo. If this is what it takes to show people how awesome our software could look, then I need to do this if I don&#8217;t want our visuals to continue to languish for another half a decade. I&#8217;m not trying to make Roland look bad, I&#8217;m just trying to get people to care about visuals. People won&#8217;t allocate time in the schedule for something that &#8220;will look really cool, trust me&#8221;. <em>You have to show them what they&#8217;re going to get.<\/em> <\/p>\n<p>Also, I&#8217;ve wanted to be part of the coding staff for years, and this seems like my chance to get management to take me seriously. I&#8217;ve written a lot of incredibly useful tools over the years, but you can&#8217;t really show off your skills by making tools that only technical people can understand and appreciate. <\/p>\n<p>And deep down, I&#8217;m hoping I can do some coding because I&#8217;m going mad working on the virtual mall and I&#8217;d rather do basically anything else.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>It&#8217;s the year 2000, and I&#8217;m still working on the virtual mall. The original plan called for lining up deals with distributors, or retailers, or existing online storefronts. I think the idea was that John Business would rent virtual storefronts to people who wanted to sell goods in our mall. I&#8217;m guessing those deals have [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[],"class_list":["post-35937","post","type-post","status-publish","format-standard","hentry","category-personal"],"_links":{"self":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/35937","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=35937"}],"version-history":[{"count":0,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/35937\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=35937"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=35937"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=35937"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}