Crash Dot Com Part 7: The Source

By Shamus Posted Thursday Dec 22, 2016

Filed under: Personal 66 comments

It’s the year 2000, and I’m still working on the virtual mall. The original plan called for lining up deals with distributors, or retailers, or existing online storefronts. I think the idea was that John Business would rent virtual storefronts to people who wanted to sell goods in our mall. I’m guessing those deals have been hard to come by, since it’s been ages since I’ve been given pictures of new stores and products to translate into 3D. At the start of the project I made a couple of demonstration stores, and I haven’t been given any real ones since then.

The lack of new content has given us some breathing room in the timetable. It’s been weeks since Roger – my boss and also a pretty good friend – stopped by my cubicle with a question or request from the people representing John Business, which means I have time to spend on other projects.

Security Through Obscurity

Is this enough obscurity? I can make it more obscure if you like.
Is this enough obscurity? I can make it more obscure if you like.

Roger really likes my work and has often wanted to get me the source code for our software. The coders guard the source jealously. Roger – being an executive of the company – technically has access to the source, but only in the sense that he has access to the machine where the source is hosted. He could drive an hour to downtown Boston to our hosting company, go through the layers of biometric securityWhich all seems pretty Star Trek here in 2000. and look directly at the machine in question, but he has no idea where the files are stored, what they look like, or how to access them.

Whenever he asks, the coders always reply with, “Why would you need that?”

In their defense, we still partly rely on a dash of security through obscurity for our software. We already have a problem with kids hacking the clientNote that in this case “client” means the software used by the kids at home. We’re not talking about a business client in the sense of a person who is a customer. Stupid confusing computer terminology. to cheat at the user-made games. Rules like gravity and surface collision are enforced entirely on the client side, and so enterprising young hackers are able to fly and noclipFly through walls. at will. This issue is really important to all of those teenagers using our software as a platform for game development. They hate designing puzzles, mazes, and other challenges, only to have hackers circumvent the rules. Hackers are usually more interested in attention than “winning” the game, so when they cheat they often look for ways to ruin the game for other people. They cause confusion and complaints and generally make the software a little less fun for everyone else.

There are layers of security to detect cheating, and counter-hacks to hide it, and security checks to counter the hacks, and hacks to counter the checks. This is a long-running game of cat-and-mouse and the developers hate it because they feel (correctly) that it’s a losing battle and a waste of their time. The coders know that obscurity makes for lousy security, but for various technical reasons it’s the only option available to us. A proper fix would take years and multiply the hardware requirements of our lightweight server by orders of magnitude.

The Server is Blind and Dumb

This is an actual screenshot from our software, sometime in 2000. If you want to see more, check out <a href='http://host.activeworlds.com/mauz/public/awpics.html'>this massive archive</a> maintained by user Mauz.
This is an actual screenshot from our software, sometime in 2000. If you want to see more, check out this massive archive maintained by user Mauz.

Right now the server is pretty dumb. The advantage is that you can host thousands of worlds on a single Linux box. Hosting a world is a lot like hosting a single webpage. Adding more doesn’t really increase the server load. It’s number of users, not the number of worlds, that creates load problems. And if you’re lucky enough to have so many people that a single machine can’t handle them, then you’re making enough money to spin up a second machine to help balance the load.

The server is technically “blind”. It knows where the objects are and where the users are, and it broadcasts this information to everyone. It knows there’s an object called “light_pole” at such-and-such a location, rotated to such-and-such a position, and that it’s owned by user so-and-so. But it has no clue how big a light_pole is, what it’s shaped like, what parts of it might be solid, what parts might be visible, or what all of these scripts attached to it might do. It also doesn’t know what users “look like”. It knows user Shamus is at location A and heading B using character model C, but it has no idea how big C is or what it’s shaped like.

As a result, when you’re playing the game, your own computer is in charge of making sure you don’t cheat. This is the same problem developers run into with DRM: The user has complete control over their computer, and so if you try to restrict them using that same computer then they will be able to circumvent your will. The only way to put the rules beyond the user’s ability to tamper is to have them enforced by the server.

If you wanted the server to enforce rules like collision, then it would need to be able to download and analyze all of those thousands of 3D models. That would mean keeping all of that geometry in memory, and performing expensive collision-checking on all of it. That task is actually pretty heavy duty. When an area is really complicated, your mid-range PC can actually slow down just checking collision for your character alone. What if the server had to do collision checks for every user against every nearby object for every one of the hundreds of worlds it was hosting? Centralized processing solutions for distributed users is something that does not scale well, particularly on the computers of 1999.

Collision is a fiendishly complex and CPU-intensive problem. This is one slide from a presentation on some of the theory behind it. Click for the whole set.
Collision is a fiendishly complex and CPU-intensive problem. This is one slide from a presentation on some of the theory behind it. Click for the whole set.

This is made harder by the fact that our engine accepts a lot of user-made content that can’t use normal optimizing tricks. Some models might contain huge numbers of polygons. These models can’t be guaranteed to be enclosed solids, which means there are certain geometric shortcuts that are available to typical videogames but not to us. These models are often built by inexperienced users who don’t understand common optimizing techniques. Worse, the whole system to parse 3D models is handled by our graphics engine, which can’t be ported to Linux, which is the preferred platform for hosting our server.

Every time someone talks about stopping cheating, the coders begin listing all the costs associated with doing things the “Right Way” and the bosses capitulate before the coders are halfway done. It’s not an impossible problem to solve, but it’s so cost prohibitive that it may as well be.

The point is that making the client a pain in the ass to tamper with is the best anyone can do to combat cheating. The coders just want to avoid things that would oblige them to spend even more time fighting this losing battle, such as having the source code leak out into the wild.

Roger doesn’t know source code from Vogon poetry, but he’s adamant that he wants a copy of the source anyway. “What if both of you get hit by a bus?” he tells the coders. “We need access in case something happens to you.”

I’m In.

This is a dramatic reenactment of real events.
This is a dramatic reenactment of real events.

One day Roger walks by my cubicle and drops a freshly-burned CD on my desk. It’s the source code for our “3D Browser” – the client. He doesn’t have any requests and there’s no reason I need this, but he knows exactly what he’s doing. Over the next few days, he conspicuously doesn’t have any work for me to do.

Once I get it to compile I go to work on it.

One of the problems with our software is that it was originally conceived and written in 1994. That’s the same year Doom II was released. Those early versions of the software were built on graphical systems that are now woefully out of date. Depending on how you count, the world of consumer graphics has been revolutionized three or four times since then. Most game developers are free to throw out their graphics engine when they feel it’s gotten too old, because they build each new game from scratch. But our world has been running almost non-stop since it opened to the public in June of 1995. There’s never been a point where we could throw out the old content and make a clean break. Most of our content is user-created these days, so there’s no way to wipe the slate clean without causing a revolt.

We’re still using primitive 1994 lighting and rendering techniques. We can only have one global directional light source for an entire world, and it has to be full brightness and pure white. There’s no specular lighting or environment maps. No fog. No control over the clipping plane. No additive blending to create effects like fire. No mip maps. No shadows whatsoever, not even ultra-primitive blob shadows like games have been using for years.

Here's a shot from one of the early Tomb Raider games. Note the round shadow under Lara and the other characters. This is crucial for giving a sense of depth and scale and making the character feel properly "grounded". These are blob shadows. It seems like a small thing, but it makes a huge difference.
Here's a shot from one of the early Tomb Raider games. Note the round shadow under Lara and the other characters. This is crucial for giving a sense of depth and scale and making the character feel properly "grounded". These are blob shadows. It seems like a small thing, but it makes a huge difference.

The thing is, our software should technically be capable of a lot of these things. Last year our programmers spent about nine months upgrading us to the latest version of Renderware, our graphics engineThis is the same graphics engine as Grand Theft Auto III, which will come out next year.. It was our first upgrade since 1995. It was such a huge leap that huge parts of the software had to be re-written to accommodate the newer ways of doing things. This meant we spent about nine months stagnating, not giving users any new features to play with. Once the upgrade was over, there was a huge backlog of features and fixes that everyone needed. It’s been a year, and we’re still dealing with the fallout of that nine-month bottleneck.

Worse, all of that work was just to get it working and backwards-compatible with all of those years of content. The program looks basically the same as it did five years agoThe only visible difference I can remember is that if you have a graphics card you’ll now automatically get texture filtering, where stretched textures will be smoothed out rather than giving you ridiculously oversized pixels. Which is nice.. The only change is that it can now use a graphics accelerator if the user has one.

To rub salt in the wound, Renderware was just bought outToday it’s owned by EA. and we will never get another update for it. We just spent most of last year upgrading our program into a graphical dead end. If we’d only known, we could have gone with another engineOr if anyone had asked me, I would have advocated for dumping the middleware and using OpenGL directly. The way we used it, Renderware wound up being little more than an ungainly wrapper for OpenGL.. This will come back to bite us over the next decade.

In the meantime, our graphics engine supposedly offers us a bunch of new effects that we’re not using.

The Fog

This is from the original Unreal. In the early days, fog effects got a bad reputation because they were often used to create smothering draw distances. (Turok was notorious for this.) But when used properly and with restraint, it was a powerful effect for very little processing cost.
This is from the original Unreal. In the early days, fog effects got a bad reputation because they were often used to create smothering draw distances. (Turok was notorious for this.) But when used properly and with restraint, it was a powerful effect for very little processing cost.

Now that I have the client source, I mess around with some of these new graphical features. I add fog. I add some lighting effects. I create a demo to show off what it might look like if we gave users control over the global light source. I add some basic modern touches like footstep sounds. After a few days, it makes for a dynamite demo. Mixed with some solid art, our client starts looking more like a modern-ish game and less like a leftover from 1996.

Fog is the big attention-getter. We’ve always struggled with draw distance. If you allow the user to see too far, then all of that distant detail will drive their framerate into single digits. You can make the draw distance smaller, but then you have problems with pop-in, where large objects suddenly appear nearby as they enter your view radius. Here in the year 2000, the preferred way of coping with this problem is to use fog. Our software didn’t use fog, which means we had to split the difference between terrible framerate and terrible pop-in.

With fog, you can hide all of that ugly pop-in behind thick fog and the lack of visibility feels like an atmospheric effect rather than an ugly limitation. It’s not a perfect solution and it isn’t applicable in all cases, but it really is a massive improvement over what we’ve been doing.

This demo causes quite a stir in the company. The special build I made gets passed around and everyone has a moment of surprise when they see the client running with my enhanced effects in place. This produced the effect that Roger desired, which is that all the various factions in the company unite and start asking, “Hey, why don’t we add all this cool stuff?”

Roland, our lead programmer, is put on the spot. This makes him look bad. I’ve had the code for three days and I’ve just blown everyone away with a graphics demo. The assumption is that these features are trivial and that it’s just three days of programming to make it all work.

Technically the contents of The List™ have already been decided and the new build is still a few months away, but in response to what I’ve done we have some impromptu feature meetings that quickly escalate into regular feature meetings, and from there to an overhaul to The List™ that will incorporate my changes. Roland is put out by this. He spends much of the meeting on the defensive, patiently explaining that he’s not against any of these new features, but he insists that they will take time to add. I pretend to stay aloof during this discussion but secretly I’m cheering the whole process on.

Today, I’m the Bad Guy

Me, in my days of villainy.
Me, in my days of villainy.

This is pretty unfair to Roland. See, I’ve done the easy part. It’s only a few lines of code to enable fog. But you don’t want all worlds to be foggy all the time with the same color of fog at the same density. The work isn’t just to enable fog, but to turn it into a useful feature. This means adding a whole new section to the interface for controlling all the properties of fog. These properties must be sent to the server, stored, and then re-broadcast to all users. This means changes to the interface, the client, the server, and a few of the utility programs people use to manage their worlds. It means testing fog in different environments and solving various visual artifacts.

Worse, the introduction of fog introduces a bunch of new questions that need to be answered. For example, the software will display a user’s name over their head, just like in an MMO. But now we come to the question: Should this name be obscured by fog, or should it always be rendered at full visibility? You can devise various situations where you might really want one or the other, depending on how the world is used. The reflexive answer from the bosses is always “make it an option”, but that means more changes to the interface, the client, the server, the tools. Keep in mind that this is just one question of many, and if we answer every question with “make it an option” we can easily make a huge load of work for ourselves. Having lots of options is good, but only if the options are useful, cover the most likely use cases, and are properly documented. It’s easy to overwhelm the user with useless dialog boxes.

Does fog work on all systems? In the same way? What about people without graphics cards? Do some systems have glitches? How do we handle the fixed image drawn on the horizonThink of those hills you’d see in the distance in DOOM, that would scroll sideways as you turned your head. This is basically the same thing. It looked AMAZING in 1993 but looks pretty dated today.? Should that be fogged? Or drawn in full brightness? Or hidden? Or make it yet another option?

This was actually taken in 2005, but it's the office where I worked.
This was actually taken in 2005, but it's the office where I worked.

It takes ten minutes to enable fog, but it takes days of experimentation and test builds to turn it into a properly useful feature. I knocked everyone’s socks off with this demo, and now people are expecting miracles from Roland because I made it look easy.

I admit I’m being a weasel right now. In my defense, I’ve long been advocating for graphical upgrades, but those features never made it onto The List™ because nobody really appreciated what an enormous difference they would make until they saw the demo. If this is what it takes to show people how awesome our software could look, then I need to do this if I don’t want our visuals to continue to languish for another half a decade. I’m not trying to make Roland look bad, I’m just trying to get people to care about visuals. People won’t allocate time in the schedule for something that “will look really cool, trust me”. You have to show them what they’re going to get.

Also, I’ve wanted to be part of the coding staff for years, and this seems like my chance to get management to take me seriously. I’ve written a lot of incredibly useful tools over the years, but you can’t really show off your skills by making tools that only technical people can understand and appreciate.

And deep down, I’m hoping I can do some coding because I’m going mad working on the virtual mall and I’d rather do basically anything else.

 

Footnotes:

[1] Which all seems pretty Star Trek here in 2000.

[2] Note that in this case “client” means the software used by the kids at home. We’re not talking about a business client in the sense of a person who is a customer. Stupid confusing computer terminology.

[3] Fly through walls.

[4] This is the same graphics engine as Grand Theft Auto III, which will come out next year.

[5] The only visible difference I can remember is that if you have a graphics card you’ll now automatically get texture filtering, where stretched textures will be smoothed out rather than giving you ridiculously oversized pixels. Which is nice.

[6] Today it’s owned by EA.

[7] Or if anyone had asked me, I would have advocated for dumping the middleware and using OpenGL directly. The way we used it, Renderware wound up being little more than an ungainly wrapper for OpenGL.

[8] Think of those hills you’d see in the distance in DOOM, that would scroll sideways as you turned your head. This is basically the same thing. It looked AMAZING in 1993 but looks pretty dated today.



From The Archives:
 

66 thoughts on “Crash Dot Com Part 7: The Source

  1. Da Mage says:

    In the days when knocking your screen over could kill a man.

    1. Echo Tango says:

      My dad had something like a….24-inch[1] monitor? 30-inch? I swear that thing weighed like, 80 pounds.

      [1] Back in the days of 4×3 monitors, this was near the upper limit for size!

      1. Jonathan says:

        I LIKE my 19″ 4×3 monitor. I’ve thought about replacing it, but it’s hard to find a newtech 4×3.
        Widescreen is wasted space and is suboptimal for reading, or for playing games made before ~2003.

        1. Daemian Lucifer says:

          Widescreen is pretty good for mutlitasking though.And when it comes to reading,you can always adjust the text to fill the whole screen,while feeling natural to read.You only have to move a bit more back than with a regular screen.

        2. Echo Tango says:

          Actually, I’ve found that splitting my windows so they are one on the left half of the screen, and one on the right half of the screen, is perfect for reading. Probably because half of a 1920×1080 screen is almost the same aspect ratio of an 8.5″x11″ piece of paper. (0.8-ish vs 0.9-ish)

    2. boz says:

      I miss those monitors. After more than a decade of LCD tech and we are barely getting to the performance of top of the line CRT monitors of the past. Between color differences, refresh rate issues, native resolution blur, response time artifacts and display lag, I wish there was someone still producing CRT monitors.

      1. Zaxares says:

        The main thing that annoys me about LCD screens is that they only work optimally when you’re sitting directly in front of them. If you try and look at the screen standing up, from the sides, or below it, it becomes much more indistinct and hard to see. In the 10 odd years since CRT monitors vanished from the computing world, LCD screens still have not overcome this problem.

        1. Daemian Lucifer says:

          Thats not true.Ever since tft lcds have become a thing,you could watch them from a lot of angles and still see clearly.Heck,Ive just read your comment by standing up and about 60° to the side with no problems.And my monitor is not new,Ive had it for 7 years now.

  2. Cozzer says:

    What terrible evilness. The very face of evil.

    Working as a programmer, I’ve been both the Shamus and the Roland of similar stories, so I can really appreciate both points of view. :P

    1. Kylroy says:

      “… it makes for a dynamite demo.”

      This is reminding me of how the Dungeon Siege team secured a contract from Microsoft by putting together a “demo reel” of their game that was actually just a movie – they had art assets but no actual *game* designed yet, so they just focused on making the best pitch reel they could.

      1. Primogenitor says:

        Which in turn reminds me of all the “bullshot” for games that we’ve been seeing over recent years – staged/pre-rendered E3 trailers that prompt people to put a pre-order in for a game that is released buggy and broken.

        See Shamus, Arkham Knight was your fault! :p

        1. Kylroy says:

          I think the bullshot problem has decreased as in-game graphics have gotten better – no point gilding the lily. Plus, it’s entirely possible for a game to deliver it’s promised graphics and still be a buggy and broken mess.

          1. Agammamon says:

            Or, you know, Aliens: Colonial Marines, Watchdogs, TW3, Borderlands 2, No Man’s Sky, among several other high profile AAA titles that got the bullshot treatment to such an extent that (bugs aside) gamers cried foul when those titles actually got released.

            1. Jeff says:

              Did I get into BL2 late or something? ’cause that game was awesome, and I’m pretty sure the folks I played with all preordered after we’d been playing BL1 together.

            2. Ciennas says:

              Hang on, Borderlands 2? I didn’t hear a thing about that one. What happened there?

              1. Agammamon says:

                Pre-release screenshots of how awesome the games AA would look – that were taken by significantly uprezzing.

            3. Tuck says:

              TW3? Do you mean something other than The Witcher 3…? Because that game received, and continues to receive, almost universal acclaim for its quality.

              1. Agammamon says:

                Bullshotting and general marketing shenanigans are not limited to games that end up actually being crap.

                TW3 was made to look uber-pretty in the marketing materials – more than it actually was IRL.

                That it was still pretty good-looking doesn’t change the fact that the marketing shots were sketchy.

                This was in counter to Kylroy’s assertion that games are generally good enough looking that publishers/studios don’t feel the need to ‘gild the lily’ anymore – they do.

                1. Blue_Pie_Ninja says:

                  Wasn’t the reason for downgrading The Witcher 3’s graphics was because the consoles couldn’t run the game with the uber-pretty graphics? I read somewhere that that demo was actually running on a high end PC but when the started to port it to consoles they weren’t able to match the same graphical fidelity so they turned it down.

      2. AndrewCC says:

        Ah, good ol’ Dungeon Siege, the game where if you turn on the aggresive setting for the party AI, the game basically plays itself with no need for your input thanks to the generous agro range of your characters and the groups of enemies evenly spaced through the corridor-like levels.

        What I’m saying is that game needed more game in it, and less oh-look-our-world-is-a-string-of-levels-with-no-loading-screens spectacle.

      3. Mousazz says:

        In the recent GameOn expo in Lithuania, one of the programmers of Awesomenauts led a presentation about his Ronimo’s general game development practices, and he showed us an example of a presentation reel for a potential video game to show to publishers. I don’t remember what it was called (I think it was “Road rage” or something similar?), but the general gist was this 2D bright, colorful game, where the main character drove a red monster truck, did sick flips off ramps and ran over zombie hordes. Of course, that video was completely pre-rendered by the art-team, and no actual code was made.

        I understand the need to present something for potential publishers, but I wished I asked him back then about his opinion on bullshots released for consumers…

    2. Mephane says:

      I can also relate. The “I’ve spent a few days making a quick demo of what our software could be like” can easily make non-technical people think that the programmers are doing something wrong for not already having all of this implemented, when indeed as Shamus explained, it takes a lot more work to make something a real feature, than just a proof of concept. And sometimes the proof of concept ends up not a viable feature at all due to other constraints that are not relevant for such a demonstration.

      1. Tuck says:

        I once spent half a day writing a quick working demo of a web page interface for a conference. Of course, this demo code then became an integral part of the main software and wasn’t properly developed for months… :)

  3. Galad says:

    This feels like a lull in this Greek Tragedy of a story, where our Intrepid Protagonist is still hoping that the light at the end of the tunnel is not an oncoming train. We’re past the midpoint, aren’t we?

  4. shiroax says:

    I think you forgot to add a break. The whole thing is on the front page.

  5. shiroax says:

    That pic sure makes you look villainous. Also, Days of Villainy should be a band.

    Btw, who’s in the pictures?

    1. MichaelGC says:

      Heh – I thought I recognised that photo. I won’t spoil for anyone trying to guess but you can get the answers from the hovertext caption halfway down this page:

      http://www.shamusyoung.com/twentysidedtale/?p=18952

  6. Mr Compassionate says:

    It’s a tricky one. You had a choice between being a bad colleague or a bad employee and ultimately your colleague isn’t paying your wage and your boss sounds like a cool guy who believed in you. Saving Roland would have involved derailing the conversation and making a patronising defence on his behalf.

    I’ve been that guy before. Two new employees sitting next to eachother being evaluated on their sales figures, it was clear we were being pitted. It was noticeably depressing and damaging to his self esteem and I feel sorry for that but again, bad colleague or bad employee. All I could do was be friendly and supportive. It’s terrible of me to say but I silently felt a tiny pang of joy when I was told he quit, because the competition was gone. Everybody under these circumstances becomes either the bad guy or the victim.

    1. Syal says:

      I’m a little lost as to how much of this Shamus knew at the time. If he hadn’t seen the source before, was the obfuscation security common knowledge?

      1. EricF says:

        The fact that Shamus hadn’t seen the source is the security feature – limiting who gets the source code was their primary security feature.

      2. silver Harloe says:

        Though I’m reminded of an old IT saying:
        “Security by obscurity isn’t.”

  7. Daemian Lucifer says:

    Which all seems pretty Star Trek here in 2000.

    Ahahahastar trek and security!Thats rich!

    Yes,yes,I know you meant the look of it is futuristicy,but just having star trek and security in the same sentence makes me laugh.

    1. MichaelGC says:

      Aye right. For physical security it’s only ever two geezers in gold with a combined life expectancy measured in minutes. For electronic security they do at least try to limit access by only handing out the unchangeable passwords to the shifty-looking and the obviously traitorous.

      1. Agammamon says:

        You young pups. Pepperidge Farms remembers when the geezers were always in red.

        1. MichaelGC says:

          Aye, I sacrificed accuracy for alliteration. Style over substance! My bad. :D

  8. geri_khan says:

    Oooh, it’s Active Worlds you’re talking about. An interesting story just got more interesting with a bit of context – I did get the feeling you were talking about something in the same vein as Second Life.

    Though I never played Active Worlds. I only know about it from this Vinesauce video from earlier this year.

    Now I’m looking forward to knowing if the mall was ever completed and is still live on the servers… but based on how the story is going, I’m going to anticipate the answer is No.

  9. Nessus says:

    Draw distance seems like it’s one of those never ending problems. Every time hardware gets a little better, devs rush to use up the entire thing adding more details to be rendered, leaving nothing for performance improvement, so were perpetually stuck in the same place in terms of frame rates, loading, etc.

    Even today, I don’t think I can remember a single game I’ve played in at least the last 10 years that didn’t have some amount of noticeable pop-in (there probably were a few, but GD if every game I can actually remember had some, so it’s functionally as a bad as “every single game”). DOF is the the new fog when it comes to hiding it, but even with that it’s still noticeable most of the time. Pop-in just seems to be something devs in general have come to just accept as a “cost of doing business”, rather than an optimization issue.

    To me this seems weird. Like: all that work going into improving shaders and polycounts and such in the name of “realism” or “immersion”, only to just allow the illusion be blatantly broken anyway by pop-in. I see so many games that look absolutely stunning… until you walk over a hill or down the wrong street, and suddenly you’re watching a car mesh or a ground texture or a group of trees crunch itself into being like some sort of surreal claymation effect.

    It’s like if filmmakers as a group all up and decided that it was just too much of a PITA to make sure the boom mike was NEVER in frame, so they decided that it was “acceptable” as long as the number of effected shots and/or the amount of the mike protruding into frame is below some arbitrary %.

    The ubiquity of pop-in, for me has become an “elephant in the room” issue. It’s a thing that’s obviously a major technical flaw, and obviously rampant, but the developer side seems weirdly intent on ignoring it.

    1. Kylroy says:

      I’ve seen games attempt to implement detail tiers to save memory (distant objects are fuzzy outlines, midrange is clear outlines with few details, close-up has full graphics), but that just replaces the pop-in problem with “sudden focus”, as a blob instantly resolves into a tree when you cross some threshold. I don’t think it’s something that will ever be really solved, because consumers will never accept the tradeoff of worse graphics on the things directly in front of them in exchange for seeing forever.

      1. Syal says:

        And the other option is tiny worlds, that feel tinier because you can always see everything.

    2. Zak McKracken says:

      To me, it’s a clear trade-off: texture and model resolution, number of effects and tiny realistic animations vs. draw distance. You can make sure that noone ever sees pop-ins but then your draw distance needs to increase to the point where an object popping in will be smaller than one pixel.

      Now, doubling draw distance will quadruple the amount of stuff you need to be drawing (assuming you’re on a plane — in space, you’d have 8 times as much), and to get the pop-in distance below the perceptability threshold, you’ll have to more than double it: If pop-ins now affect objects of 10 pixels in size, you’d have to increase the distance ten-fold. So — who wants to reduce the detail of all the things by a factor of 100, just to avoid that unsighlty effect?

      => There’s very clearly a trade-off, and quite correctly, as graphics are looking better and objects more detailed, pop-in effects become less and less noticeable, and affect objects that are smaller and smaller on-screen.

      I’m thinking that what’s needed is a procedural way to reduce detail dependent on distance, so that any polygon which goes below a critical size on-screen can be removed (or created when it goes above), fluidly.
      Ideally, that should not operate on size but on visibility. If a polygon defines the silhouette of some object, it needs to be fairly small to be removed but if it’s on a uniformly-coloured, slightly curved surface that’s facing you, it can go much earlier. Now, who wants to write the code to optimise this behaviour, in a way that won’t affect the display framerate, but without requiring artists to generate a hierarchy of LoD representations of all assets, dependent on viewing angle and current lighting situation? This does not seem impossible (if you do automatic generation of LoDs at compilation time, maybe, and are allowed to use loads of RAM?) but I completely understand why this is not a solved problem.

      It seems to me like pop-in is something which is absolutely required unless your scenes are small enough to show it all with no problem (but then, why don’t you add some more detail…?), but it’s also really hard to get exactly right, and to balance the benefits with the costs. So, there will always be someone who minds the drawbacks of one solution more than the drawbacks of another…

      1. Decius says:

        When making a system to decide whether to render something, it has to be much faster to decide not to render the thing than it would have been to just render it.

        Distance is fast. Clipping planes are faster. Both have serious problems.

        1. Zak McKracken says:

          That’s what I was trying to get to: You can either clip (which you might hide with fog or DoF) or just keep it all in. That means you’ll either have visible pop-ins (or lots of fog/DoF), or
          there’ll be a lot of geometry drawn unnecessarily, slowing everything down and forcing you to downgrade the detail. In most cases, you’ll have both because simply going by distance does not remove nearly all geometry that could safely be removed, and trying something more sophisticated will require some additional computing itself … so that’s why any game with large spaces will have things popping in in a not-so-perfect way sometimes.

      2. Echo Tango says:

        Don’t game engines and/or art software already automatically produce different levels of detail of an object automatically? We’ve got algorithms[1] to reduce the number of polygons, down-sample textures[2], etc. I have to assume that one of the final stages in releasing a game, is that once the game is basically complete, you run all the visual assets through some software, so that each game object gets some different levels of quality, for different draw-distances.

        [1] Here’s one for meshes, and here’s a video on image resizing.

        [2] I count bump-maps and lighting-maps here. It’s 2D data that gets applied to your 3D model.

        1. Zak McKracken says:

          Yes, that does exist, but it can’t be the perfect solution because the same problem repeats on a smaller scale: How do you decide to switch to a lower LoD? If it’s based on distance, then downgrading just a bit earlier can be a great help. How visible the transition will be depends a lot on things you don’t really want to keep track of: Is the silhouette of the thing visible against a bright background, or does it stand in front of similarly-coloured stuff? is there hard light on it or is it all uniformly lit, so that a change in the shape won’t be as visible?

          I mean, these days it’s mostly leaves and grass blades popping into existence, rather than buildings but it’s still visible, and will probably only gradually become less so. Every new bit of graphics accelerator power will be divided up between improving effects, increasing detail, increasing resolution and increasing drawing distance for the high levels of detail. And increasing resolution does definitely not help with making pop-ins less visible…

    3. Agammamon says:

      ITs why there are so very few open world games – the amount of work required to handle those level of details means that most games stick you in what is functionally a corridor. That way you can NEVER go down the wrong alley and see a mesh from a bad angle.

      D:OS is one of the few games I know of where they’re open that the scenes are only meant to be viewed from a limited angle and there will be problems if you rotate the camera – and then allow you the option to rotate the camera anyway.

    4. Sabrdance (MatthewH) says:

      It has long seemed to me that draw distance is something that should fall under “art problem” not “technical problem.” I do not object to games that have draw-distance issues. Skyrim, for example, had the grass pop-in, and that was a little disorienting at first, but after a while you just don’t notice it. There are other games -alas the titles are eluding me at the moment -where you had guns and scopes that could see further than the draw distance. This bugged me horribly. What is the point of creating a game with that kind of combat distance when you can’t actually shoot that far? That annoyed me far more than pop-in.

  10. wswordsmen says:

    Something tells me the next part will start out along the lines of: “This demo backfired horribly and now the mall project is even stupider.”

    1. Zak McKracken says:

      My guess: Roland really does not appreciate Shamus’ ways and prevents him from joining the coders’ ranks.

      Around 2000, I may have actually behaved in a very similar way. These days, I would make a point of emphasizing that deploying those features to customers means a lot more work than what the demo required.

      And then I would still underestimate hat effort by a factor of two.

      1. Sabrdance (MatthewH) says:

        We know from Part 5 that he will get into the Coder’s Ranks. Actually, the sinking feeling I have, based on the set-up, is that they’re going to solve the pop-in problem mentioned in Part 5 for the ugly, horrifically designed, far too large mall where the back half pops in as you approach, and you can see it because the mall is made of glass -that problem is going to be solved by filling the Soup Can with fog.

        Thus wrecking the point of having shops inside the mall, because now you can’t see them.

        1. Zak McKracken says:

          oops — I may have been too fast there :)

          Anyway, this has to backfire in some way.

  11. I don’t know how happy Mr. Roland is going to be, but your code seems to be working thus far.

    “The server is technically “blind”. It knows where the objects are and where the users are, and it broadcasts this information to everyone.”

    This might also be what led to strange graphical glitches in early 2000’s games. How does this play into the virtual mall though?

    1. Agammamon says:

      Its acerbates the problem of running the thing on client hardware. You’re not streaming content (where the heavy lifting can be done by your dedicated-and-purposely-designed hardware) you’re pushing even more stuff to compute on whatever grandma happens to have lying around to check her AOL.

      Very few systems of any type are ever designed with the expectation that they’ll still be in place in a decade and when they end up being so . . .

      WoW, for example, either got really lucky or they had some people with some damn good foresight to choose the artstyle they did. It was able to survive, fairly unchanged, for nearly a decade – meaning they didn’t have to worry about having to choose between looking dated or losing customers on low-end rigs they couldn’t afford to upgrade.

      1. So most systems are built to be replaced? I know WoW definitely had a stride of luck. A game that lasts for over a decade is absurd unless it’s intended to do so.

        1. Agammamon says:

          Most systems are built *with the intention* of being replaced in a few years. Very few of them actually are.

          Some aren’t, obviously, but most consumer level stuff is expected to be gone in 5 years or so. It kind of has to be because the costs of ensuring that your stuff scales well at the beginning with tech improvements is incredibly expensive and a lot of projects wouldn’t eve get off the ground if they were shooting for still being operational in 20, 30, 40 years.

          Look at a bank software – a ton of that stuff was written in the mainframe days in languages that aren’t used anymore. And its still being run and supported even if through a bunch of stop-gap measures, kludges, and jurry-rigged fixes.

          The Y2K problem was a result of this sort of thing – the upcoming UNIX problem is a result of this. People expected their work to be obsolete and replaced long before they hit the date rollover.

          1. I’ve always wondered how and why companies would rollover devices, in this case graphics, in order to continue profiting. The idea that hardware is still plataeuing is amazing. I can only imagine what future hardware might create graphically.

  12. Grimwear says:

    I find Shamus’ comment about needing to show them how great the program could look incredibly interesting especially concerning marketing today. I remember way back in 2010 with Sharp’s new tv and the commercial with George Takei that states how many colours their tv can show while our current ones can’t. It’s still common to this day with all the HD channels and clearer picture commercials being shoved in my face. They keep showing us all these device improvements on devices which cannot show them. It baffles me. It was the same thing with 3D tv. Unless I want to constantly go to Best Buy and look at all the new tvs I’ll never know what I’m missing out on and have no desire to go spend thousands every few years for improvements.

    1. Lanthanide says:

      I think you don’t understand the point of advertising.

      1. Grimwear says:

        Slightly dismissive comment but ok. I mean it’s great if companies want to go “look at this extra feature or button! You press it and a cup holder pops out. So amazing you need it.” And I can imagine myself holding a drink in my hand going huh I can use a cup holder for when I put it down. But now tell me that the image is “even more clear” or better yet “a colour your tv can’t even show you!” and it’s stupid. Ask someone to imagine a new colour. It can’t be done. We have no frame of reference or any idea how to even approach such a thing. Same thing with a clearer picture when what we’re currently using is perceived to be as clear as we can imagine.

        1. Falcon02 says:

          I certainly found it kinda odd and off putting, especially as a lot of those commercials did seem to “try” to show it. Often times by trying to provide an over exaggerated representation (such as the 3D TVs with the objects reaching out of the TV as seen from the side).

          But, I gave them some slack, as your ability to advertise features not render-able on current tech is limited.

          The advertiser’s goal in this case is to
          A.) Get you so hyped up in anticipation of this new feature you might buy it sight unseen.
          B.) Get you curious enough you’ll get up off that couch and head to Best Buy or what not to get yourself a real demo

          You can certainly do either of those without faithful representation in the Ad, and I do think the George Takei “Yellow pixel” Ads did play on this, with him specifically lamenting how you couldn’t see how awesome the image was on your old TV.

          Though, I’m still kinda curious how they would really demo the “Yellow” pixel thing in store. I’m sure they had pictures/video of some cool looking “Yellow” object that somehow was supposed to look better with dedicated “Yellow” Pixels. Quite possibly with the split screen demo many HD screens used where one side had a “simulated” image in the old tech.

  13. tmtvl says:

    The advantage is that you can host thousands of worlds on a single Linux box.

    …Fine, have a little Christmas bonus (^・ω・^ )

  14. Nick Powell says:

    Drama! How will this tale of intrigue and betrayal end?

    I’m really liking these. Actually I like most of your long multi-part posts. I’m fairly indiscriminate. But these ones are good too.

  15. Paul Spooner says:

    I just love the desk fan pointed, not at Shamus, but at his constantly overheating computer.

  16. Scampi says:

    For some reason, I just felt the desire to play a “survival horror shopping game”. What a pity the genre doesn’t exist and I’m not a programmer.
    I wonder why nobody ever gamified shopping in such a way that it would be fun just to reach the place where your product can be found.
    Imagine a foggy mall filled with zombies and creatures who crave mortal flesh while the poor customers just went in for their fill of cereal, some burgers and a new dust mop…can they successfully aquire the goods needed for their everyday household?
    I can tell you, I’d enjoy this approach to online shopping.^^

  17. Jonn says:

    I think you accidentaly a word in the first paragraph:

    I'm guessing those deals have been hard to come by, since it's been ages since I've been? given pictures of new stores and products to translate into 3D.

    Appreciate the article, looking forward to the next part.

  18. Oleyo says:

    Hey, the second picture is the view from my cubicle! And there is my 21″ KDS monitor that was about 2 feet deep. Took that home and used it until the late 2000s I believe.

    Can’t believe how young you look there Shamus!

    PS: Sorry to hit these old articles up, but this is a nice nostalgia trip, and I am alternating between cringing and smiling. Thanks, Shamus!

    1. Shamus says:

      Oh man. I was hoping you’d stop by. I thought about you a lot while writing this. It’s easy to laugh about this now that it’s 16 years(!!!!) in the past.

      For context for everyone else: The guy I’m replying to is Mike from the story above.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply

Your email address will not be published. Required fields are marked *