{"id":48466,"date":"2019-11-07T06:00:41","date_gmt":"2019-11-07T11:00:41","guid":{"rendered":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=48466"},"modified":"2019-11-11T11:18:50","modified_gmt":"2019-11-11T16:18:50","slug":"programming-vexations-part-9-the-problem-with-engines","status":"publish","type":"post","link":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=48466","title":{"rendered":"Programming Vexations Part 9: The Problem With Engines"},"content":{"rendered":"<p>In the previous entry I talked about the lack of game-specific types and features in C++, and how this leads to library proliferation, compatibility problems, and a massive duplication of effort. The idea was that a language designed for games ought to contain types that are common to all games. Several people argued in the comments that you shouldn&#8217;t add these sorts of things to the language itself, but rather provide them through the standard library.<\/p>\n<p>This leads into a side argument over whether or not we should consider the &#8220;standard library&#8221; to be part of the language, which is one of those questions like, &#8220;Is the bun part of the hot dog?&#8221; where everyone thinks the answer is obvious, and are then horrified to discover another group of people who think the opposite answer is the obviously correct one. So then they take turns hitting each other in the face with the dictionary.<\/p>\n<p>Welcome to the internet, I guess.<\/p>\n<p>But since we&#8217;re here, I might as well sort this out for people who work in sensible careers rather than becoming programmers.<\/p>\n<p><!--more--><\/p>\n<p>If you&#8217;re a programmer, then you might think of a language this way:<\/p>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_core.jpg' width=100% alt='' title=''\/><\/div><div class='mouseover-alt'><\/div><\/p>\n<p>At the center is the language, its syntax, functionality, and the basic types it supports. Then we have the standard library. It is, essentially, a bunch of code written in that language. The standard library is specifically designed to prevent the &#8220;everyone needs to make their own&#8221; problem I described last week.<\/p>\n<p>For example, the C language doesn&#8217;t have a built-in way to print strings of text to the screen. It <b>does<\/b> provide a way to output a single character at a time. You <b>could<\/b> write a bunch of code to take a string and shove it out to the console one character at a time, but since outputting strings to the console was so common in the early days, this idea was implemented in the standard library. You can see it in the classic Hello World example code:<\/p>\n<pre lang=\"c\" line=\"1\">#include \r\n\r\nint main()\r\n{\r\n   printf(\"Hello, World!\");\r\n   return 0;\r\n}\r\n<\/pre>\n<p>That first line tells the compiler to bring in &#8220;stdio.h&#8221;. That&#8217;s short for &#8220;Standard Input-Output&#8221;<span class='snote' title='1'>Fun fact: For my first year or so in C, I thought it was called &#8220;studio.h&#8221;. I was slightly confused when I finally typed it in manually rather than copy-pasting it from my previous project, and discovered that it was in fact NOT &#8220;studio&#8221;. If I had a time machine, I&#8217;d use it to send 1990 Shamus a proper book on the language so he didn&#8217;t have to muddle through so much of the language in the dark.<\/span>. We then use that on line 5 when we call &#8220;printf&#8221; to output a string.<\/p>\n<p>I should point out that for a developer, the line between &#8220;core language&#8221; and &#8220;standard library&#8221; is more academic than practical. The language ships with the standard library, basically everyone uses it, and so rank and file programmers just think of it as &#8220;part of the language&#8221;.<\/p>\n<p>From here the line gets blurry. Out beyond the standard library we have &#8220;common libraries&#8221;. In the C++ world this would be something like the <a href=\"https:\/\/www.boost.org\/\">Boost library<\/a>. This stuff is a little more sophisticated and not quite as universal as the standard library, but it still covers a lot of common things, with the aim of giving the language ecosystem one good implementation of a feature rather than having a thousand half-baked incompatible ones competing for mindshare as with my VectorBox vs. DasVektor example from last time.<\/p>\n<p>As you move outward into the realm of &#8220;everything else&#8221;, libraries get more specific to particular domains, and less common in their usage.<\/p>\n<p>I should stress that all of this is how a common programmer would view things. From the standpoint of someone developing and maintaining the language, the world probably looks more like this:<\/p>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_core2.jpg?' width=100% alt='' title=''\/><\/div><div class='mouseover-alt'><\/div><\/p>\n<p>They don&#8217;t care what libraries are common and they don&#8217;t want their beautiful language getting blamed for the shortcomings of all those terrible libraries floating around in the wild.<\/p>\n<p>All of this is to agree with people in the comments who said that Vectors, matrices, and other game-specific types ought to go in the standard library and not in the core language. I agree, I just didn&#8217;t think it was worth making that distinction. And now that recklessness on my part has led to this over-explanation.<\/p>\n<p>I ended the last entry with a promise that we&#8217;d talk about the drawbacks of using a game engine to turn C++ into a language for game development. So let&#8217;s do that now&#8230;<\/p>\n<h3>Start () Your Engines<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/stock_engines.jpg' width=100% alt='' title=''\/><\/div><div class='mouseover-alt'><\/div><\/p>\n<p>I should note that the term &#8220;engine&#8221; doesn&#8217;t have a formal definition. There&#8217;s no structural difference between the external libraries we talked about last week and a proper game engine. If a library is really large and is focused on a particular problem, then at some point people will start calling it an engine rather than a library.<\/p>\n<p>Last week I proposed starting with raw C++ and then trying to construct a game using libraries. One for window management, another for networking, then sound, image manipulation, rendering, physics, and so on. Essentially we&#8217;re trying to build a game engine piecemeal. This is a huge pain in the ass because all those disparate libraries were made in isolation and weren&#8217;t designed to inter-operate. They&#8217;ll have redundant code and lots of annoying little incompatibilities.<\/p>\n<p>Finding good libraries is hard, integrating them with your project is harder, and getting them to work together is a long and thankless task. <a href=\"?p=23380\">This is how I operated on my projects over the years<\/a>. I got away with it because my projects were always small in scope and not intended for commercial release. Even at that limited scale, getting all the parts working together was often a headache. You end up with this frankenstein project that feels like it&#8217;s held together with duct tape and positive thinking.<\/p>\n<p>Rather than building an engine out of spare parts like an episode of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Scrapheap_Challenge#Junkyard_Wars\">Scrapheap Challenge<\/a>, why don&#8217;t we just use a game engine?<\/p>\n<h3>Yeah, Why DON&#8217;T We Just Use a Game Engine?<\/h3>\n<p><table class='nomargin' cellspacing='0' width='100%' cellpadding='0' align='center' border='0'><tr><td><iframe loading=\"lazy\" width=\"1024\" height=\"576\" src=\"https:\/\/www.youtube.com\/embed\/SOtxjOLst2k\" frameborder=\"0\" allowfullscreen class=\"embed\"><\/iframe><br\/><small><a href='http:\/\/www.youtube.com\/watch?v=SOtxjOLst2k'>Link (YouTube)<\/a><\/small><\/td><\/tr><\/table><\/p>\n<p>Jon Blow isn&#8217;t a fan of using commercial game engines. You can hear him discuss the problem in <a href=\"https:\/\/www.youtube.com\/watch?v=SOtxjOLst2k\">this talk<\/a>. As someone who does a lot of work with procedural content, I have a lot of the same concerns. I often find myself fighting against traditional game engines because they&#8217;re usually built around a few basic assumptions. Those assumptions will funnel developers into certain genres. That&#8217;s great if you&#8217;re using an existing design for your game, but it can be restrictive if you&#8217;re doing something experimental or unconventional.<\/p>\n<p>I feel like we&#8217;re due for another Terrible Car Analogy\u2122: When it comes to mechanical engines, there&#8217;s no such thing as an &#8220;all-purpose engine&#8221;. The engine to drive a lawnmower is fundamentally different from the engine to move a car, which is fundamentally different from an engine for an airplane, which is different from an engine you&#8217;d use to generate electricity. You can&#8217;t just buy a &#8220;generic engine&#8221; and then adapt it to one of the tasks above. At least, not if you expect it to do a good job.<\/p>\n<p>The same is true for game engines. Good luck trying to make <i>Minecraft<\/i> or <i>No Man&#8217;s Sky<\/i> with the Unreal Engine. Good luck trying to make <i>DOOM 2016<\/i> or <i>The Last of Us<\/i> with Unity. I&#8217;m sure it&#8217;s <b>possible<\/b>, but it&#8217;s not the best tool for the job and you&#8217;ll end up fighting the engine the whole way.<\/p>\n<p>Maybe <a href=\"https:\/\/en.wikipedia.org\/wiki\/Frostbite_%28game_engine%29\">one engine<\/a> is good for making shooters. The entire engine is built around the idea of loading a premade level by a designer. You can&#8217;t even gain access to any rendering features until you load a level. That&#8217;s great if you&#8217;re making <i>Shoot Guy V: The Shootening<\/i>, but it&#8217;s a dealbreaker if you&#8217;re trying to make a dungeon crawler with procedurally generated maps. Another engine sort of allows for procedurally generated spaces, but the engine assumes you&#8217;re doing photorealism and the entire shader path is designed around that idea. There&#8217;s no way to get it to do line-art style graphics without a lot of really ugly hacks, when it would be easy to set up this sort of thing outside of the engine.<\/p>\n<p><a href='https:\/\/www.youtube.com\/watch?v=wHjIoyEeCdo'><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_unreal4.jpg' width=100% alt='Shot from Unreal Engine 4. (2015) It&apos;s amazing how close we can get to photorealism when nothing needs to be animated and there aren&apos;t any mammals in the shot.' title='Shot from Unreal Engine 4. (2015) It&apos;s amazing how close we can get to photorealism when nothing needs to be animated and there aren&apos;t any mammals in the shot.'\/><\/div><\/a><div class='mouseover-alt'>Shot from Unreal Engine 4. (2015) It&apos;s amazing how close we can get to photorealism when nothing needs to be animated and there aren&apos;t any mammals in the shot.<\/div><\/p>\n<p>This is what life is like in C++. You can use an existing engine with all the big problems solved for you, but it also shoves you into a cookie-cutter design paradigm. Or you go all the way down to the metal and implement your own solution for physics, audio, particles, AI, animation, rendering, input, pathing, collision, culling, camera, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Line_of_sight_(gaming)\">LoS checking<\/a>, lighting, particles, HUD, and networking. If you try to import libraries for all of those things then you&#8217;ll end up in a nightmare because the language itself is lacking Utility Belt features that all of these libraries will need.<\/p>\n<p>I&#8217;m a medium-sized fan of Unity<span class='snote' title='2'>Notwithstanding the <a href=\"?p=42495\">appalling documentation<\/a>.<\/span>, but I maintain there are downsides to using engines that we should be willing to think about.<\/p>\n<p>While engines are cheaper than they&#8217;ve ever been, they can still be pretty expensive. The versioning system that engines use make it difficult to revisit a project decades later and get the game running on the new generation of machines. You probably won&#8217;t be able to take your source code that uses Unity 2.0 and build it using whatever version of Unity exists in 2035<span class='snote' title='3'>And today&#8217;s version might not compile for a dozen different reasons.<\/span>. A lot can happen in 20 years. What if Unity gets bought out or goes under? What if the docs vanish? What if legacy versions are removed?<\/p>\n<h3>The Fragility of Legacy Code<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_matrix.jpg' width=100% alt='Why would anyone bother with an old movie when you can just watch something newer with better special effects?' title='Why would anyone bother with an old movie when you can just watch something newer with better special effects?'\/><\/div><div class='mouseover-alt'>Why would anyone bother with an old movie when you can just watch something newer with better special effects?<\/div><\/p>\n<p>Some people roll their eyes at these concerns, as if playing 20 year old games is an outlandish idea. I think this hobby suffers from an unfortunate lack of respect for its own history. Movies and television had the same problem. Nobody realized how much those early creations would matter and they weren&#8217;t always careful about preserving things. As a result, a lot of really historically important work is just <b>gone<\/b>. TV and film eventually learned their lesson, but video games aren&#8217;t quite there yet.<\/p>\n<p><i>The Matrix<\/i> turned 20 this year. Imagine if <i>The Matrix<\/i> was somehow &#8220;too old to watch&#8221; on modern hardware!<\/p>\n<p><b>Maybe<\/b> your particular version of Unity will still be available in 20 years and <b>maybe<\/b> your license will allow you to do a re-release and <b>maybe<\/b> it will build properly using the IDE of 2039, but that&#8217;s a lot of uncertainty for something as valuable as a commercial game.<\/p>\n<p>Like Blow, I have this sense that the industry has spent the last 10-15 years building a generation of games that are going to be hard to preserve. You can&#8217;t just look back at the 90s and say, &#8220;GoG got <i>Duke Nukem 3D<\/i> running on modern machines, so I&#8217;m sure we&#8217;ll be able to get <i>Doom 2016<\/i> running on the machines of 2036&#8243;. Our tool chains keep getting more complicated, and there&#8217;s a real possibility that our projects will end up tied to a particular setup of hardware, operating system, and drivers that won&#8217;t exist in 20 years. Just because we can emulate a Nintendo 64 doesn&#8217;t mean we&#8217;ll be able to emulate a Windows 10 machine or a PlayStation 4. The newer machines are astronomically more complex, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Moore%27s_law\">Gordon Moore<\/a> isn&#8217;t as helpful as he used to be.<\/p>\n<p>You can build a Ford Model T in your garage using replica parts, but you can&#8217;t build a replica 2005 Ford Fusion<span class='snote' title='4'>Boom! Two Terrible Car Analogies\u2122 in one entry!<\/span>. Machines are getting more complicated, and at some point that complication becomes too much of an obstacle to hobbyist projects.<\/p>\n<p>This isn&#8217;t to say that using a game engine is a bad thing or that you shouldn&#8217;t do it. Using a game engine can save you a ton of work. A lot of indie projects would simply not exist without the likes of Unreal and Unity doing the heavy lifting. I&#8217;m not suggesting that games would be better if we went back to the bad old days of everyone making their own engine completely from scratch. I&#8217;m just saying that using a commercial game engine introduces new points of potential failure that wouldn&#8217;t otherwise exist.<\/p>\n<p>I think it&#8217;s worth experimenting to see if we can find a compromise between the bare-bones nature of C++ and the monolithic infrastructure of a modern engine. Maybe there&#8217;s a sweet spot between these two. If the language can offer you the first half of the engine (data types, standardized ways of talking to the hardware) and allow you to make the other half of the engine (managing data and sending it to the hardware) then maybe that will give us most of the time-saving power of an engine while also offering the relative safety and portability of a homebrew system.<\/p>\n<p>I honestly don&#8217;t know. All I can say is that a hybrid system like this would suit my workflow, and I&#8217;m probably not the only one.<\/p>\n<h3>A Language Specifically for Games<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_stardew_language.jpg' width=100% alt='But probably not THIS language.' title='But probably not THIS language.'\/><\/div><div class='mouseover-alt'>But probably not THIS language.<\/div><\/p>\n<p>A language designed for games ought to provide elements common to all games, but leave everything else to the designer. The situation we&#8217;re in with C++ is that game engines are too monolithic \/ restrictive, while the language itself is too barebones. <b>What we need is a tool designed for the job at hand<\/b>, and one that will give us expressive<span class='snote' title='5'>Being able to express very complex ideas in a small section of readable code.<\/span> code without sacrificing performance. That&#8217;s hard to achieve in a generic language, but becomes much more doable if the language designer knows you&#8217;re going to be solving videogame problems and not (say) writing operating systems or device drivers.<\/p>\n<p>Last week I complained about the problems of implementing Utility Belt features like vectors, matrices, bounding boxes, and other game-specific data types. Keep in mind that the Utility Belt isn&#8217;t the only thing you need. That was just the most obvious example out of many. It&#8217;s not clear to me where you should draw the line between features provided by the language, and things the developer should be expected to do for themselves. Creating the window used for gameplay? Talking to the graphics hardware? Loading image files? Audio? You could make a good case for or against providing any of these things as standard libraries to accompany a language.<\/p>\n<p>As an ideal to shoot for, I suppose I&#8217;d suggest including as many things as possible without trapping the developer in any particular design. I realize that&#8217;s pretty vague, but I think it works pretty well as a guiding principle. In any case, I feel comfortable saying that we could easily be doing better than we are right now with C++. If we jump to a new language that also doesn&#8217;t include Utility Belt stuff, then we&#8217;ll be dragging a lot of our C++ restrictions and headaches with us.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the previous entry I talked about the lack of game-specific types and features in C++, and how this leads to library proliferation, compatibility problems, and a massive duplication of effort. The idea was that a language designed for games ought to contain types that are common to all games. Several people argued in the [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[66],"tags":[],"class_list":["post-48466","post","type-post","status-publish","format-standard","hentry","category-programming"],"_links":{"self":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/48466","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=48466"}],"version-history":[{"count":16,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/48466\/revisions"}],"predecessor-version":[{"id":48490,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/48466\/revisions\/48490"}],"wp:attachment":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=48466"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=48466"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=48466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}