{"id":48402,"date":"2019-10-24T08:27:12","date_gmt":"2019-10-24T12:27:12","guid":{"rendered":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=48402"},"modified":"2019-10-24T08:27:12","modified_gmt":"2019-10-24T12:27:12","slug":"programming-vexations-part-8-the-problem-with-libraries","status":"publish","type":"post","link":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=48402","title":{"rendered":"Programming Vexations Part 8: The Problem With Libraries"},"content":{"rendered":"<p>Whenever people get talking about the things you need in a game development language, we usually end up with a handful of features like manual memory management, pointers, low-level code, the lack of safety features like bounds-checking, and so on. One of the big items on this list is the ability to use external libraries.<\/p>\n<p>Being able to use libraries is indeed a good thing. I certainly wouldn&#8217;t want to attempt to use C++ without access to a good set of libraries! At the same time, I think we often overlook the fact that importing and integrating external libraries is enormously expensive. A game developer will complain that they don&#8217;t want to have to roll their own sound library and someone will dismiss them saying, &#8220;Bah. There are lots of free sound libraries out there. Just pick one and start coding!&#8221;<\/p>\n<p>While it&#8217;s true that there are free libraries all over the place, people are really bad at factoring in the cost of using them.<br \/>\n<!--more--><\/p>\n<h3>The Utility Belt<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_utility_belt.jpg?' width=100% alt='This image will make slightly more sense later in the article. For now, just go with it.' title='This image will make slightly more sense later in the article. For now, just go with it.'\/><\/div><div class='mouseover-alt'>This image will make slightly more sense later in the article. For now, just go with it.<\/div><\/p>\n<p>In gamedev, a vector is a variable that stores 3 values: X, Y, and Z. These are used constantly. Every 3D object in the scene is made up of triangles, every triangle is made of 3 points, and every point has a vector to keep track of its position in 3D space. The player&#8217;s velocity as they double-jump over lava is stored in a vector. The imaginary line from (the muzzle of the protagonist&#8217;s gun) to (the point where the player is aiming) is stored in a vector. Things like mouse and joystick input are often stored in a vector<span class='snote' title='1'>Sometimes the engine you&#8217;re working with has 2D vectors for this, but if not the programmer might store the input values in X and Y and leave Z unused.<\/span>. The position of the camera is a vector and the direction the camera is pointing is another vector. Every light in the scene uses a vector to store its position, and directional lights<span class='snote' title='2'>Like a flashlight or a searchlight.<\/span> will use another vector to keep track of which way the light is pointed.<\/p>\n<p>Nearly every system in a game either contains or manipulates vectors. Sounds come from a location, so they need a vector. Physics involves spatial calculations, so physics objects definitely need vectors. Loading geometry like characters and level data will need vectors. So will particles. Animations. Rendering code. Collision detection. Camera placement. Lighting.<\/p>\n<p>Even if you&#8217;re just making a 2D side-scroller, you still need 3D vectors. You need 2 dimensions for the left-right and up-down movement within the game, but you&#8217;ll use the Z axis for depth, which you&#8217;ll need if you want a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Parallax\">parallax<\/a>ing<span class='snote' title='3'>The background scrolls slower than the foreground to give the scene a feeling of depth.<\/span> background. Even if your game is played on a pure 2D plane with no attempt to create the illusion of depth, you&#8217;ll still need some vectors for manipulating the camera because the graphics hardware communicates in vectors and you need that thing to draw the scene.<\/p>\n<p>The thing about vectors is that <b>they don&#8217;t come with the language<\/b>. Without a vector data type to handle these values, you&#8217;d be forced to juggle the individual X, Y, and Z components:<\/p>\n<pre lang=\"cpp\">player_position_x += offset_x;\r\nplayer_position_y += offset_y;\r\nplayer_position_z += offset_z;\r\ndistance_to_origin = sqrt (player_position_x * player_position_x + player_position_y * player_position_y + player_position_z * player_position_z); \r\n<\/pre>\n<p>That&#8217;s a lot more verbose and error-prone than this:<\/p>\n<pre lang=\"cpp\">player_position += offset;\r\ndistance_to_origin = player_position.Length ();\r\n<\/pre>\n<p>That&#8217;s a trivial example. The problem gets completely out of control once you start doing real work. I don&#8217;t want to say that nobody has ever made a game without vector data types, but I feel confident in saying that doing so would be extremely cumbersome. Everyone making games has to write or import their own vectors. You can&#8217;t begin working on a project until you have these.<\/p>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_axis.jpg' width=100% alt='In your typical 3D scene, you usually need several vectors: One for position, one for rotation \/ orientation, and sometimes one for scaling.' title='In your typical 3D scene, you usually need several vectors: One for position, one for rotation \/ orientation, and sometimes one for scaling.'\/><\/div><div class='mouseover-alt'>In your typical 3D scene, you usually need several vectors: One for position, one for rotation \/ orientation, and sometimes one for scaling.<\/div><\/p>\n<p>If you&#8217;re going to have vectors, then you&#8217;ll also need to be able to manipulate them. You need to be able to add them, subtract them, measure their length, compare them to each other, scale them, copy them, reflect them off each other, and a host of other things.<\/p>\n<p>In addition to vectors, a proper game development environment ought to have a bunch of other stuff. Here are several, in order of descending need from &#8220;you can&#8217;t do anything useful without this&#8221; to &#8220;it would be nice to have this&#8221;.<\/p>\n<ol>\n<li><b>3D Vectors<\/b>. Every game above trivial complexity will need these.<\/li>\n<li><b>Matrices<\/b>. A Matrix is a grid of numbers used to manipulate vectors. Let&#8217;s say you&#8217;ve got the zombie model in Minecraft. By default it will be positioned at 0N, 0W, at the bottom of the world. But perhaps for gameplay purposes it needs to be at 150N, 2503W, 64 meters above the bottom of the world. That location would of course be stored in a vector. Let&#8217;s say also that the zombie model needs to be turned sideways<span class='snote' title='4'>Maybe the player just killed it.<\/span>. The programmer would then set up a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Transformation_matrix\">transformation matrix<\/a>. Multiply all the vertices in the zombie by this matrix, and it will be moved to the desired location.So we need a bunch of code to create a matrix, move it, rotate it, compare it to others, and use it to <a href=\"https:\/\/www.xkcd.com\/184\/\">transform<\/a> vectors.<\/li>\n<li><b>Color values<\/b>. When you talk to the graphics card, it often wants to know what color things are. Sometimes you need to know the color values of individual pixels or polygons. It&#8217;s really nice to have a variable with red, green, blue, and alpha<span class='snote' title='5'>Used for transparency.<\/span> color values that can be easily stored, compared, and manipulated. Actually, we should probably have two different types of color values. One which stores color channels as real numbers between 0.0 and 1.0, and another that stores each channel as a whole number in the range of 0 to 255. The former is useful for manipulating color values mathematically, while the latter is how you need to store the color values to be used in texture maps.<\/li>\n<li><b>Bounding Boxes<\/b>. These cubic volumes are handy for figuring out what needs to be drawn or if you should bother doing expensive collision checking on two objects. A bounding sphere isn&#8217;t quite as common, but still nice to have.<\/li>\n<li><b>Quaternions<\/b>. A <a href=\"https:\/\/en.wikipedia.org\/wiki\/Quaternion\">4-dimensional variable<\/a> for performing various feats of dark sorcery. Don&#8217;t ask.<\/li>\n<li><b>2D Vectors<\/b>. Sometimes you just need an X and Y value. Good for mouse movement, joysticks, map positions, positioning HUD elements on the screen, and lots of other random stuff.<\/li>\n<li><b>Textures<\/b>. Nearly all games deal with texture maps. Texture maps are made up of those color values I mentioned above. It can be really useful to have a specialized container for holding texture data.<\/li>\n<li><b>Triangle groups<\/b>. Triangles are defined by three numbers, each number referring to a specific vertex, allowing your graphics card to draw the triangle by playing connect-the-dots. It&#8217;s often convenient to have some sort of container for storing these lists of numbers.<\/li>\n<li><b>Integer Vectors<\/b>: This is like a vector above, except it can only store whole numbers. Really useful for passing over multidimensional arrays and doing stuff with grids. And game designers LOVE grids.<\/li>\n<\/ol>\n<p>Basically, every game needs vectors and their attendant types. For convenience, let&#8217;s refer to this collection of types as the gamedev&#8217;s Utility Belt. Batman carries around <a href=\"https:\/\/www.youtube.com\/watch?v=k_B_n-Rbros\">shark repellant<\/a>, and game developers carry around vectors and matrices<span class='snote' title='6'>See? I told you that Batman image would make sense later.<\/span>. If <b>every single game<\/b> needs them, then why not make them part of the language?<\/p>\n<p>At this point someone will suggest we just take an existing language<span class='snote' title='7'>By some strange coincidence, it always ends up being their favorite.<\/span> and add the Utility Belt to it through a library. That&#8217;s how we&#8217;ve been doing things for decades.<\/p>\n<h3>Vexation #3: Shopping for Libraries is Not Programming<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_library.jpg' width=100% alt='Have you ever had to visit multiple libraries to see if they have a special book you&apos;re looking for? No? Oh right. We have the internet now. Well, it used be be a problem, and it was a lot like looking for programming libraries. You need to invest time just to see if the library has what you want.' title='Have you ever had to visit multiple libraries to see if they have a special book you&apos;re looking for? No? Oh right. We have the internet now. Well, it used be be a problem, and it was a lot like looking for programming libraries. You need to invest time just to see if the library has what you want.'\/><\/div><div class='mouseover-alt'>Have you ever had to visit multiple libraries to see if they have a special book you&apos;re looking for? No? Oh right. We have the internet now. Well, it used be be a problem, and it was a lot like looking for programming libraries. You need to invest time just to see if the library has what you want.<\/div><\/p>\n<p>Libraries are fantastic. One programmer comes up with a solution to a problem. Maybe they make a physics engine, a particle engine, an audio engine, or whatever. They offer this code as a library that you can drop into your project so you don&#8217;t need to implement all that stuff yourself.<\/p>\n<p>That&#8217;s the platonic ideal, anyway. What actually ends up happening is that someone releases a library, but their code uses some dated language features. The documentation is poor. Importing a library is rarely a drag and drop operation, and sometimes you might need to burn an entire day of productivity just getting it to compile with the rest of your project and figuring out how it works. And then once you finally get it working, you discover it doesn&#8217;t really solve your problem. Maybe it&#8217;s too slow, or missing a key feature. It probably has tons of code dedicated to doing stuff you don&#8217;t want or need. Is that unwanted code being executed? Should you modify this library to suit your project, or should you go back to Google and look for another? It&#8217;s going to take you several more hours of work to answer that question.<\/p>\n<p>I&#8217;m not saying libraries are useless or that we shouldn&#8217;t use them. I&#8217;m just saying that <a href=\"?p=9557\">library integration is difficult and time-consuming<\/a>. The more libraries you have, the greater the odds that you&#8217;ll run into conflicts between them.<\/p>\n<p>I think the cost of library integration is vastly underestimated in the field. Everyone has lost a day and a half to a library that turned out to be useless, and the typical response is to assume you just got unlucky. <i>Well, that library was a bust, but things would have turned out differently if I&#8217;d tried a different one!<\/i> And maybe that&#8217;s true, but part of the cost of importing a library is <b>finding<\/b> a library. The more libraries you need and the more you need those different libraries to exchange information, the more difficult it gets.<\/p>\n<h3>Nothing is Turnkey<\/h3>\n<p><div class='imagefull'><img src='https:\/\/www.shamusyoung.com\/twentysidedtale\/images\/vex_random1.jpg' width=100% alt='I couldn&apos;t find a stock photo to represent the abstract problems of library integration, so here&apos;s a screenshot from Minecraft.' title='I couldn&apos;t find a stock photo to represent the abstract problems of library integration, so here&apos;s a screenshot from Minecraft.'\/><\/div><div class='mouseover-alt'>I couldn&apos;t find a stock photo to represent the abstract problems of library integration, so here&apos;s a screenshot from Minecraft.<\/div><\/p>\n<p>So if all games need the Utility Belt, then do we really want to force every single developer to shop around for libraries to get this feature?<\/p>\n<p><b>But Shamus! This Utility Belt isn&#8217;t THAT big a deal. You can write your own version in a couple of days and importing one shouldn&#8217;t cause conflicts. You&#8217;re being a big baby!<\/b><\/p>\n<p>The problem is that if the Utility Belt isn&#8217;t part of the language, then everyone will be faced with this same problem, including people <b>making<\/b> libraries.<\/p>\n<p>Let&#8217;s say that we&#8217;re very lucky and we find ourselves in a universe where there are only two competing Utility Belt libraries. DasVektor is a robust and full-featured library and it still gets regular updates. On the other hand, the DasVektor documentation pages (and even some of the classes) are in German. Even if reading German isn&#8217;t a problem, the docs are a bit of a patchwork and bits of them are way out of date. So DasVektor doesn&#8217;t get a lot of use. There&#8217;s also VectorBox, which is missing some features and hasn&#8217;t been updated in five years. On the other hand, the documentation is really good and it&#8217;s all in English.<\/p>\n<p>At this point you&#8217;re probably nodding your head. &#8220;Yes Shamus. This is not news. Everyone knows about the problems of competing standards. This is going to be a problem no matter what language you use.&#8221;<\/p>\n<p>Except, that problem feeds into the difficulty with libraries. Let&#8217;s say I make my project and I choose VectorBox as my Utility Belt. I import that. No problem. Great.<\/p>\n<p>But now I need a library for doing positional audio. That requires the use of vectors. Since the language itself doesn&#8217;t provide them, the person who wrote this sound library had to make their own. So now my program has two different kinds of vectors. They work in similar ways and do roughly the same things, but they&#8217;re just different enough to be incompatible.<\/p>\n<p>Then we have this physics library. It uses DasVektor for its vectors. So now I&#8217;ve got both DasVektor and VectorBox in my project. The standards war isn&#8217;t taking place on forum threads out in the wild, it&#8217;s taking place <b>inside my project<\/b>.<\/p>\n<p>Next I import a particle library. Particles need vectors, which means this library will also come prepackaged with yet another vector solution. As it turns out, this one is based on an older version of DasVektor that&#8217;s not really compatible with the newer version. Do I search for another library, or attempt to update this one?<\/p>\n<p><b>Shamus, you&#8217;re thinking about this all wrong. What you&#8217;re looking for isn&#8217;t a language, it&#8217;s a game engine. Go download Unreal Engine or Unity and it already has all the Utility Belt crap you&#8217;re looking for, along with all the other stuff for audio, particles, physics, AI, and so on. You don&#8217;t implement game engines at the language level!<\/b><\/p>\n<p>I&#8217;d argue that the language ought to support types that are common to all problems in its intended domain. If someone made a language specifically for generating web pages<span class='snote' title='8'>PHP is one such language.<\/span> that doesn&#8217;t include a variable for dealing with text strings, then the language isn&#8217;t doing its job. Sure, it&#8217;s not that hard<span class='snote' title='9'>Assuming you&#8217;re just using English ASCII text. If you decide to do things the Right Way\u2122 and have your strings to support Unicode, then you are basically doomed. That project will be monumentally bigger than whatever silly web page problem you&#8217;re supposedly working on.<\/span>.\u00a0 to make your own data type for manipulating strings of text. But if it&#8217;s not part of the language then <b>every single programmer<\/b> will need to do that. Aside from the problems of integration I discussed above, this results in a tremendous duplication of effort.<\/p>\n<p>There is no such thing as a game that doesn&#8217;t need vectors. It seems really unreasonable to require an entire game engine just to get access to basic data types, but next week I&#8217;m going to explore this idea and talk about the problems of using game engines to turn C++ into a language for games.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Whenever people get talking about the things you need in a game development language, we usually end up with a handful of features like manual memory management, pointers, low-level code, the lack of safety features like bounds-checking, and so on. One of the big items on this list is the ability to use external libraries. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[66],"tags":[],"class_list":["post-48402","post","type-post","status-publish","format-standard","hentry","category-programming"],"_links":{"self":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/48402","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=48402"}],"version-history":[{"count":9,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/48402\/revisions"}],"predecessor-version":[{"id":48411,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/48402\/revisions\/48411"}],"wp:attachment":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=48402"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=48402"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=48402"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}