Whenever people get talking about the things you need in a game development language, we usually end up with a handful of features like manual memory management, pointers, low-level code, the lack of safety features like bounds-checking, and so on. One of the big items on this list is the ability to use external libraries.
Being able to use libraries is indeed a good thing. I certainly wouldn’t want to attempt to use C++ without access to a good set of libraries! At the same time, I think we often overlook the fact that importing and integrating external libraries is enormously expensive. A game developer will complain that they don’t want to have to roll their own sound library and someone will dismiss them saying, “Bah. There are lots of free sound libraries out there. Just pick one and start coding!”
While it’s true that there are free libraries all over the place, people are really bad at factoring in the cost of using them.
The Utility Belt
In gamedev, a vector is a variable that stores 3 values: X, Y, and Z. These are used constantly. Every 3D object in the scene is made up of triangles, every triangle is made of 3 points, and every point has a vector to keep track of its position in 3D space. The player’s velocity as they double-jump over lava is stored in a vector. The imaginary line from (the muzzle of the protagonist’s gun) to (the point where the player is aiming) is stored in a vector. Things like mouse and joystick input are often stored in a vectorSometimes the engine you’re working with has 2D vectors for this, but if not the programmer might store the input values in X and Y and leave Z unused.. The position of the camera is a vector and the direction the camera is pointing is another vector. Every light in the scene uses a vector to store its position, and directional lightsLike a flashlight or a searchlight. will use another vector to keep track of which way the light is pointed.
Nearly every system in a game either contains or manipulates vectors. Sounds come from a location, so they need a vector. Physics involves spatial calculations, so physics objects definitely need vectors. Loading geometry like characters and level data will need vectors. So will particles. Animations. Rendering code. Collision detection. Camera placement. Lighting.
Even if you’re just making a 2D side-scroller, you still need 3D vectors. You need 2 dimensions for the left-right and up-down movement within the game, but you’ll use the Z axis for depth, which you’ll need if you want a parallaxingThe background scrolls slower than the foreground to give the scene a feeling of depth. background. Even if your game is played on a pure 2D plane with no attempt to create the illusion of depth, you’ll still need some vectors for manipulating the camera because the graphics hardware communicates in vectors and you need that thing to draw the scene.
The thing about vectors is that they don’t come with the language. Without a vector data type to handle these values, you’d be forced to juggle the individual X, Y, and Z components:
player_position_x += offset_x; player_position_y += offset_y; player_position_z += offset_z; distance_to_origin = sqrt (player_position_x * player_position_x + player_position_y * player_position_y + player_position_z * player_position_z);
That’s a lot more verbose and error-prone than this:
player_position += offset; distance_to_origin = player_position.Length ();
That’s a trivial example. The problem gets completely out of control once you start doing real work. I don’t want to say that nobody has ever made a game without vector data types, but I feel confident in saying that doing so would be extremely cumbersome. Everyone making games has to write or import their own vectors. You can’t begin working on a project until you have these.
If you’re going to have vectors, then you’ll also need to be able to manipulate them. You need to be able to add them, subtract them, measure their length, compare them to each other, scale them, copy them, reflect them off each other, and a host of other things.
In addition to vectors, a proper game development environment ought to have a bunch of other stuff. Here are several, in order of descending need from “you can’t do anything useful without this” to “it would be nice to have this”.
- 3D Vectors. Every game above trivial complexity will need these.
- Matrices. A Matrix is a grid of numbers used to manipulate vectors. Let’s say you’ve got the zombie model in Minecraft. By default it will be positioned at 0N, 0W, at the bottom of the world. But perhaps for gameplay purposes it needs to be at 150N, 2503W, 64 meters above the bottom of the world. That location would of course be stored in a vector. Let’s say also that the zombie model needs to be turned sidewaysMaybe the player just killed it.. The programmer would then set up a transformation matrix. Multiply all the vertices in the zombie by this matrix, and it will be moved to the desired location.So we need a bunch of code to create a matrix, move it, rotate it, compare it to others, and use it to transform vectors.
- Color values. When you talk to the graphics card, it often wants to know what color things are. Sometimes you need to know the color values of individual pixels or polygons. It’s really nice to have a variable with red, green, blue, and alphaUsed for transparency. color values that can be easily stored, compared, and manipulated. Actually, we should probably have two different types of color values. One which stores color channels as real numbers between 0.0 and 1.0, and another that stores each channel as a whole number in the range of 0 to 255. The former is useful for manipulating color values mathematically, while the latter is how you need to store the color values to be used in texture maps.
- Bounding Boxes. These cubic volumes are handy for figuring out what needs to be drawn or if you should bother doing expensive collision checking on two objects. A bounding sphere isn’t quite as common, but still nice to have.
- Quaternions. A 4-dimensional variable for performing various feats of dark sorcery. Don’t ask.
- 2D Vectors. Sometimes you just need an X and Y value. Good for mouse movement, joysticks, map positions, positioning HUD elements on the screen, and lots of other random stuff.
- Textures. Nearly all games deal with texture maps. Texture maps are made up of those color values I mentioned above. It can be really useful to have a specialized container for holding texture data.
- Triangle groups. Triangles are defined by three numbers, each number referring to a specific vertex, allowing your graphics card to draw the triangle by playing connect-the-dots. It’s often convenient to have some sort of container for storing these lists of numbers.
- Integer Vectors: This is like a vector above, except it can only store whole numbers. Really useful for passing over multidimensional arrays and doing stuff with grids. And game designers LOVE grids.
Basically, every game needs vectors and their attendant types. For convenience, let’s refer to this collection of types as the gamedev’s Utility Belt. Batman carries around shark repellant, and game developers carry around vectors and matricesSee? I told you that Batman image would make sense later.. If every single game needs them, then why not make them part of the language?
At this point someone will suggest we just take an existing languageBy some strange coincidence, it always ends up being their favorite. and add the Utility Belt to it through a library. That’s how we’ve been doing things for decades.
Vexation #3: Shopping for Libraries is Not Programming
Libraries are fantastic. One programmer comes up with a solution to a problem. Maybe they make a physics engine, a particle engine, an audio engine, or whatever. They offer this code as a library that you can drop into your project so you don’t need to implement all that stuff yourself.
That’s the platonic ideal, anyway. What actually ends up happening is that someone releases a library, but their code uses some dated language features. The documentation is poor. Importing a library is rarely a drag and drop operation, and sometimes you might need to burn an entire day of productivity just getting it to compile with the rest of your project and figuring out how it works. And then once you finally get it working, you discover it doesn’t really solve your problem. Maybe it’s too slow, or missing a key feature. It probably has tons of code dedicated to doing stuff you don’t want or need. Is that unwanted code being executed? Should you modify this library to suit your project, or should you go back to Google and look for another? It’s going to take you several more hours of work to answer that question.
I’m not saying libraries are useless or that we shouldn’t use them. I’m just saying that library integration is difficult and time-consuming. The more libraries you have, the greater the odds that you’ll run into conflicts between them.
I think the cost of library integration is vastly underestimated in the field. Everyone has lost a day and a half to a library that turned out to be useless, and the typical response is to assume you just got unlucky. Well, that library was a bust, but things would have turned out differently if I’d tried a different one! And maybe that’s true, but part of the cost of importing a library is finding a library. The more libraries you need and the more you need those different libraries to exchange information, the more difficult it gets.
Nothing is Turnkey
So if all games need the Utility Belt, then do we really want to force every single developer to shop around for libraries to get this feature?
But Shamus! This Utility Belt isn’t THAT big a deal. You can write your own version in a couple of days and importing one shouldn’t cause conflicts. You’re being a big baby!
The problem is that if the Utility Belt isn’t part of the language, then everyone will be faced with this same problem, including people making libraries.
Let’s say that we’re very lucky and we find ourselves in a universe where there are only two competing Utility Belt libraries. DasVektor is a robust and full-featured library and it still gets regular updates. On the other hand, the DasVektor documentation pages (and even some of the classes) are in German. Even if reading German isn’t a problem, the docs are a bit of a patchwork and bits of them are way out of date. So DasVektor doesn’t get a lot of use. There’s also VectorBox, which is missing some features and hasn’t been updated in five years. On the other hand, the documentation is really good and it’s all in English.
At this point you’re probably nodding your head. “Yes Shamus. This is not news. Everyone knows about the problems of competing standards. This is going to be a problem no matter what language you use.”
Except, that problem feeds into the difficulty with libraries. Let’s say I make my project and I choose VectorBox as my Utility Belt. I import that. No problem. Great.
But now I need a library for doing positional audio. That requires the use of vectors. Since the language itself doesn’t provide them, the person who wrote this sound library had to make their own. So now my program has two different kinds of vectors. They work in similar ways and do roughly the same things, but they’re just different enough to be incompatible.
Then we have this physics library. It uses DasVektor for its vectors. So now I’ve got both DasVektor and VectorBox in my project. The standards war isn’t taking place on forum threads out in the wild, it’s taking place inside my project.
Next I import a particle library. Particles need vectors, which means this library will also come prepackaged with yet another vector solution. As it turns out, this one is based on an older version of DasVektor that’s not really compatible with the newer version. Do I search for another library, or attempt to update this one?
Shamus, you’re thinking about this all wrong. What you’re looking for isn’t a language, it’s a game engine. Go download Unreal Engine or Unity and it already has all the Utility Belt crap you’re looking for, along with all the other stuff for audio, particles, physics, AI, and so on. You don’t implement game engines at the language level!
I’d argue that the language ought to support types that are common to all problems in its intended domain. If someone made a language specifically for generating web pagesPHP is one such language. that doesn’t include a variable for dealing with text strings, then the language isn’t doing its job. Sure, it’s not that hardAssuming you’re just using English ASCII text. If you decide to do things the Right Way™ and have your strings to support Unicode, then you are basically doomed. That project will be monumentally bigger than whatever silly web page problem you’re supposedly working on.. to make your own data type for manipulating strings of text. But if it’s not part of the language then every single programmer will need to do that. Aside from the problems of integration I discussed above, this results in a tremendous duplication of effort.
There is no such thing as a game that doesn’t need vectors. It seems really unreasonable to require an entire game engine just to get access to basic data types, but next week I’m going to explore this idea and talk about the problems of using game engines to turn C++ into a language for games.
 Sometimes the engine you’re working with has 2D vectors for this, but if not the programmer might store the input values in X and Y and leave Z unused.
 Like a flashlight or a searchlight.
 The background scrolls slower than the foreground to give the scene a feeling of depth.
 Maybe the player just killed it.
 Used for transparency.
 See? I told you that Batman image would make sense later.
 By some strange coincidence, it always ends up being their favorite.
 PHP is one such language.
 Assuming you’re just using English ASCII text. If you decide to do things the Right Way™ and have your strings to support Unicode, then you are basically doomed. That project will be monumentally bigger than whatever silly web page problem you’re supposedly working on.
The Best of 2011
My picks for what was important, awesome, or worth talking about in 2011.
The Biggest Game Ever
How did this niche racing game make a gameworld so massive, and why is that a big deal?
Shamus Plays LOTRO
As someone who loves Tolkein lore and despises silly MMO quests, this game left me deeply conflicted.
Spec Ops: The Line
A videogame that judges its audience, criticizes its genre, and hates its premise. How did this thing get made?
Bad and Wrong Music Lessons
A music lesson for people who know nothing about music, from someone who barely knows anything about music.
77 thoughts on “Programming Vexations Part 8: The Problem With Libraries”
Shamus, I seem to remember your How I Learned series talking about not enjoying math in school or perhaps not seeing the point of it. Knowing what you know now about the utility of vectors and matrices, do you have any thoughts about how your math teachers could have better engaged you?
(Personally, I always loved math for its own sake, but I will nevertheless agree that math doesn’t start to get really fun until calculus. But the problem is, in order to be good at higher math, you have to have a strong foundation of numbers, which you don’t get without the repetitive drilling of arithmetic facts. It’s sort of like scales in music – you have to know them and know how the notes relate to each other before you can really progress. I have said before and will no doubt say again that math is a decades-long experience of delayed gratification.)
I, too, would be curious to hear Shamus’ response to this. The uninitiated might not expect a programming article to talk about linear algebra, group/ring theory, and trigonometry.
In other news, today I learned the quaternions thing I saw in abstract algebra has actual applications!
I was taught matrices in school. They seemed baffling and pointless. If only someone had said, “these can be used to rotate and scale sprites,” I would have paid a lot more attention.
Did you take calculus? As it turns out, calculating a derivative is really just matrix multiplication.
Sshh, next thing you’ll be telling everyone that solving simultaneous equations is really just matrix maths…
They turn up everywhere!
I disagree about the numbers bit. I was complete trash at arithmetic but through an accident ended up in the advanced part of a 4th grade math class where they were covering intro-level algebra. I never did build a strong arithmetic foundation (I would probably fail the test where you fill out a multiplication table under a timer) but I really enjoyed calculus and ended up with an engineering degree.
From my small hobby-level experience with games and 3D programming stuff, it seems like linear algebra is super important for them. Though maybe it’s not as much once you get a decent matrix library. From over a decade of writing embedded software, I can say that most math is pretty unnecessary for it.
I’m not fast at arithmetic either to this day. But you still have to know those facts, even if you’re not fast with them. I used to tutor kids in math, especially algebra, and watching them try to factor second order equations, simplify expressions, or perform other basic algebraic operations was painful. They all had to use calculators for everything and had to use a lot of brute force/trial and error. For example, to simplify the square root of 52, one kid had to use a calculator to divide 52 by 2, then use it again to divide 26 by 2, then they were unable to recognize that 13 was prime and sat there dividing 13 by every integer 1-12 looking for another factor. And that was one of the smarter kids I tutored.
I don’t even want to talk about the rainbow method. I haven’t seen a high schooler in years who is capable of factoring even something as simple as x^2 + 5x + 6 without the rainbow method.
To go back to the music analogy, you might be able to teach yourself Minuet in G Major through trial and error – I know, because I did – but without a good foundation in fingering technique and so on, it won’t be good. (Which I know, because it sounds terrible when I play it even though every individual note is correct.)
Why do we try to teach kids that they should factor numbers by hand? There’s no good algorithm for it; the only decent way to do it is to have a complete multiplication table and find the number you want to factor in the product column. I tend to think that memorizing times tables is not a terribly good use of education time, since nobody’s ever far from a calculator these days.
I’m an engineer and I’ve never had to find the prime factors of a number by hand. I’ve worked with RSA encryption, where the whole thing is based on the (presumed) fact that factoring large numbers is hard. I’ve written programs to crack small RSA keys, but never by hand. If the goal is to teach people that composite numbers are made up of prime numbers, you can do that without forcing them to spend time hand-factoring numbers.
“But the problem is, in order to be good at higher math, you have to have a strong foundation of numbers, which you don’t get without the repetitive drilling of arithmetic facts. ”
I question the need for the ridiculous drills we see in public education, since they tend to belabor the point. Yes, that stuff is important. So is learning your alphabet. But we don’t assign kids homework of singing the A-B-C sing for an hour just to make sure they’re good on the fundamentals. You don’t learn to cook by measuring out a cup of flour over and over 100 times. You just try to bake a cake. If you mess up, you figure out what you did wrong and try again.
My own suggestion for how to teach math would be to move to doing “useful stuff” as quickly as possible. I didn’t become interested in math until I discovered those bits on my own, but by then I was a senior and it was too late to course-correct.
My daughter is attending college and doing proper classroom math for the first time at age 23, and she has all the same complaints that I did at age 14: These are interesting ideas at first, but the sheer volume and frequency of drills crosses from excess into sadism. As a result, it obliterates any sense of curiosity. She’s at the top of the class, but it’s mostly because she’s stubborn, not because the work is interesting to her.
I get that some kids need to do the same damn problem 50 times to get the idea, but that’s not remotely true of all of us. Maybe the problem is that the curriculum needs to aim at the center of the bell curve rather than focusing on the bottom performers?
I don’t know. All I can do is suggest what would have made the class more useful to me, and whenever I do that I always have people explaining to me that doing things my way would ruin this for some other group of kids. I’d love to see someone do some actual science on this and figure out how much work is really needed. As it stands, it feels like the system is designed with the mindset of “I don’t know if kids will learn this after 3 problems or 30, so let’s just make everyone do 50 to make extra-sure.”
Sure you can try full recipes, but it’s not advisable, grinding measurement XP is a much faster way to train Cooking.
You joke, but that is I believe how many cooks train in professional cooking. Not “measure a cup of flour 100 times”, but more like “you’re cutting carrots into matchsticks till we have no carrots”.
Yup, that’s what on-the-job training actually looks like.
Trainee chefs spend most of their time doing jobs like turn “massive bag of potatoes” into “massive pot of peeled potatoes”.
Trainee machinists used to spend much of their time cleaning the machines, maybe realigning and zero-ing out etc.
– Of course, CNC has almost killed that profession.
There’s a couple further problems, too. One *really* weird thing about almost every math textbook I ever had was the way they focused on certain drills. They would have a concept, but would try to force students to do a huge number of random problems without covering obvious questions or explaining how it relates to other math. This is INTENSELY frustrating to the kind of people who might be most interested. It’d be like forcing children to study a language by making them memorize and repeat one syllable and its words, but never introducing grammar.
I know most everyone I ever knew hated word problems, whereas I thought they were fun. I actually thought we should turn it around and have students create them, for two reasons. First, it can help them actually integrate knowledge structurally. Second, once you are making them they won’t seem so intimidating.
Word problems are basically my entire career as a system administrator. “$USER3 claims $SERVER is $STATUS, using $APP on a $PC in $WHERE. $USER2 tried to help, but $NOCLUE, so it must be $YOURPROBLEM. Please fix it $MYWAY.” Just yesterday, I got 10 duplicate tickets from people complaining that a server wasn’t responding to HTTPS requests, which I closed with “Correct! Try HTTP!”.
Hardest part of training new people is getting them to understand “never let the user diagnose the problem”. Contract sysadmins in India are really bad at this, and need a lot of nurturing to develop diagnostic skills.
Back when I used to support student labs full of network-booted Sun 3/50 workstations, there was a very specific problem that could only be fixed by a three-finger salute (technically, “L1-A b”). Junior admins were given strict instructions never to do this in front of users. Because 90% of users will try it to solve every future problem they run into before asking for help, whether it’s a network outage or a paper jam.
“It’d be like forcing children to study a language by making them memorize and repeat one syllable and its words, but never introducing grammar.”
Hell, that makes me think of Japanese classes I’ve had. You learn a little bit of grammar/particles/whatever (the connecty bits you *need* to make actual sentences), but they drip feed it to you over time. I’d have stuck with it anyway, but the available cheap night classes (because I’m not paying full enrollement just for fun) stall out at 3rd term due to insufficient enrollment, before picking up again at like 5th or 6th with the people who got that far during the main university course. And hey, what do you wanna bet there’d be better retainment if it felt like you were progressing on actual speech faster by not being drip fed the parts you need to function?
As for word problems in math, I liked them up until the point when I stopped liking them. Early on it’s pretty easy to figure out what needs to be assigned to what- you’re basically doing algebra before you actually have algebra in class. But eventually it gets to the point where you’ve got a handful of similar but slightly different equation forms and the word problem is trying to parse the writer’s intent into what type of equation you need, but also often through their own bad writing.
For comparison, multi-variable calc was basically where I dirted out. I’d coasted through math with little effort up to that point, and could see the actual applications of things (physics with calc, advanced statistics, etc), but with no work habit once I hit a wall I stopped. And of course not having used any of it in a decade I’d have to dig out my books to do anything now. The bit about matrices though, yeah they should *definitely* tell people that’s a core part of graphics, because while I didn’t have problems with them, I just didn’t care, so they were just a bunch more bulk rather than something interesting.
What you describe was my frustration with math classes in elementary and middle school. I remember being pulled aside and talked to after class for refusing to memorize multiplication tables and dumb mnemonics for operator properties. I am and have always been a hands on learner, I don’t learn anything at all from memorization approaches, I learn by immersing myself in the subject and no teacher could manage that. I learned more about math by learning programming on my own and building a lot of things from wood and metal.
When I learned to solve sine and cosine with a ruler and compass to fold metal tubes into square ends. I said aloud, “why didn’t they show this to us in school”. It made the meaning of those operators so damn obvious for what they represented physically in the real world. The textbook in my trig class never really solidified what the sine and cosine operators did to perform their coordinate conversion magic, was a purely technical description.
This is an ongoing debate, one that will probably not be solved in the US due to the US’ mindset with regards to education and standards, but I suggest a reading into the “Math Wars” of 1989.
Math is an extremely difficult subject to breach because it’s one where many teachers don’t know how to draw an effective line between the realistic and the abstract. A great example is calculus. Calculus is especially at the forefront of this issue because it’s the first subject that has an abstract topic (effectively, the study of infinity), but is very much an arithmetic-like field when talking about applicability and calculations. It’s difficult to convey why an integral of an equation has a particular answer unless someone teaches you analysis (often a mid-level undergraduate level course in America), so there’s no intuition that a teacher can impart on you on why, say, an integral yields a particular result. So, in their minds, their only choice is to force memorization/drilling to get it to work.
Other countries that are more advanced in education have solved this by incorporating these sorts of analyses earlier into education, hence why you see such a huge gap in college education with regards to mathematics between domestic and international students in the US; they already have some backing in mathematical analysis earlier into their education. But in America, the fast raising of those standards is considered unacceptable, especially with the backlash over increased education and the “uselessness of math” in many parents’ minds, so the only solution IS to drill. Memorize because the reasoning behind the way things work in mathematics is considered “too advanced” to students.
And while calculus is an easy thing to point out in this case, this essentially applies to all mathematics in the pre-college level in America. Basically, it comes down to the idea that somewhere between intuition and applicability should come mathematical analysis (proofs, number theory, or the idea of “playing with numbers”, etc.), but that’s an undesirable aspect in America because the applicability is what most students/parents want. So drill drill drill to be able to apply it faster.
I dunno, my three-year-old does seem to have received that homework assignment regarding the alphabet song…
They don’t actually have memorization of math facts (or anything else, for that matter) in public ed anymore, at least not around here. Now the focus is on “tricks” for doing fast mental arithmetic, without making sure the kids understand why the trick works. The result is kids who can’t do math effectively even when they’re smart enough to grasp the concept, because they just don’t have the foundation. Like the square root example I gave above – that kid fully grasped the concept of simplifying a square root, but he was so innumerate he had to brute-force it, making a simple problem that should have taken literally longer to write down than to solve instead take several minutes of tapping on a calculator. Extrapolate that to basically every step of an algebra problem, and now you see the issue.
I don’t know. I always liked math, but I was in a Montessori program during elementary school, which allowed me to move at my own pace, so I wasn’t stuck doing addition when I was capable of algebra.
And I should clarify – it’s specifically addition and multiplication facts (which are the same as subtraction and division facts) I am referring to. There’s no reason to do the same type of algebra problem again once it’s grasped (I never did as a student unless the teacher required me to, which was unusual). But there absolutely is reason to drill the times tables, just as there’s reason to practice scales. Otherwise you end up not knowing that 13 is a prime number or that there are no sharps or flats in the C major scale.
Anyway, this isn’t about programming so I will drop it :-)
Totally agreed! I’m sure memorization is annoying for some students but the switch to “tricks” is a disaster for those who are not naturally gifted at maths (and those that are gifted would probably do well no matter which way you try to teach them). We have one child in each category and the one for whom it doesn’t come naturally was just lost because it was impossible to build foundational knowledge on dozens of little tricks. You need to understand the basics and know your addition and multiplication facts as a foundation before learning the tricks. Done in that order the tricks can be very satisfying as they are effectively shortcuts and if you have the foundation then many kids will be able to recognise why the trick works and hence the trick will stick in their memory.
But trying to learn the shortcuts before you know anything else is a disaster…
Fortunately our local schools in NZ seem to have realised this and are bringing back more foundational work, although it is too late for our kids.
I’ve never been a fan of ‘pile of examples’ class periods or massive homework sets myself, and I’m a math teacher. Fortunately for me, I work in higher education, where I have a bit more flexibility in terms of what I assign to students.
Figuring out how many problems to give students is tricky. I like to think my assignments are aimed at the median student. For what it’s worth, I’ve experimented with optional homework, supplemented with brief quizzes to check subject mastery. That got me course evaluations along the lines of, “If you want me to do homework, give me a grade for it.” Thanks a bunch, Alfie Kohn.
I don’t know why math worked for me. Some bad advice I followed is to not look at “requirements” or “majors” but instead do whatever class seems the most fun (while remembering that easy is not fun). Hard math seemed the most fun (none of that subjective stuff like the humanities has) so I took a bunch of them.
Another thing I did in high school and college is to look for the class majors take on the topic (not just math but any topic) and skip the accessible general-ed version of the class. That gets you with peers that like math and will give you good and crazy ideas instead of peers that hate math but need to take it anyway. I’ve been to a math class for business majors and dropped it the first week because everyone was dead and I could tell there wouldn’t be proofs or explanations of why a method works.
Perhaps the solution is tracked courses? That way we get rid of the one-size-fits-all.
I was only ever inclined to do math and practice the repetition when I needed to pass a test. But after that, I saw no use for it.
I became really interested in my own personal finances, however. Partially because I wanted to see what would happen if I spent x amount or put it into a % growth investment. But I don’t think that I needed to do hundreds of math problems to understand that.
I make puzzle games in Unity, more for my enjoyment than anyone else’s (I’m very much new to game design). If I suddenly need an RTS-style camera or a blood splatter effect or something I grab one from the Unity asset store and it’s working in 5 minutes.
Every now and then I think “It’s lazy of me to just stick with what I know, I should learn how to make games in C++ like the pros do!” and this series has been fantastic at convincing me to stay in my comfort zone.
How the hell does anyone ever make a game without the engine doing all the hard stuff for them?
If you’re working on AAA games, you probably work for a big company that has their own established libraries, written in-house, or you’re using Unreal Engine.
If you’re working on a non-AAA game, you can probably just use Unity or whatever.
In short, you don’t.
In games programming, there are three camps:
a) The teams writing games engines/games libraries
b) The teams writing games using the aforementioned engines/libraries
c) The teams not managing to program a game
As Shamus has mentioned many times, modern
gamesprogramming is far, far more complex than any one team could possibly do. You have to use a set of libraries.
The pre-packaged games engines like Unity, Unreal, Source, Godot etc are really a massive set of libraries that are designed to be used together.
Some people skip the “engine” and instead select their own pack of libraries. The downside of that is you then have the cost of finding a library for each feature you need.
So really, using Unreal or Unity is outsourcing most of the library searches in the article.
As I’ve said previously, my only really programming experience is with Neverwinter Nights, and the ever-common Nordock module. You’ve got a whole set of libraries for the ATS crafting, another set for the Hardcore Rules, another set for the Subraces, one for housing, another for replacement PC and NPC OnEvent scripts, and probably a few more that I’m missing. And then there are the modifications to make these libraries compatible for each other. I just remember going down SO many rabbit-holes (not having any other programming experience) trying to figure out how to change certain things while juggling compatibility with multiple features.
Bonus points if the incompatibility only applies to certain versions – yes, you could upgrade it and get the latest features, but it would break the other libraries you’re already using!
Been there. Yeah, the HCR has a new version! Ok, managed to kind of figure that out. Didn’t upgrade to the newest ATS. Totally replaced the Subrace system. Added a new AI script system. And all of this required much, much trial and error, and asking for help on forums.
It gets worse when you have multiple groups or products sharing libraries. I’ve seen multiple cases of “Team A has to upgrade to the latest version to get functionality X, but that breaks Team B because the functionality they need is broken or has changed in that version and so they start to fail if we use the latest version”. This always causes issues at the management level because Team A can easily make the case that they need to do that to make their feature work and so Team B should just fix the incompatibility issues, but Team B can rightly point out that there was no time in their schedule to fix problems like this because it isn’t part of their features, and so either Team A has to fix the problems in Team B’s code or else use the old library and fix their feature a different way. Usually, the team with the most political clout or at least the side that risks the most money wins, even if at a technical level it’s not the right decision.
In big, non-game projects, this isn’t likely to be the case in my experience. You simply don’t use new libraries because the process for getting them included in the actual installation is pretty intense. What we generally end up with is a couple of architecture or cutting edge designers looking at what third party libraries to include, proving they’re worth the effort, and then pushing that out to everyone else. But, again, I work on very big telecoms software where not breaking things is more important than getting something out in a hurry.
I will agree here, though, that this is indeed the one case where a new language for gaming seems entirely reasonable: if games rely on basic data structures that other fields don’t, then it’s a really bad solution to try to add those as basic data structures in languages where those data structures aren’t as important, and as it turns out many new languages were indeed created because what their base cases were ended up being complicated to implement in the other languages but since it was basic it was a big improvement to design a language that treated those basic cases as basic.
To be fair that’s the same problem of investing (and sometimes wasting) effort on libraries, just moved into a dedicated role instead of distributed across the whole team.
Well, to be fair to me, Shamus’ original comment was that we all had lost significant time to a library that turned out be useless, which hasn’t happened to me.
You make it sound like it is the case, it’s just somebody else who did it.
Which is probably just as true of a big game project where individual low level (as in on the totem pole) programmers probably aren’t authorized to just pick whatever libraries they want.
It probably is the case, but one guy “losing a day and a half adding a library” is going to be pretty small as a proportion of your budget.
Well, though, they don’t really lose time to it because it’s their job to investigate that and determine if it’s useful or not.
And on big game projects that was kinda my point: you never add libraries willy-nilly so wasting a day to find out that you shouldn’t have used that one rarely happens. Even ensuring that they’re compatible isn’t a job for most designers at that point. So you have people whose job it is to do that, and while you might be able to get them doing something else if you didn’t need to worry so much about libraries, at the end of the day you’re always going to have to include some libraries for things you don’t want to do so you’ll never save much on big projects.
On small ones, yes. Big ones, no, or else things will fall apart into a disorganized mess really quickly.
That’s a very… strange… way of thinking about project management Daimbert.
A project will have X number of man-hours and Y amount of money available to it. If something is simpler and/or quicker then it will require less X & Y resources and less expertise. It doesn’t really matter which individuals consume which parts of the available resources when it comes to meeting the overall project goals.
Or to put it another way: “You” can be plural. It can refer to an organization or project team. It doesn’t need to refer to an individual.
So, what we have is a large organization that has multiple big products that all have a varying size of individual projects.
The organizational level, for various reasons, needs to have people making sure that all of the third party libraries and products we use are reasonable and legal, ensuring that all of the licenses are up-to-date, and so on and so forth.
At the product level, we also need people to ensure that everything is compatible and, potentially, compatible with the products that depend on us and that we are depend on.
At the project level, it’s rare that a designer needs to add a library that we aren’t already using. But if they do, they are of course going to need to be able to demonstrate to the above groups that it’s necessary and compatible with everything else we’re doing.
There’s always some leeway, of course, but at the end of the day few people will try to add a library that they aren’t sure is going to work or will solve their problems. This makes the actual wasted time for looking at the wrong library minimal. As I said in the initial comment, at that stage we have all of these procedures because it’s more important to ensure that things don’t fail than it is to do them quickly or, sadly, at times even technically properly.
I make both front-end and back-end software by day, and we don’t do anything remotely this bureaucratic. We use whatever libraries are easy to integrate against, and it’s up to each of us to ensure that we don’t tightly couple ourselves to third-party code, beyond just making sure we don’t break anything.
Libraries typically have to be installed with the product and compiled against, and that impacts the installation and build procedures, and we have people whose job it is to make that work. We also have potential legal issues if we use things that we couldn’t use without permission. And since we often have over 100 people on multiple continents working on a product, we need to make sure that everything works for everyone because any breakage of compilation or installation can cost a lot in terms of employee time. Plus we might have other products using us and depending on us which makes things even more tricky. So you don’t get to change things in any noticeable way without everyone making sure that it makes sense.
Interesting to hear that’s how you guys do it.
I see choosing which third party libraries to incorporate as a management decision. Much of the information necessary to advise that decision should come from the developers but it seems to me someone with a big picture view (team leader or, more likely, higher) should be making the decision.
Maybe if the software is write once and not likely to be maintained beyond initial release then it isn’t so important?
Even if you were only writing software once, you’d want to use a library you’re familiar with, or that has good documentation, or… you get the idea.
Either way, the only way people get good at picking good libraries is by using them, and learning what ones are good or bad. I’ve had many technical decisions decided by people with varying degrees of seniority, experience, etc; They all make mistakes. If they’ve got a good library that makes my life easer, that’s great – otherwise, I’ll pick my own, to save myself a massive headache fighting with a poor library.
And this is how “It would be useful for me to know a bit of C++ so I can understand the programmers at work better” has turned into “Well, OpenGL isn’t too bad, but I’ve got no idea how to add in these other support libraries I need unless I chuck out my Visual Studio projects and rewrite everything in CMake” for me.
This isn’t even conflicting libraries (well, I’ll have to convert between the vector types at some point :P )
CMake literally squirts out MSVC project files on Windows.
So just run CMake manually, telling it which version of Visual Studio you have, and then add the project files it makes to your Visual Studio solution.
(It gets more interesting if you’re doing CI/CD, but it doesn’t sound like you want to do that)
I think it’s important to distinguish between a language and the language’s standard library.
Python doesn’t have regular expressions built into the language, but everyone uses the re module when they need regular expressions because it’s distributed with the language.
C doesn’t have a way to print strings built into the language. It has a standard library function that everyone uses, which can be implemented on any platform where printing is meaningful. The most visible change from Python 2 to Python 3 was moving printing from a keyword in the language to a built-in function (basically part of the language’s standard library).
C++ (though I hesitate to use it as an example of something good) doesn’t have linked lists built in, but its standard template library comes with a linked list implementation that everyone can use.
Fortran, which is what LAPACK is written in, doesn’t have an intrinsic matrix type; matrices are arrays just like in C. It does, however, have operations like vector dot product in its standard library.
I don’t think you necessarily want vector math built into the language with something like a vector cross-product operator or a matrix-multiply operator. I think you want a standard library implementation of those things and a language convention that treats particular data types as matrices and vectors that the functions accept. Maybe this is all nuance that you didn’t want to stuff into the already-long post. If it is, well, I’m sorry for being such a pedant.
No, the standard library is part of the language.
In the case of C++ it’s explicitly specified in the ISO standard.
The reason some languages split off some or all of their functionality into modules is so that each program only bears the “costs” (such as setup/teardown time, size etc) of the parts of the language that they actually use.
– If you never use a C++ linked list, then your binary doesn’t (have to) contain that code. If you never use regexes, the Python engine doesn’t (have to) load it when starting your program.
The standard library is part of the set of tools you get with a language. It can reflect the language’s philosophy (e.g. batteries included) but I disagree about it being part of the language itself. Languages have grammar and syntax. Standard libraries are typically just bunches of code that are so frequently useful that it would be silly for every programmer to reimplement them.
The fact that the standard that defines the language also specifies a standard library doesn’t mean that standard libraries are parts of languages. And the reason they’re split out into modules isn’t just so programs that don’t use them don’t pay for them. C++’s linked lists are just templates. Nothing in the STL costs anything if you don’t use it. On the other hand, libc is frequently a single dynamically linked binary. Even if you never use stdio.h, odds are good that you still have printf and malloc linked in.
If you never use virtual methods or exceptions in C++, you don’t pay for them. Those are language features that come with a cost. They’re not broken out into the standard library because they’re part of the language itself.
It can be assumed that if you’re writing C, you’ve got functions. But it’s perfectly valid to write C without the standard library. I do it somewhat frequently. It’s still C.
My last argument is that a compiler for C doesn’t need to know anything at all about the standard library. Many do, because being able to optimize memset is useful, but for most languages there’s no need for compiler writers to even know what’s in the standard library. If you can write a standards compliant compiler without it, it’s not a language feature.
I think you’re overselling how painful the lack of a standard vector type is. I mean, I could see some benefits from having one, but you can really do without.
Virtually every 3D-manipulating library out there will use “tuple of three numbers named x, y, and z” as its vector type. Usually a tuple of three floats, though they can be double-length or some other variants sometimes. Even then, it’s usually a matter of adding a one-line “convertToVector” function for every type you use, and calling them at the periphery.
That said, I agree that importing libraries in C++ in a pain in the ass, especially 3D rendering libs. Depending on how the library was made, importing a lib can go from extremely easy (header-only libs) to infuriatingly hard (libs that require their own CMake file to set up). Recent languages are much better about it, and usually have a streamlined import process.
Although those are technical problems. I think Shamus overestimates how hard it is to find libraries that are in the middle of the “easy to import, well-documented, uses up-to-date technologies” venn diagram. In the case of 3D rendering, nothing’s stb libs are pretty good on all the above fronts.
(unless you want a full 3D rendering library without using a pre-made engine; eg, you want Ogre3D and not Unity3D; then your options all suck for the reasons Shamus described)
I posted a comment, made a minor formatting edit, got marked as spam. Maybe I hit the “too many links” limit? Weird that it seemingly only triggered on edit.
I think one spam trick is to make a more legitimate post to get past the filters and then edit in rubbish at a later date. Perhaps that’s why edits can trigger flagging.
I’m not surprised that edits can trigger flagging – (e.g. I’d expect it if I added more links to the comment), but IIRC I just fixed a typo or slightly restructured a sentence.
“If you decide to do things the Right Way™ and have your strings to support Unicode, then you are basically doomed.”
Quoted for truth. And these days trying to do things in ASCII can be difficult because all the cool GUI tools want you to use their Unicode half-solution, and so they hide ASCII.
Unicode will be more generally supported as time goes on, simply to facilitate international software. Lots of web tools / libraries / frameworks already support Unicode well.
I’m sure that’s of great comfort to future developers…
Well the future has been here for more than ten years. Python 3 has proper unicode support and it was released in 2008. I am not a web developer but I am quite sure a lot of languages that are meant for the web will have had this support for quite some time.
On the other hand. None of these languages is suitable for the workloads Shamus has decribed which are required for games.
Vectors are higher level structures than should be handled natively in a programming language.
You yourself see that a given rigid object needs six numbers to describe its position in Cartesian space, but then speak as though it needs multiple vectors for the one thing “In your typical 3D scene, you usually need several vectors: One for position, one for rotation / orientation, and sometimes one for scaling.”, when you really need one six-axis vector for position and a scalar for scaling.
It doesn’t make sense to implement vectors in a programming language, because very few programmers will want to do the same things with vectors. Some will want to do translations and rotations separately, some will try to coerce rays to three values, and some will try to do pathfinding on arbitrary networks using vectors.
Let’s just implement different kinds of vectors and call them Euler vectors, Bayesian vectors,…
I’ll note that not even Fortran, a computer language designed to do math problems, has a default vector data type or structure. (Or at least it didn’t. I suppose things may have changed with more recent Fortran standards.) When I was doing Fortran coding, I’d just use a one-dimensional array to represent a vector.
MAYBE it would be worthwhile to have basic functions within the programming language to do addition, subtraction, scalar multiplication, dot products, and cross products with one-dimensional arrays.
But in order to implement that correctly at the software level, the entire idea of ‘n-dimensional vectors’ is shown to not represent reality- if the programming language inherently allows thumbstick inputs to be stored as vectors and for cross products of vectors to be calculated, then the thumbstick vectors have ALL of the axes that any vector does. (Not that you’d get anything useful from such a cross product, anyway).
“Calculating the location of the vertices of an object, given the shape of the object and a six-element array describing its location and rotation” seems like the kind of thing that a library would be useful for. But that presumes a standard method of describing vectors in arrays, which hits the ‘competing standards’ situation very hard. For that matter, it presumes standardization on shapes of objects, since rotation requires a point of rotation, and some users might expect that to be the origin of the coordinate system in which the shape of an object is described, while other people might expect it to be the center of the object, even if the coordinates of the object are all positive. Attempts might be made to require a ‘point of rotation’ be specified, which is going to really frustrate that one guy who tries to make a Marker which is supposed to be a single point that just creates a glow or omnidirectional, but now he needs to specify the rotational center of a single point?!
Since different users will have different needs, making the vertex handling part of the programming language will block most users from using the language at all. At least with libraries it’s theoretically possible for the needs of two users to be met at some point.
Surely if our goal is ‘language for AAA game development’ the needs of the users are going to be pretty similar? When there’s a large team working on an open world game they need to have a shared understanding of how basic concepts like position and rotation are defined.
The sort of thing I do in Unity seems pretty convenient.
Vector3 diff = target.transform.position – attacker.transform.position;
float distance = diff.magnitude;
What’s the advantage of a ‘six-axis vector for position’?
The advantage of using a six-axis vector for position is that it is adequate to describe position, where a five-axis vector is not. You can ignore three of the axes in calculating the distance between centers of rotation, although not the distance between edges.
It feels to me that if a ‘position’ requires more than x, y and z, that’s cramming too much information into a single concept. Rotation should be kept separate. But maybe we’re using the same words to mean different things.
Think about how a coffee cup moves when it is being drunk from.
Now, consider how it would move if it only had three axes of movement.
Rejoice! Java comes with a default Vector class! The problem is that it’s more a list of arbitrary length and less a proper mathematical vector. A Vector can be a list of anything rather than just a list of numbers, so it’s not designed with mathematical operations in mind. And I’m pretty sure that Vector has been rendered irrelevant by ArrayList at this point anyway. So not really a vector class then. Sorry. The libGDX Java library, on the other hand, includes two proper vector classes, one for 2D vectors and another for 3D vectors. Among other things, the 3D vectors are used in the Sprite class. As Shamus suggested, the first two elements are for position and the third for rotation. I’ve used the 3D vectors that way for game objects in a 2D physics simulation as well.
As for libraries more generally, my question is do we really need a new programming language for games or do we just need a few gaming-specific libraries that play nicely with each other? Wouldn’t that more or less accomplish the same thing as Jai but with less effort? With libGDX, I don’t need one library for sound and another for graphics because libGDX includes packages for both those things. Granted, libGDX is totally unsuited for AAA development, but I don’t see why its AAA equivalent couldn’t exist if somebody really wanted it to. Heck, isn’t that what proprietary game engines like Frostbite are, at least in part?
2D and 3D vectors don’t, practically speaking, exist.
What exists are vectors about which only two or three dimensions are known, and vectors in which all but two or three dimensions are zero; all vectors have all the dimensions.
Infinite-dimensional vectors seems like they would take up a lot of memory…
You could just say that any dimension not explicitly initialized is zero. This is already common practice with sparse matrices.
(Of course, some people would reasonably prefer their code error out if it tries to measure something’s fifth-dimensional velocity)
If the vectors are of indeterminate length, they’ll be slower to use than ones built for a fixed number of dimensions. For many problems that amount might not matter enough to care about, but this series started out talkign about game-engine development where there’s very high requirements for performance.
The slowdown is one additional memory operation per time accessing a vector; instead of multiplying the vector ID by the size of a vector to get the offset, multiply the vector ID by the size of an offset to get the offset of the offset, then look up the offset.
With better optimization, the vector ID can be the offset of the offset, skipping a multiply step but reducing the available space for vector IDs.
With perfect optimization, the vector ID can be the offset, which is faster than the reference condition of fixed-size vectors, but that further reduces the available space for vector IDs and makes assigning a new vector ID MUCH more expensive, especially when old IDs have been released.
Fifth dimensional velocity is rate of change of pitch.
You don’t need infinite-dimensional vectors, just every dimension. Only track the ones you care about, if you’re worried about memory footprint.
But don’t fall into the mistake of thinking that dimension that you aren’t tracking don’t exist.
It doesn’t seem like a particularly dangerous mistake. The difference between, “Yetis don’t exist,” and “Yetis exist and there are zero of them,” is pretty abstract.
The error is in saying “Yeti don’t wear clothes”, then in saying “Wookiees are just brown yeti, then in segfaulting when Chewbacca puts on a bandolier.
Or when someone needs to know the magnitude of a cross product of vectors.
Isn’t the point of frameworks? Halfway between language and engine, providing a unified set of tools for a domain of programming?
I am certainly not denying the bad experience with C++ and libraries, but, as a mitigating info, for what is worth…
1 – Bringing a type into the language is dangerous
A vector, 3D or not, doesn’t really need to be a builtin/primitive of the language. But having it part of the standard library means everyone agrees on its definition, and that means all game libraries would use it… until it is discovered it has some flaw, and then, you have a versioning problem, right inside the standard library. That’s to avoid that problem that type needs to be near perfect from the start.
A solution would be for the game devs to build their own standard library (on C++, it could take over a namespace, like “sgd” (standard game development), and game developers/game industry could work together so that “sgd” becomes the common library for low-level primitives. Once, done, all other libraries would have little choice but support that ASAP. But building a standard library is hard.
2 – Libraries are becoming much easier on C++
– CMake is the the-facto cross-platform standard for “project files”
– Conan is the the-facto cross-platform standard for dependencies management
– VCPKG is a serious contender, there, as, if you’re on Windows (and now, Linux), it brings you libraries at little to now cost, integrated in your IDE so using a library (default includes, dependencies, linking) is as easy as including one of its headers.
I despise CMake, personally, but I can’t argue with a de-facto standard. Conan is a great way to handle packaging and dependencies if you have large projects.
VCPKG is just awesome. Once the tool is installed, you can look at the available libraries, then bring them on your machine with a single console instruction. That done, it is made automatically available to your Visual Studio (if you use that IDE), so all you need is include the header, and it will link with the necessary libraries, and even copy them automatically in your output directory, right where your EXE or your DLL is generated.
You can find it on Github, with instruction to install it with a few console instructions (assuming you have git installed): https://github.com/microsoft/vcpkg
VCPKG currently supports 1236 libraries (I put together the complete list of packages here: https://drive.google.com/open?id=1MXvsyIQPoieBRHkArAkUneiCoFRSFhboZQTC2qALZhs)
Really, if you are using Visual Studio on Windows (I have no first-hand experience with the Linux version), having VCPKG will CHANGE your life.
P.S.: Fun Fact: I know from hearsay that the C++ Standard Committee and the C++ Game Developers are really not happy with each other. There are a few people trying to mend the gap, but the genuine priorities of each seem to not align. That’s sad, but at least, communication, like Shamus’ posts on the subject, or even attempts at alternative languages (like Jai, or Rust), are welcome to find understanding.
Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*
You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>
You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?
You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.
You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!
You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>