No content today. Except for this.

By Shamus Posted Tuesday Apr 14, 2020

Filed under: Programming 43 comments

This no-content post is brought to you by sloth, incompetence, and malfeasance. I have lots of content in the pipeline, but I recently got sidetracked by a project that I figured ought to take “a few hours”, and has now devoured much, much more than a few hours.

It all started when my son Issac showed me a Garry’s Mod level he was working on…

This is what it looks like in Garry's Mod.
This is what it looks like in Garry's Mod.

I thought this looked really cool. We started talking about what you could do with this. Eventually I got the idea in my head to load this map in Unity. That seemed like a fun afternoon project. After all, Gmod uses the Source Engine 2 BSP format, which is based on the Source Engine format, which is based on, no kidding, Quake 2. That engine is literally older than Issac. Some of the games that use these maps are open source. It should be trivialFamous last words. to load that into a modern setting and render it. I might need to flip some coordinates around or do a little scaling to get the map right-side-up and sized properly, but that’s easy. I can probably bash this out in a couple of hours.

Days Later…

Still Garry's Mod.
Still Garry's Mod.

So I underestimated this project by an order of magnitude or so. The main problems were:

  1. It’s been like 18 months or so since the last time I looked at Unity. I never fail to be amazed at how much you can forget in that span of time. I spent ages stumbling around, Googling things, and then slapping myself in the forehead saying, “Oh, right, right. THAT’S how that works.” It’s exactly like being a newbie where you don’t know what you’re doing, but then when you look it up you feel dumb because you realize you used to know this.
  2. Yes, it’s good when technology is old because that means the documents are mature and all the likely pitfalls have been documented and explained on StackOverflow. But that only helps up to a certain point. If something is TOO old, then modern knowledge-sharing sites won’t cover it and all the docs are 404 links. This is the only document I could find. It’s good, but it’s not detailed enough to work as a full spec. There are some very important details that it leaves out and that can’t be intuited or guessed.
  3. Yes, being old is good because graphics engines were simpler 23 years ago. They didn’t have to contend with shaders or any of the complex modern lighting systems. It was just a bunch of polygons that you shove over to the GPU. Kid’s stuff, by today’s standards. HOWEVER, those games were ALSO pushing up against the hardware limitations of later-90s devices and so they had lots of optimizations that would be pointless (or even counterproductive!) by today’s standards, but which make the code very hard to follow. Yes, a ground-up implementation of Quake 2 would be easy to do today, but the versions of yesteryear needed to be much more complex, and that complexity makes it seem very convoluted.

This was all very clear after the first couple of hours. At that point a responsible man would shelve the project and get back to work on the blog. This was obviously turning into a boondoggle and I had more important things to work on. (Like, for example, content for this blog.) But at that point I just couldn’t let it go. I wasn’t working on it because I needed to see a BSP in Unity. I wasn’t working on it because I had a brilliant idea for a game. I was working on it because I couldn’t rest until I’d solved the problem.

Here is my re-creation of the source engine! Just kidding. This is still Garry's Mod. My results are... not this good.
Here is my re-creation of the source engine! Just kidding. This is still Garry's Mod. My results are... not this good.

I’m not going to do the math and figure out how much time I threw into this hole, but it was more than a reasonable person would spend on a project with no particular goal beyond, “It might be cool if I did this.” After $unreasonable_number of hours, I FINALLY managed to read the geometry, figure out how the polygons were formed, get them all facing the right way, and sort out the surface normals.

The result?

This is the best I could do. Maybe if I throw more time at it, it could look slightly better.
This is the best I could do. Maybe if I throw more time at it, it could look slightly better.

The other thing that ate a bunch of time is that I was a fool and tried to use Unity’s new raytracing system. I’ve said before that Unity’s docs are always criminally incomplete. The docs will tell you 75% of what you need to know, but then that last 25% is folk knowledge that you can only obtain by pleading for help from other users.

If that’s the state of their regular documents, then you can imagine what a patchwork the beta documentation is. I slammed my head against this problem for about half a day, just trying to render a simple box room with a few small objects and a couple of light sources. Everything the engine was doing claimed that it was raytracing, but it was very obviously still using current-gen shadow volumes and regular shaders. Also, sometimes the docs talked about “raytracing” and other times it talked about “path tracing”. Now, from the point of view of the gaming public, it’s 100% okay to use these terms interchangeably. That’s fine. I do it myself. You can’t afford to stop and explain to Joe and Jane gamer the difference between the two every time the topic comes up. But I feel strongly that if you’re making an engine for developers, then you should be 100% clear about what the engine is supposed to be doing.

As an example of what a disaster area the docs / tutorials are. You get exactly one example project of raytracing / path tracing. If you open it in the latest version of Unity, then Unity will complain that it was built using an older version of the platform and might not work. (Which is true. It doesn’t render properly.) So then you download the previous version of Unity. That version claims the project is too NEW, and it won’t even compile. So the one example program they offer is for some in-between version of Unity that is no longer available. I’m sure you could fix the demo program, if you already knew how it was supposed to work. But since the point of this project is to teach you that, you wind up with a chicken vs. egg problem.

In the end, I threw away all the raytracing stuff. It’s obviously not ready for casual inspection, even by hobbyists. I’m sure it’ll be ready for general use in another couple of months, and the documentation will be done in the next decade or so. Maybe I’ll try again if I live to be 60.

Anyway. That’s why you didn’t get a post today.




[1] Famous last words.

From The Archives:

43 thoughts on “No content today. Except for this.

  1. ydant says:

    But… these are the kind of posts I come to this site for!

    1. Kyrillos says:

      Exactly. I know this is probably a huge headache for Shamus, but if he wanted to bash his head into the Unity wall for a while and make a couple more posts of getting this level working, I’d read that.

    2. Echo Tango says:

      Hopefully the time-spent to article-value ratio isn’t too horrible for this post, compared to other normal ones. I know a lot of editing, writing, etc goes into the other ones, but only Shamus has the statistics. :)

    3. raifield says:

      Same, it was the “create a city with a few blocks” series of posts that made me a regular reader of this blog. I love this sort of stuff. I’d love for his proto-Minecraft software thing to be revisited at some point.

      1. DrCapsaicin says:

        I’ve read Shamus’ stuff for a long time, but Pixel City might still be my favorite article series he ever did. I have limited C++ knowledge (used it for grad school, but never for graphics) so knowing just enough to be dangerous and to follow the more heady stuff has always been some of the best stuff he has done, in my opinon.

    4. Kronopath says:

      +1 on having you dive into the details of this project and explaining your trials and tribulations. As another person who came to your blog from Pixel City, I’d be more interested in this than in most of the gaming retrospectives I’ve seen here.

  2. Nimrandir says:

    [looks at Shamus’ best effort image]

    You know, either my memory is playing tricks on me, or that looks kinda like the interior of the Remnant vaults from Andromeda.

    Were you moonlighting without telling us, Shamus? ;-)

  3. Draklaw says:

    From your unity screenshot, it looks like it could be (old school) ray tracing and not path tracing. The difference is that you won’t get global illumination (aka indirect lighting) which makes the original level look cool.

    I’m not sure that today’s hardware is ready for path tracing, except maybe in simple cases like minecraft. This might explain why it is not enabled by default in unity.

    1. Actually both are completely adequate for photo-realistic lighting. There’s ambiguity between the two terms, but usually the difference is that Raytracing traces from the light source and Pathtracing traces from the camera.

      Both support Global illumination, it’s just that certain paths are easier to detect with one or the other. This is why some photorealistic render engines like Luxrender use something called Bidirectional Path tracing, which actually does both at the same time and connects the rays in between.

      Perhaps unity uses these terms differently though, so I don’t know

      1. Draklaw says:

        Ok, terminology is a bit messy with ray tracing, ray casting and path tracing.

        Ray casting: as far as I can tell, the general idea of intersecting ray with geometry to solve problems.

        Ray tracing: ray casting applied to image synthesis. This can include a lot of technics, that’s why I talked about “old school” ray tracing, where rays are cast from the camera and secondary rays are used for shadows, reflections and refraction, but not to compute global illumination.

        Path tracing: a stochastic method where randoms light paths are traced from the camera and averaged over each pixel, or as wikipedia’s says: “Path tracing is a computer graphics Monte Carlo method of rendering images of three-dimensional scenes such that the global illumination is faithful to reality.”. It supports global illumination as well as acoustics (although it takes ages to converge in cases where important paths are unlikely to be found).

        Several methods use the concept of casting rays from the light, like photon mapping or bidirectional PT, but I don’t think it has a dedicated name.

  4. Armstrong says:

    As someone who dabbled in abstract BSP mapping for fun in the past, I must say that gmod map looks really cool! I especially love those lights placed in the cracks between the ceiling hexes.

  5. Lino says:

    Wrestling against un-intuitive systems, lamenting on poor design decisions, and getting angrier and angrier as the situation develops? Yup, that’s the kind of content I signed up for!

    1. Ester says:

      I also wouldn’t mind it if this story got a few more entries.

    2. DaveMc says:

      Yeah. We’re not saying that we *want* you to torture yourself for our amusement, Shamus, but … Well, I guess that is what we’re saying. :)

      [But not past the point where it goes from “Ho ho, isn’t this silly?” and into “This is actually harming my mental wellbeing.”]

  6. Gargamel Le Noir says:

    You could have made this a Programming post while removing the 3 first lines and we’d have been happy with it Shamus. Seems like you can’t help making interesting content, even if you set to write yourself an excuse slip.

    1. Nimrandir says:

      One of the best parts about this post is how many different categories it could fit under. It’s currently under ‘Projects,’ but ‘Programming’ also would have worked, and the bit about Unity documentation probably qualifies as a ‘Rant,’ or even ‘Random.’

      Heck, if you include my comparison of the image to Andromeda levels, it’s even related to Mass Effect!

  7. Sleeping Dragon says:

    No comment today. Except for this.

    I’m here primarily for your comments on storytelling as most of your programming posts go above my head. That said, this is precisely the level where I still find your struggles interesting and amusing without having to figure out technical details. Also, there is still something visually intriguing about your final results image, must be the concentration of geometric shapes that draws attention.

  8. ccesarano says:

    I know this might sound weird, but thank you for this notice. It was still an interesting read, but when I saw a lack of content throughout last week I began wondering if maybe something had happened. Nothing major, just the possibility that you got sick or something, but considering the current state of the world that could mean a lot of things.

    So I consider “stubbornly trying to make something work or else I can’t sleep” good news compared to the potential alternatives.

    1. Nimrandir says:

      Very true. I almost posted in the comment thread of last week’s ‘whatcha playing’ post asking if a new retrospective was in the works. In an unusual stroke of optimism on my part, I didn’t even consider bad alternatives.

  9. Echo Tango says:

    Were you loading the level straight into Unity, or was this more of a command-line program, that just converted file-formats, that you then loaded into Unity? I’m curious to know which you chose, and which one is easier – at the beginning, or over time.

    (I do “normal” programming by day, and have only face-planted into the Unity tutorials once, before deciding I don’t like using special scripting languages, or having static objects from levels rule the code[1], and can’t work without a code-oriented IDE.)

    [1] Instead of always just spawning objects from your own code.

    1. Shamus says:

      The goal is for my program to read the BSP, turn it into usable polygons, and let you run around in it, with no external conversion required. That’s what it does, aside from the missing lighting, entities, texture data, collision data, etc.

      I’ve made some progress since I finished this post.

      * Levels above a certain level of complexity would be missing huge sections. I really thought there was something wrong with my program. Am I parsing the BSP wrong? Am I not translating it properly? Is there some special-case conversion I don’t know about? After slamming my head against this for over two hours, I discovered this isn’t a problem with the map OR my code. I was just running into the hard limit of 65535 vertices in Unity. That’s reasonable, although I’m baffled that Unity didn’t see fit to give me a warning. “Hey boss. That model is too big so I threw half of it away.”
      * The level is now textured, although I still haven’t worked out how to use the ORIGINAL texture positioning.

      1. Richard says:

        I use the Assimp library to load these.

        You might find the MD2, MD3, MD4 and MD5 importers useful for understanding the format.

        Also, Unity does in fact document that mesh views have a hard limit of 65535 verts. You can actually have bigger buffers if you render them in blocks, but I wouldn’t recommend doing that.

        Splitting a mesh efficiently is a ‘hard’ problem, however you’re starting from a BSP which means someone already did the hard part for you.
        Which is nice :)

      2. neolith says:

        That’s a problem that should be solvable.

        Unity by default stores mesh data with a 16bit index, hence the 65k limit for vertices. It does that, because very old graphics cards and certain mobile devices do not support anything larger. At the time the engine was made that was a much bigger problem than it is now. Things have evolved and these days we are able to switch to a 32bit index, giving you a whopping 4billion vertices per mesh. However, 16bit is still the default because of memory

        You should be able to easily change your code like this:

        GetComponent().mesh = myMesh = new Mesh();
        myMesh.indexFormat = UnityEngine.Rendering.IndexFormat.UInt32;


        That being said, if you used Unity’s own importers in the past to get existing meshes into the engine and your mesh hit the 65k limit, you would get several split up meshes combined into one IIRC. Nowadays the engine switches said meshes to UInt32 automatically.

        PS: I love posts like these, because I really like to read about other people creating stuff. It always gives me new ideas. :)

        1. Richard says:

          However you shouldn’t do this unless you’ve proven that it’s definitely better for your meshes.

          Doubling the size of the index buffer is only worthwhile if it is actually going to make a big difference to the number of draw calls.

  10. DasMaddi says:

    I work with the Source Engine daily for my standalone project, and documentation for the engine is definitely not great. Most of it comes down to the outdated Valve wiki, and learning from the numerous discord servers, which is not an ideal method whatsoever. But the “campiness” of the engine is what keeps people around in this day and age I suppose.

  11. tmtvl says:

    all the docs are 404 links

    This is far too late to be of any help, but when I run into these kinds of situations, I like punching the URL into the Internet Archive.

  12. Simplex says:

    “I was working on it because I couldn’t rest until I’d solved the problem.”
    I sometimes suffer from this too, it can be a blessing or a curse.

    1. Philadelphus says:

      Yeah, this happens to me too. Thankfully it’s pretty rare, it only seems to be triggered by problems with my operating system. (Sometimes I wish my PhD research would trigger it, I’d probably be done by now…)

  13. John says:

    This is why I stick with 2D graphics. I can just about manage “draw this image here, scaled like so, and rotated this amount.” Maybe one of these days I’ll get around to applying color filters or something so that I can palette-swap a sprite. 3D is intimidating and utterly beyond my ken.

  14. Joe says:

    “Gmod uses the Source Engine 2 BSP format, which is based on the Source Engine format, which is based on, no kidding, Quake 2.”
    I have a very distant memory of Carmack talking about the Q2 engine. He said that parts of it dated back to Doom 1. Did you use those parts? I dunno. But that dates back to either 1992 or 93.

  15. Socks says:

    Really digging the hexagon approach. Could be space-age, or ancient civilisation, or…

    The “If something is TOO old” missing documents is equally as annoying as the outdated information available (e.g. OpenGL glBegin etc… which has a specific term?). You sort of have to know what you are looking for before you search.

    I appreciate tools like gcc that allow me to specify which version of c/c++ I want to use.

    Related, raytracing in minecraft coming soon: RPS Minecraft Raytracing.

  16. evileeyore says:

    I’ve always felt that today’s header pic missed the opportunity to have the last line read “And the Authorities Have Been Notified”, just to at least have a version of “notice/noted” in it.

    1. Glacious says:

      I’ve had the same thought. Consider yourself notified.

      I’ve always found the header picture funny and laughed about what city council would post a sign like that. Then my brother sent me a photo of the very sign when he stumbled across it one day and the floor was smacked an uppercut by my jaw. Not only is the sign in my city, but it’s on a bus route I drive regularly! I’ve been past it hundreds of times without noticing. Now I get a chuckle every time :)

      1. Nimrandir says:

        It’s not as good as the “Obey This Sign” I saw in a Zaxby’s, but I enjoy it nonetheless.

  17. Christopher Wolf says:

    This has been the greatest post in the history of posts. I have spoken.

  18. ElementalAlchemist says:

    I do have to wonder why you continue to persist with Unity despite every time you post about it proving how terrible it is.

    1. Shamus says:

      What do you imagine the alternative is?

      Is there some perfect development environment out there that everyone has been keeping secret? Or should I go back to building everything by hand in C++?

      1. Jamie Pate says:

        godotengine is pretty spiff (and was extremely easy to pick up)

        qodot is a plugin for godotengine that reads bsp files

        trenchbroom is a level editor for bsp files (though I haven’t tried it)

      2. ElementalAlchemist says:

        Or should I go back to building everything by hand in C ?

        If you were developing a commercial game, I’d probably say yes. I have had some minimal exposure to Unity on the (art/non-coding) development side, and there were some interesting features, especially a decade ago when such things were pretty novel. But as a game purchasing enduser, Unity is a goddamn trashfire of an engine that I avoid like the plague. Even big dev studios struggle to curb its worst tendencies (see, for example, stuff like the massive memory leak issues Obsidian’s Pillars of Eternity had which also plague every other Unity game I have played).

        But if you are just messing about with non-commercial stuff for your own amusement, I’m curious why you haven’t given Unreal 4 a go (or at least you haven’t posted about doing so, that I can remember). It’s hardly without its own set of problems, but I suspect it may alleviate some of the issues you have run into with Unity.

        In this specific case, maybe you would have had more luck with id Tech 4 (i.e. the Doom 3 engine), since that is open source and presumably shares some heritage (albeit distant/indirect).

  19. Decius says:

    “the documentation will be done in the next decade or so”

    How very optimistic.

    1. aradinfinity says:

      It might be slightly more accurate to say, “the documentation will be /no longer worked on/ in the next decade or so.”

  20. +++ says:

    Did you know that Dusk SDK is essentially the same idea? Takes in a variety of .bsp formats, renders them in Unity. So hey, taking days to parse geometry seems alright when put against what – years? – it took Dusk devs to roll out their thing.

    Now, the sport of programming aside, there were more practical ways to achieve similar results, like using Noesis or Blender to read the file and export to something readable by Unity. I’m not sure how robust those are with the variety of bsp versions, but I’ve had success importing fan-made Quake 1 extended bsp formats, so I’d assume the more mainstream ones should be good to.

  21. Taxi says:

    Man it’s grazing me so much to hear Source engine was based on Q2.

    Anyway I’m sure lots of other people have commented on that already so I’m off back to reading the hilarious adventures of porting a Quake engine map format to the documentation hell that apparently is Unity.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply

Your email address will not be published. Required fields are marked *