Procgen Everything

By Paul Spooner Posted Tuesday Aug 13, 2019

Filed under: Projects 34 comments

I promise we’re going to finally get to actual programming in this post, but first a huge aside.

When I started this project, it was 2010, and my wife had just purchased me the Star Trek Enterprise D blueprints. I was inspired. But it was an odd kind of inspiration. The kind that starts building connections into other parts of your life and making you think strange thoughts.

It began innocently enough. I noted, as I was pouring over the plans, a certain similarity to the masonry script I had written a few years prior. I was feeling the need to get back into coding, and this seemed the perfect opportunity! Just re-purpose the old code to make starship floorplans! How hard could it be?

They are practically the same already. Good work everyone.
They are practically the same already. Good work everyone.

It didn’t hurt that I had some experience making space ships from floor plans. Back in 2007, I started working on a 3D model of a spaceship based off of some bitmap floorplans designed as headquarters for a Starwars fan-fiction group. The project was never really “finished” because there was never really a goal. It was mostly motivated from “wouldn’t it be cool to have a 3d model of this whole spaceship that we’ve been writing stories about?

When I made this, I had not figured out the 'cool hull plates' technique. Arguably, the case is the same today.
When I made this, I had not figured out the 'cool hull plates' technique. Arguably, the case is the same today.

So I figured it wouldn’t be any trouble to have the computer build the space ships instead of doing it by hand.

It Was A Lot Of Trouble

Turns out I’m a crazy person, and the project stalled out. But not before producing what I am resurrecting in this article.

You see, I began working backward to a starting point, simplifying over and over until I had something I could wrap my brain around. The star-trek guys started with the Enterprise C, which was an iteration from the B, which was based on the A which was, from the looks of it, based off a toilet seat cover and some toilet paper rolls. A “shape first” approach. Which, when all you really need is the outside of the starship, is a fine place to begin. Sure you have the problem of having the geometry “read” properly as to scale and function, but those are problems with potential solutions, as I have been attempting to demonstrate with this series so far.

So, me being disagreeable, I decided to start at the other end. With the internals. I didn’t just want to wave my hands at the problem and say “it has space magic technology that goes “vroom” and “pew pew” when it needs to.” I wanted these ships to make real sense. The Enterprise D had a crew of 1014, but what did they all do? What was the skeleton crew? Sure, there were 4 staff on duty in the sick bay because that makes for good drama when one is a main character, two of them are holding down a patient, and the last one gets killed by the disaster of the week, but I wanted more technical answers. I wanted to be able to perform a calculation!

So I built a calculator.

Science again!
Science again!

Well, technically I used a giant absurdly powerful hardware calculator to run a calculator based calculator platform to run a different software calculator to iteratively program another, different, smaller, less powerful calculator. In the engineering business, they call this:

Re-inventing the Wheel

And while it’s often used with a deeply derogatory connotation, and while that connotation was probably appropriate in this case, there are certain advantages to re-inventing the wheel. Sometimes, the wheels you need just aren’t readily available! Sometimes you want to better understand the wheel instead of just buying one off the shelf! And sometimes the wheel you want to re-invent is a super-ordinate meta-wheel which, as it rolls, stamps out other smaller wheels. You also shouldn’t try to re-invent the metaphor.

Get to the Point Already!

But that’s the point, isn’t it? Once I started making a calculator for calculating how many crew cabins you would need in an arbitrary space-ship, I realized it was a very small step to making a calculator for calculating how many abstract production groups you would need in an arbitrary resource network balancing system. So I did that, and it worked fairly well.

Before further excursions into madness, I’m going to mention that rockets have a bad problem that, while not unique to them, is at least fairly uniquely bad for them, where they need to carry fuel, and the more fuel they carry, the more fuel they need. This calculator was designed to solve just this sort of iterative self-referential problem.

But then I had a further realization that if the calculator could be based on nodes, which in turn could contain other nodes, then it would be able to calculate everything. And if I could calculate everything, then I could:

Procgen Everything

Don’t you see? It’s all so elegant! The basis of a net-production network undergirds all sustainable systems, and when non-sustainable resources are given abstract trans-finite sources, one can calculate not only the spatial dimensions, but the temporal dimensions as well! N-dimensional computation would be trivial! Every problem had only to be defined as a linearized solution space! And even non-linear solutions could be approximated by a set of discrete linear segments, represented by nodes! The spatiotemporal output maps onto an effectively infinite set of implementations, allowing for style and preference, all based on a single unifying ruleset used to generate the nodes! It could:

Procgen Everything

And you could procgen game mechanics, and time itself, and minds, and stories! The possibilities!
Do it!

Procgen Everything!


Let's take a moment to enjoy this proc-gen star-field. Man, it just never gets old! Who needs ships?
Let's take a moment to enjoy this proc-gen star-field. Man, it just never gets old! Who needs ships?

So, that was ten years ago.

I still hope to be able to procgen everything. At some point. I’ve written in fits and starts about it from time to time, which you’re welcome to read if you like the craziness of the above.

But in the meantime, I’ve made tremendous strides, and today I am happy to announce that the calculator can very nearly handle nested nodes. I mean, it definitely works for nested nodes, but it can only solve one layer at a time, though it should be a fairly trivial to add the ability to…

Ahh, see, there I go again.

To avoid dissipating into the infinite sea of the possible, I’m going to strictly limit myself to the here and now. As in: Here is what I have right now.

The calculator is which is a Python program. I wish I had the time to go over the whole thing in depth, but you can read the code and the comments if you are interested.

It takes a specification file (Specification.txt is an example), and solves the specification, and outputs specification files of the solutions. You can then copy-paste those specifications into other specification files, and, well, you get the idea. There are a few examples of the output in that folder as well.

Although the program itself is written in Python, the specification file itself is a very simple flexible format which I made up, and which I hope is easy enough to pick up yourself.
For example, if you use this as Specification.txt:
!Potted Plant
petals 5
odur 1
Pot -1
Soil -0.21
Soil -15
Potted Plant -7
!Box of Pots
Pot 5
!Bag of Soil
Soil 12

then the calculator will spit out this in the “Garden.txt” file:
? generated in 3 iterations.
? Originally calculated from Specification.txt
petals 35.0
? petals generated 35.0
odur 7.0
? odur generated 7.0
Soil 7.530000000000001
? Soil generated 24.0
? Soil consumed -16.47
Pot 3.0
? Pot generated 10.0
? Pot consumed -7.0
? All local nodes and resources are as follows:
? Soil -15.0
? Potted Plant 7.0
? Bag of Soil 2.0
? Box of Pots 2.0
? node results for Garden complete

The details of how to write your own specification file are in the example specification file.

As you can see it can calculate anything at all, as long as that anything can be expressed as a set of nodes that produce and consume resources. I would start listing examples, but we all know where that leads.

And yes, there are a lot of things wrong with and I want to make it more robust and add error checking and all the rest, but I’ve already missed two weeks of articles fiddling with it instead of releasing it as-is. So, here’s what I propose. While I work on getting back around to exploring a few more of the visual aspects of starship design that I’d like to touch on, perhaps the more programming minded of you could suggest improvements to the calculator, and the systems minded could build specification files that result in the kind of starship content manifests that you would like to see visualized. And then, eventually, I’ll get back around to improving the calculator, and maybe we’ll make a bit faster progress together than I could working on this in spare moments snatched here and there.

I’m looking forward to reading what you come up with! Thanks for putting up with my insanity in this post. I’d really like to procgen everything, but you’re going to have to bear with me for a while. Or, you know, do it yourself.


From The Archives:

34 thoughts on “Procgen Everything

  1. Jeff says:

    I’m going to complain again about the circles in the star-field.

    Aside from being way too busy for a background, if that’s what you think a starry sky looks like you need to get your eyes checked. I mean that literally, in that seeing halos around light sources are indicative of conditions that range from minor to severe.

    1. Paul Spooner says:

      Sure, it’s not quite as good as the real thing, but I’m not going to accept blame for trying:

      1. The Wind King says:

        The thing that’s throwing me off about the the star field, is that some circles cut others off, and then cover the lower layer circles, if that makes sense…

        If not, take the average Photoshop expy, create three layers, and then make a “SOLID” circle in each layer that partially over laps with the lower layers, this will give you the effect that I’m seeing here.

        What you should be getting, is a slight fade in from the lower circles, so replace the “SOLID” circles with “OUTLINE” circles and then soft erase the most overlapped parts of the lower layer circle.

        Other than that, colours a little flat and you have waaaay too many circles, but it looks good.

      2. CloverMan-88 says:

        To be honest, your version of those flares look more like rendering errors. It would look much better if you just got rid of them. I know that it’s hard to let things you worked on go, but sometimes that’s the right way.

      3. Leeward says:

        Those are lens flares. They should only occur around extremely bright stars, and only if you want it to immediately make people think they’re looking at a picture taken by a camera.

      4. eVie says:

        If they’re meant to be diffraction patterns, then they need to overlap, not merge.

  2. evilmrhenry says:

    I’m not completely sure what’s up with the specification file format, but needing to roll your own format is almost always a bad sign. I’m not sure why you can’t use json or whatnot.

    1. Paul Spooner says:

      Thanks for pointing it out. I wasn’t aware that Json was a thing.
      Took a look, and there’s a bunch of open and close tags that I don’t want to deal with. My format is even lighter weight than Json! At the disadvantage of being less flexible.

      1. David F says:

        The benefit of using a standard format like json is that there are standard libraries to read and write them, so you and anyone else who works with these files can just use that rather than writing and debugging a parser (even a simple one). In this case, I think I would recommend YAML, which uses indentation-based blocks similar to python, and also allows comments (which json doesn’t).

        1. Linux is a Lie says:

          The comments of YAML with the brackets of JSON (for auto-formating) would be best.

        2. Retsam says:

          Personally, I recommend JSON over YAML. The YAML spec is just filled with weird edge cases and implementation differences, while JSON is dead simple. The classic example with YAML is the “norway problem”, where people try to use “no:” as an abbreviation for norway (e.g. localization files), but “no” is one of the 22 options (!) of how to write “false” or “true” according to the YAML 1.1 spec, so “no:” is interpreted as a boolean.

          IMO the only real downside of JSON is the the lack of support for comments, but plenty of JSON parsers support comments, anyway.

          Alternatively, I believe INI is a pretty sane configuration format, that supports comments.

          1. raifield says:

            I’d say YAML over JSON, only because YAML made sense to me immediately when I was learning AWS in a way that JSON, despite being somewhat similar, never did. Too many curly brackets, I guess.

      2. Linux is a Lie says:

        I mean, if you’ve just got a Python script, your “spec files” could have just been other Python files with static dicts defined in them, for the different thing to generate.

        1. Leeward says:

          I like this approach. I use it often for configuration when I don’t care about the people who are going to be modifying the files (or they’re all me).

        2. methermeneus says:

          Pretty much all common text-only spec formats are crap for any specific purpose; the only good thing they have going for them is that, because they’re widespread, there’s a million libraries to help you parse, compress, uncompress, and format them. Rolling your own isn’t a bad sign; it means you understand your specific needs better than someone making an abstract multi-purpose format who has never even talked to you or considered your project, which is perfectly reasonable.

          That said, your specific use case wouldn’t be too terrible with YAML (or the not-so-good ol’ ini format, which is pretty close to what you’ve already got), or tolerable with Json. Avoid xml at all costs (there’s pretty much no situation where that’s the best choice), and while CSV is also technically close to what you’ve got so far, it’s a terrible idea if you want to make it a human-readable/-writable format.

          But, I’m commenting as a response to Linux is a Lie because static Python dicts are probably the best and most flexible format if you’re already using Python. They’ll even let you programmatically create the starting conditions. (Once I got the hang of list comprehensions, my eyes were opened to a whole new world.)

          1. Paul Spooner says:

            Thanks for the encouragement!
            List comprehensions are, indeed, lovely (I’m using a couple in the calculator).

            I would have used static python structures with exec and print, but I wanted to make the spec format independent of the calculator implementation. So that, if at some point in the future someone wanted a different calculator implementation, they wouldn’t have to parse python structure syntax. Also, even the fairly minimal Python annotation starts being annoying when dealing with lightweight info like this. I really like being able to copy-paste resource and node requirements around without worrying about commas and parentheses.

            Yeah, the ini and cfg formats were an inspiration. The spacebar is just so much more convenient than the equals sign!

            One quirk of the spec format here is that it ignores un-named nodes, so you can “comment out” a node definition by putting the node tag on the previous line, and then the comment tag on the node name line, and all the node/resource definitions get captured by the blank node, and then discarded.

  3. AzaghalsMask says:

    After really not getting what you were talking about in the article here (still enjoying the writing style though), I went and had a look at the specification.txt . To my surprise (for somebody with extremely limited coding knowledge), it actually made sense and seemed a useful tool for various tasks. Interesting, and well done.

  4. Mark Sachs says:

    Real talk: “ProcGen Everything!” is the tar pit of hobbyist game development. Deciding you can “ProcGen Everything!” is signing up for a project so large that either you get basically nothing done but design noodling in ten years, or you build a vast rickety structure that you realize ten years in has to be torn down because you made the wrong design decisions at the very beginning. Or you’re Tarn Adams, I guess, but the Tarn Adamses are one in a billion.

    I’d recommend backing up all the way to “hey, I could procgen a cool looking spaceship if I don’t worry about the details too much” instead. You can keep design noodling, design noodling is fun, but don’t kid yourself that it’s leading anywhere.

  5. Leeward says:

    So the first thing I noticed was this:

    #some functions to use as hash keys
    def Local():return
    def ChildProduction():return
    def ChildConsumption():return
    def Net():return
    def ContainsNodes():return
    def SolveSet():return

    Where did you come up with the idea to do this? I’ve never seen anyone use functions for this purpose before. In most (non-Python) languages, this would be the domain of an enum. In Python, I usually see people just use strings.

    1. Paul Spooner says:

      I would have used strings, but the way I was storing the data at one point, there was the possibility of namespace clashes between the dictionary names, and the user-entered resource and node names. I think that possibility doesn’t exist any more. Not sure what impact it has on performance either.

      As to where the idea came from, I think it was the Python tutorial, which outlines what object types can be used as hash keys. At one point, I wrote a little Python program which had functions as hash keys, and hash values, and were called with functions and returned functions to be used in other hashes. I don’t remember what I was trying to do at the time, but I think that’s where I got comfortable with using functions as hash keys.

      1. Chad Miller says:

        I actually had a use case for something like this very recently. I started with someone else’s code that used dicts as drop tables, had string keys, then randomly selected a key to decide what monster/item/etc to spawn. I got to remove some large branching if’s by just making the keys whatever function spawned the thing and telling to call “whatever key we randomly selected”.

        1. Paul Spooner says:

          Very nice! When the option is appropriate, I like to use tuples containing coordinates (or seed values) as keys, with the processing function as the value. Then you call the function with the key, which allows you to associate multiple keys with the same function.

  6. Cubic says:

    Interesting topic! While I’m sure it’s advanced considerably since then, it made me think of books I haven’t opened for years, like The Algorithmic Beauty of Plants, which uses L-systems for procedural generation, and another one I haven’t been able to locate yet, which was about generating villas in the style of Palladio.

    (I did find what might be the original paper though, Stiny and Mitchell’s The Palladian Grammar:

    1. Paul Spooner says:

      I played with L-systems on various occasions, but have always found them somewhat disappointing. When I wrote my plant generator, I had it build them from the outside in, starting with the foliage shape and working inward.

      As far as procedural architecture goes, I’d love to write a generator for this kind of architecture at some point:
      But the more conventional architecture generation is pretty great as well. I touched on it on my last trip outward toward the infinite possibilities, and would be interested in another pass at it. The underlying patterns of all these designs have a tantalizing appearance of similarity. Perhaps what’s needed is a procedural generator GENERATOR, to generate the systems that generate … the… hmm. Right.

      1. tmtvl says:

        Sounds to me like you haven’t seen Damian Conway’s “Three Little Words” talk, which deals with tooling and how many layers of indirection you may need.

        1. Paul Spooner says:

          Very fun talk! Reminds me of Jai, making meta tools.

      2. Cubic says:

        Ho ho ho, or perhaps evolve the generator that … But that way lies madness.

  7. D-Frame says:

    I’m completely lost. Are we procedurally generating 3D geometry yet?

    1. Paul Spooner says:

      Sadly, no. I mean, yes, I have made tools to do that, but not yet for spaceships as I had set out to do in this series. But what is a spaceship really? Is it the 3D geometry itself? Or the association of the perception of the presentation of the geometry with an imagination of a fantasy of half-remembered flight? Is it a dream of being at home among the stars? Or is it the objectified journey to the stars themselves? Perhaps, in the end, the real spaceship is the friends we made along the way.

      1. D-Frame says:

        Uhh… I only just realized it says “the project stalled out” somewhere in the first half of this post. Does that refer to the 2010 project you mentioned or this one? Is this the end?

  8. Sean says:

    I love how this post is basically an extension of your calculator in that every reader is a potential node, following the specification of consuming your code, and producing refinements to your calculator. Genius!

    1. Paul Spooner says:

      Haha! Yes, I’m glad you appreciate the effort! It, um, doesn’t seem to be working very well at the moment. But that’s in keeping with the code proper, so I suppose that’s appropriate.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply

Your email address will not be published. Required fields are marked *