Retrogaming and AI – A Surprising Tangent

By Paige Francis Posted Monday Aug 28, 2023

Filed under: Epilogue, Paige Writes 21 comments

When I proposed this week’s article, I believed it would be a break from retrogaming content. Not that retrogaming is the only thing I ever intend to write about, or ever DO write about, but I feel my retrogaming setups are still in a state of “incompleteness.” Some recent AI news has been on my mind, though; and considering I’m a Gen-X’er, my “get off my message board!” instinct was being triggered. However, it wasn’t until I started fleshing out what I was going to talk about that I realized retrogaming was actually an important part of the story, or at least the news item I wanted to talk about.

Like Dogbert, I am extremely talented in getting to the point, so sometimes it’s a challenge to create an entire written piece on what to me are simple concepts or arguments. What is holding me back from doing just that, right now, is that just flat-out stating the argument would automatically put some people on their guard and lead to conflict, and I neither want that outcome nor consider it useful. Discussion = good. Walls = bad.

What am I talking about? The nature of Artificial Intelligence. Specifically, what *IS* artificial intelligence vs. what *ISN’T* artificial intelligence.

I am technically a Biologist of minimal training and experience and a programmer of limited talent, and my life-long computer experience is Systems and Networks Administration, with a little Database managing thrown in. I’ve messed with computers and OS’s down to the soldering level (I’m not good at it) and generally fit Jerry Pournelle’s “ultimate end-user” appellation. However, my allegiance is to philosophical understanding of intelligence. Anyone who has spent time arguing in this sphere knows that Douglas Adams nailed his prediction that the ultimate split in deep thinking would be not religious/non-believer or moral/ammoral, but between engineers and philosophers. In layman’s terms, is a duck that which quacks like a duck, walks like duck, swims like a duck, eats, poops, and procreates like a duck? Or is all that merely appearance and labels, and what makes a duck is actually ineffable? To put it a third way: if a program *ACTS* in a manner we expect Artificial Intelligence to act, by what criteria can we say it is NOT actually AI? That would be opposed to the philosophical argument that there is, as it is most commonly described, an element of inspiration, non-intuitive leaps, or imagination to true intelligence; things that cannot be replicated by algorithms.

“AI” is an overused term in 2023. It has been for a while, of course; as well as terms like “robot,” “android,” and “cyborg.” The broad sci-fi conceptualization of a robot (we will use Asimov here) is of an ambulatory, autonomous, humanoid-like construction with complex-enough circuitry (later, programming) that the construct can “make a decision” based on input sensors that would approximate the decision a human would make given the same stimuli. Asimov, as most early writers of robots did, conceived of this process being accomplished through complex, miniaturized circuitry: logic facilitated by capacitance and resistance. In programming terms, you assign variables to the inputs, you use complex, massive matrices to weight and compare the variables in a routine, and the final value pops out of that sub-routine and gets thrown in to the master decision making process as one of thousands, if not millions of variables, each weighted by a larger process. “Thinking” in this manner is achieved through algorithms rather than impedance.

It is this “thinking” that we’re talking about. We can save the debate over whether battlebots are actually robots or just homemade R/C vehicles for another day. The point is; however, that Asimov clearly delineated that his robotic brains, “positronic” brains, used sorting algorithms…incredibly complex, circuitry-based sorting algorithms. Asimov grew increasingly circumspect about whether this should be considered artificial intelligence, but it is certainly clear he meant to construe that idea originally. The positronic brain (and at least one competing brain of a different design) that thought best, was the brain with the superlative quantity and complexity of circuits. Or, in our more modern thinking, algorithms. It was a construct such as this that was capable of devising a plan to save humans from themselves, and enact that plan even against their will, since it was for the survival of mankind overall. But only to an extent. That brain, even supplied with the rarest of positronic “bugs:” the ability to read minds, could only account for certain outcomes. The robot in possession of the brain eventually required the aid of a human. Therein, of course, you could say was the spark of inspiration. But considering that “inspiration” was the kernel of the idea that one could quantify history in such a way that THE FUTURE COULD BE PREDICTED WITH MATH, I’m not entirely sure that counts in the argument, and it all falls down eventually, more-or-less. And that’s probably a lot more talk about Robot, Empire, and Foundation than anyone came here for!

However, I think this probably frames the issue very nicely; and that brings us to the inspiration for these week’s diatribe. YouTuber and programmer “FoxMaster” has created an “AI ‘bot” that plays Tomb Raider 1. Nothing unusual there; several programmers run YouTube channels built around creating “AI ‘bots” that play games. Code Bullet, one of my favorite YouTubers, has used pre-made neural-net AI routines to master many casual games such as Tetris, Snake, and Hill Climb; jumping platformers, and bipedal walking simulators. FoxMaster didn’t have to teach Lara to walk of course; the basics of ‘bot-controlled-Lara are actually pretty simple. An algorithm can actually make precise measurements of the game world, facilitating accurate jumping and platforming. In the three published gameplay videos and dozens of shorts, you can see this capability utilized: the polished routine never moves the character model more than necessary, and never runs when walking is adequate. In fact, FoxMaster has implemented programming routines that de-emphasize the ‘bot’s ability to play the game too well, by prioritizing more important elements.

Because Tomb Raider is more than a platforming arcade game, the ‘bot had to be capable of more complex behavior. Resources, namely in the forms of ammunition and med kits, are limited; so management of these supplies is a priority. And of course, puzzle-solving can only rarely be bypassed or brute-forced…some puzzles must be solved. FoxMaster eschewed including puzzle-solving-specific routines…he favored a solution that encompassed the entire game. That is, the ‘bot must fundamentally interact with the environment as a whole, starting from no programming other than knowing what the character herself, Lara Croft, would know; and view the game-world through Lara’s eyes. Again, in layman’s terms, as website put it, “AI generates self-awareness, ‘Tomb Raider” Laura awakens! The video game revolution is here”

Indeed, FoxMaster titles the videos in the series “Self-aware Lara Croft.” And to be honest, arguing whether the ‘bot is “truly” self-aware is a separate argument…for the purposes of the ‘bot, it really is self-aware. FoxMaster had a sub-routine “watch” recorded video of the Tomb Raider 1 tutorial level being played; recording its own data points that allowed it to identify the Lara Croft model separate from the environment in *almost* all camera positions and zoom levels. Building on that, the ‘bot can “see” only the textures displayed on the screen at the moment. Important, less common textures are identified and stored in “long-term” memory, along with proper interactions. “Grass,” “plant,” and “rock” textures cycle through short-term memory, as the actual nature of the texture is largely unimportant. Priorities and goals are established by “viewing” an increased-contrast screen shot and identifying high-contrast spots that indicate depth of space or the presence of light; the two things that indicate additional areas that can’t be “seen” or access to another room. Location within the game-world is established by parallax; precise movement and location is informed by identifying the repeating edges of textures along with the “depth perception” of parallax and contrast changes.

But as fascinating as all that is, those are not the parts that have really grabbed the headlines. In pursuing the idea of a “self-aware” Lara Croft, FoxMaster created a ChapGPT routine that utilizes a database of Lara Croft’s “traits,” extracts of Lara’s voice from the game and a cloned voice (I’m sure you’ve all seen some videos about voice cloning recently), and Google searches to create Lara Croft’s spoken observations about what is happening around her and what she’s looking at. This creates the “illusion” that Lara Croft is thinking about and responding to the game-world; a key factor that most people would consider real artificial intelligence. However, even FoxMaster clearly explains that all he is doing, essentially, is 1) a reverse image search, 2) a Google search for information based on the results of (1), 3) curating and sorting of the results into a list of keywords weighted by Lara’s “personality database”, which leads to 4) a ChatGPT discussion prompt to create a Lara-esq comment based on the personality inputs and search information keywords and the context of the current or recent action in-game, output in Lara’s captured and cloned voice, and then 5) fed back to the Lara ‘bot to “say.” Oh, and he also added capturing “sound cues” to simulate Lara’s hearing, and enabled that as input for the personality/ChatGPT script. As you can imagine, this process is NOT instantaneous…FoxMaster states that the ‘bot is set to “pause” the game every time a search goes out, then restart the game when the results return. This usually takes about 4 minutes. When the video is edited without the pauses, of course, it certainly LOOKS like a self-aware Lara Croft is fully interacting with the environment. And from a certain point of view, she is. Knowing that the game has to pause each time Lara needs to simulate a “thought” doesn’t negate the collating and searching process that creates what, in all appearances, is an actual, realistic “thought.”

The cherry on top, to me, though, as that FoxMaster feeds Lara’s “thoughts” back in to the ‘bot routine. This expands Lara’s commentary, as she can actually comment on what she’s thinking about. Lara “knows” what she has recently said, and can compare that to the current context. This occasionally produces moments of apparent insight, although the disconnect from actual thinking becomes apparent when Lara doesn’t act on what she just “thought,” even when the observer understands she just considered the solution to a puzzle or a better way to deal with a situation. And that really brings us to the flip side. While this TRULY IS a fabulous and meaningful creation by FoxMaster that, as he himself says, only illustrates a hint of what could be possible with more work; the cracks that show the counter-arguments are readily apparent. And again, FoxMaster discusses the reality behind the polished videos readily.

We just talked about the huge pauses in the game while Lara’s “thoughts” are generated by internet searches and a ChatGPT routine. That’s more of a “suspension of disbelief” breaker to most observers, I feel. However, even in the final videos, you see that pathfinding is still mostly random and accidental rather then methodical. This could likely be improved with better long-term memory weighting and recall, and I suspect that’s an area FoxMaster is focusing a lot of attention on. The more problematic area related to these tools is complex puzzle solving. FoxMaster doesn’t go into too much detail in the second and third videos, but in the first he relates that the Lara ‘bot is just about hopeless at dealing with more than one switch at a time, and complex puzzles like the early gear-search puzzle or having to solve puzzles in order were at that point impossible. I will point out that the Lara ‘bot solves the gear puzzle in the third video, but there is clearly a LOT of editing going on. I suspect this is mostly for time, as any of these puzzles could be solved randomly with time and long-term memory…a LOT of time. And FoxMaster has related that for every 20 minutes of finished gameplay footage, the ‘bot actually plays the game for hours, if not days.

The hyper-precision that the ‘bot is inherently capable of has been documented in a number of shorts. They can be impressive such as the repeated instances of Lara’s cool-headedness,

Script: 'Run Lara!' 'No.' 'Why won't you run?' 'I don't actually run everywhere, you know. When I want to be careful, I walk.'
Script: 'Run Lara!' 'No.' 'Why won't you run?' 'I don't actually run everywhere, you know. When I want to be careful, I walk.'

or frustrating, as in this image posted by FoxMaster earlier today:

The significance of this image bears explaining: The ChatGPT personality is capable of commenting on Lara’s inability to find a way beyond her current area, based on cooldown timers. In this particular case, the Lara ‘bot informed the viewer “There is a lever but I can’t reach it.” The ‘bot’s routine, of course, would prioritize reaching a lever it can see on the current level. However, in the entrance area of the Hidden Valley, there should not be any visible lever. Except there is, by mistake. Take a look at that brown blob a little down from the upper-right corner of the image. That’s actually a lever texture from a room in a completely different area of the level, and it should be blocked from view. This is a mistake, but the Lara ‘bot could see it immediately, prioritized it, and was able to to see and calculate that she couldn’t physically reach it…there are no connecting textures which would indicate a path. The problem here is twofold, despite the real in-game mistake. Most players, especially in an early playthrough, wouldn’t notice the texture and likely wouldn’t be able to identify it even if they did. Additionally, in the eventuality they did both, most would deduce that it was an artifact of some kind irrelevant to the current situation. The player is clearly not “meant” to be able to travel directly to that switch. The Lara ‘bot, as complex and capable as it is at this point, was not able to deduce that the texture was unimportant…the weighting of the algorithms that make it possible for the ‘bot to pathfind in the first place also kept it fixated on that switch as the way forward. The most recent full video has the Lara ‘bot moving past this section of the Hidden Valley, but FoxMaster didn’t document what he had to do to solve this hiccup. Hopefully he will do another debriefing video discussing the state of the algorithms.

To me, understanding the programming that creates the actual gameplay routines indicates the lack of actual intelligence. But more fascinating is the personality and commenting system. That brings us back to the argument about artificial intelligence versus “simulated” artificial intelligence. The tell, here, IMO, is that the Lara ‘bot can, for all intents and purposes, intelligently “comment” about what is going on around her. Yes, even with the four-minute pauses; the searches are emulating the memory recall of a lifetime of education and experiences. To that end one could argue that these routines are a simulation that produce a result effectively indistinguishable from actual memory recall. The lack of connection between what SHOULD be pertinent thoughts and subsequent action may eventually be a correctable problem…but a complex one. Because, of course, FoxMaster isn’t simulating real life, he’s simulating “Lara Croft playing Tomb Raider 1.” Actions still have to be constrained by what’s possible in the game world, and what can be accomplished by the game engine. But, if THAT egg can be cracked, we really are sitting right in the middle of the argument.

If the Lara ‘bot ACTS like Lara Croft playing Tomb Raider 1 in every possible way, how is that distinguishable from an actual Lara Croft AI playing Tomb Raider 1? Is it really as simple as turning memory derived from a source OUTSIDE the game into workable action within that game? Is that simulated inspiration?

Go watch FoxMaster’s videos on YouTube! Have fun in the comments!

See you next week!


From The Archives:

21 thoughts on “Retrogaming and AI – A Surprising Tangent

  1. Pythor says:

    Definitely should have included at least a link to the first video, which is this one.

    1. Thank you for posting the link. Not including it was an oversight. The original intention was to embed the first YouTube video, but we’re having trouble with the code for embedding at the moment.

  2. Teltnuag says:

    It sounds like you’ve stumbled yourself onto the classic Chinese room thought experiment, one of the main arguments against the existence of strong AI.

    1. Dues says:

      I’m not sure the Chinese room is an argument against a strong AI. It seems like a strong argument that you can’t create consciousness using a big lookup table. But I feel like even that proves too much.

      In the unlikely event that everything we know about neurology is overturned and scientists prove that your brain uses a big lookup table, would you conclude that you weren’t conscious? Or just that we were more confused about consciousness than we thought?

      I make the distinction between during AI and conscious AI, because a AI can already do things that humans can’t without consciousness. It remains to be seen if they require consciousness to surpass humans in all fields.

      1. Daimbert says:

        The Chinese Room is actually more limited than that, as it sets out to prove that you can’t get understanding from symbolic processing but need semantic processing as well. In short, you can’t just match a symbol and then take an action based on that matching, but you need to actually UNDERSTAND what that symbol means. The example here is, I think, making that mistake as well as even though what Lara says SOUNDS like it reflects an understanding of the situation we can see when we look beneath the hood and test it out more that it really doesn’t.

        I make the distinction between during AI and conscious AI, because a AI can already do things that humans can’t without consciousness. It remains to be seen if they require consciousness to surpass humans in all fields.

        I wrote an essay for a Philosophy of Mind course once (that I’ve posted on my blog) that argued that we can act based merely on being AWARE of something but we wouldn’t need to be CONSCIOUS it, since to me being conscious is pretty much determined by having phenomenal experiences of the thing. Existing AI can do a lot with what would be at best simple awareness, but we don’t have any evidence of phenomenal experience yet. But on top of that, we still have the question of meaning. Neural net AIs pretty clearly don’t have meaning, since you can in theory apply them to completely different domains and if the inputs happen to align properly they’d still work, and for real intelligence — let alone consciousness — you are going to have to have meaning.

        1. This is more the question I was getting at. If the Google searches could be translated to meaningful action within the game, you could at least argue a simulation of applying meaning.

        2. Fizban says:

          Mentioning meaning has given me the last nudge to form the thought I’d had on the post/topic:

          So yes, obviously the bot doesn’t understand meaning, but what is meaning? The Larabot clearly does more than just the language presentation of ChatGPT, adding some google searches to do what many articles seem to think the chat program already does (find and use new information), and apparently does have some ability to store some recent information, but of course humans do more than that. You basically need a bunch of feedback loops, constantly taking in new information and building your own (faster) database which can then be referenced and compared to new information, able to go back and forth “asking” new questions and refining your own tables. Except those tables might be better described as clusters, each around one meaning or concept, which have various links to all the other concepts.

          In short, I think (with my absolute layman’s bare understanding) that these emerging tools might be on the right track, the whole neural net concept behind them I’m pretty sure is supposed to be working through linked clusters? But they’re exponentially smaller in scope than are necessary to create a sufficiently humanlike intelligence: ChatGPT is really good at writing-word concepts in the specific form of what words are likely appear next to each other based on what words appeared next to each other, and can thus string together words pretty well based on input word strings, but that’s it. Whereas for a human, each “word” links to dozens of higher and lower level thought/meaning levels (and memory sequences!) that have nothing do do with stringing words together.

          And that’s where the processing and speed problems come in, ’cause if you can’t build a brain, you can’t run a program that thinks like a brain. It looks like with a whole bunch of modern computers and time we can now emulate one small slice of human thought. Granted, language is one of the more useful places to start for making a product and can be used to describe almost everything else so it’s pretty foundational, but that’s it.

          1. Daimbert says:

            Meaning is a really, really complicated and controversial notion, but one of the reasons that I say that neural nets clearly don’t have meaning no matter how many clusters you link is that the system doesn’t have a representation of the elements they are working on and so are always working blindly. If you look at an old-style inference engine, for example, what it does is store logical statements and various premises, so you get something like this:

            1) if a, then b.
            2) a
            3) b (derived from 1 and 2).

            You can look at how it represents things to itself and see the elements that the system is supposed to understand. You can also easily write a handler to get it to explain its thinking and how it came to the conclusion it did. You can argue over whether this is real understanding or whether this is just more direct symbolic processing, but at least the system critically depends on it, of its own according, representing the data in the right way and in a way that’s meaningful.

            We don’t have that for neural nets. There is nothing in a neural net that represents, for example, any specific proposition or fact about the domain (usually, the world). You can’t find anything like that in the neural net structure, and getting it to explain its “reasoning” is nearly impossible since it doesn’t have ANY kind of representations for that reasoning. You give it an input and, if it’s trained properly and the connections are set right, it gives you an answer. That’s why I say that neural nets can be hooked up to completely different domains and still work: if they happen to have a set of connections that work for the inputs given they’ll still produce the right output. I commented on a post of mine once — that I can’t be bothered to look up at the moment — that in theory you could take a neural net that you had trained to play chess and turn it into one that does calculus and it might actually perform BETTER depending on how the connections are, which is not the case for us, inference engines, or anything that we’d expect to understand what it was doing. So neural nets, to me, are moving so far away from meaning that no matter how many things you hook up to them you still aren’t going to get meaning.

            Also, I think we have to be careful about conflating the hardware and the algorithm. Looking at how the hardware works isn’t usually the best way to figure out what the algorithm is. So we may not need to figure out how to build a brain to be able to figure out what the brain does when it is being intelligent/conscious, and in general both algorithms and the Philosophy of Mind theory of Functionalism say that once we figure it out it should be able to run on any hardware, and in fact focusing on the brain has potential issues for AI since AIs won’t ever really have brains, but focusing too much on what the brain does runs the risk of us determining that you need the biology for that, and so could NEVER have Artificial Intelligence/Consciousness because you need the right sort of biological organism for that.

  3. Octal says:

    That’s pretty interesting!

  4. Shu says:

    What a fascinating read. Looks like I have a bunch of new videos to watch.

  5. Syal says:

    It makes me happy to think that LP Bot might actually be real soon. Lara’s “perhaps a puma” is right in line with LP Bot’s “without error”.

  6. RCN says:

    This is cool, I wonder what other games this could be an interesting experience with.

    Duke Nukem? Eh, Duke is just a jerk. Didn’t age well.

    Rayman? What’s even Rayman’s actual personality?

    Mario? Ditto to Rayman.

    Half-life? It’s been done. And frankly, without Inside Gordon’s Mind there would be no personality to speak of from Gordon Freeman.

    Carmageddon? This… might actually work?

    1. I think most of the more-pure platformers, it’s just a matter of playing the game as best you can. Duke Nukem played by an AI ‘bot with a personality filter that could affect how they played the game…that could be VERY interesting. Would he eschew healing because ‘real men don’t need it?’ Would he tend to get stuck on levels any time he see’s strippers? (TBF, it’s been years since I played and I only really remember the tiniest bits of the game.)

      1. RCN says:

        Of course, the inherent problem is that the most obscure the character the less the “AI” has to work with.

        Which is the reason Carmageddon is a long shot, unfortunately.

    2. Syal says:

      Sonic has already had it.

    3. tmtvl says:

      American McGee’s Alice, obviously. The AI’s quirks would even work well with the setting.

      1. Ooh, I didn’t think of that. My nomination was Resident Evil HD Remaster.

  7. Daimbert says:

    Commenting on the “engineering vs philosophy” comments, I actually am both, having done a Computer Science undergrad where my Honours Project was in AI and working as a software designer for over 25 years, but also having gotten a Masters Degree in Philosophy where Philosophy of Mind was one of my focuses. And for me I do note that the engineering side wants the system to do things whereas in Philosophy of Mind the focus is more on making it work like a human, but even then I think we focus too much on success. If you want to build an AI that really thinks like a human, in my opinion we really need to pay attention to where we, ourselves, screw up, because those screw ups are almost certainly artifacts of the process we’re using. There are many, many ways to get to the right answer, but far fewer to get to a specific wrong answer when you usually get it right, and an AI that screws up the way we screw up is far closer to being one that thinks like us than one that always gets the right answer, especially in those cases where WE don’t.

  8. Kazeite says:

    Oh yeah! I’ve seen this one!
    It’s fascinating how “Lara” can comment on music cues and how the can clip through the dead enemies :)

  9. Paulo Marques says:

    Nothing interesting to say, but since no one said it, and some people have the good sense to not look at YT comments, that actually isn’t a bot. It’s very entertainingly made and vaguely believable, but there’s a few things where the techniques don’t work like that, but also where the directing doesn’t fit the description – the most evident is how Lara is commenting on information she isn’t actually being fed.
    Still, an interesting piece of performative art.

    1. There has been a lot of confusion about the videos. FoxMaster himself puts a title card at the beginning of every video that the Lara ‘bot is NOT AI. I’m starting to see videos pop up on YouTube specifically about “this isn’t AI!” Well, duh. That’s the whole point of what I was writing. It’s NOT AI, most of the things we call “AI” are NOT AI. But there is a long-lasting argument over the line between AI and “simulated” AI. The most entertaining part of FoxMaster’s videos is the ChatGPT-generated commentary…that’s what is getting people’s attention. FoxMaster has documented in several videos how the prompts fed to ChatGPT are generated through gameplay. Control input is simple. Control of the Lara character can be realized using any number of gameplaying “AI” routines. The emergent behavior that results from these scripts has been documented in FoxMaster’s “bloopers and outtakes” videos…after all, these scripts are still fundamentally trial-and-error. Puzzle-solving is the concept that most challenges the game-playing code. I’ll be talking tonight about a more complex puzzle that FoxMaster documented recently.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply

Your email address will not be published.