Good Robot #33: Good Robot at EGX

By Shamus
on Sep 23, 2015
Filed under:
Good Robot

It’s been almost three months since the last update. This is unfortunate. I’d really like to post these on a weekly basis, but I couldn’t afford to take time away from working on the game for talking about the game.

This week Good Robot is headed to EGX 2015. Pyrodactyl will have a booth on the show floor and will be letting the public try the game for the first time.

On one hand, it’s nice that the public is going to see the game. On the other hand, it’s a shame that trade shows are the deafening, crowded, sensory-overloading marathons. In an ideal world, we’d get to put someone in front of the game, watch them play, and then interview them afterwards to see what worked and what didn’t.

This is how Valve does playtesting, and I think it’s the way to go, particularly if you’re unsure of your mechanics. If you’re just making a cookie-cutter shooter, then you probably don’t need to have playtesters try the game while you observe them. But if you’re doing something new – even if it’s just new to your team – then it’s invaluable to be able to make sure that the experience you envisioned is the one they’re having.

It’s not “Playtesting” without the “testing” part.

In this photo, from left-to-right: Arvind.

In this photo, from left-to-right: Arvind.

Sure, you can send out builds to playtesters, but you can’t watch them play. I’ve found that asking people to explain their impressions after-the-fact is vastly inferior to being able to observe their playing. They might tell me, “I died a lot.” But if I was watching them myself I’d be able to tell what the actual problem was:

“Hm. Players don’t know how to dodge homing missiles. In fact, they don’t seem to realize it’s possible. Maybe we need to make the early missiles slower.”

Or:

“Players keep passing up the chance for healing, even when near death. I think we need to make them more visible.”

Or:

“Some players are hiding in alcoves and playing peek-a-boo with the enemy robots. This is a slow way to play, and eventually they get bored. Maybe we need to add robots that are good at ‘digging out’ an entrenched player rather than allow them to exploit the game in a way that makes it boring to play.”

Or:

“Players always smile when they see the boss for the first time. But they get bored and restless halfway through the fight. I think the boss introduction is making a promise that the boss fight isn’t keeping.”

Spoiler: This is not a game about making friends.

Spoiler: This is not a game about making friends.

Obviously players aren’t going to tell me about not seeing health resources they didn’t know existed, or the difficulty of pulling off a move that they didn’t know was possible. They’ll just send an email back saying, “It’s too hard.” And then when six people all say that, we end up nerfing foe damage and making the player more powerful, which makes the game easier without fixing the problems that kept it from being fun.

I’m convinced there’s just no substitute for watching someone play without any direction or coaching from the developer. It’s the single most important form of feedback you can get. And testers only get one shot to play through the game for the “first time”. After that, they will be dragging their previous experience with them and skewing the results. Which means you need a steady supply of new players to observe. That’s basically unobtainable for a little indie house like us outside of a trade show.

So EGX represents our first chance to see if people play the game as we expect, or if they do something completely different.

On the other hand, these people are tired, footsore, hungry, and functionally deaf and mute. We can’t get them much in the way of instruction and it’s not reasonable to expect a detailed exit interview. The screen will be washed out with reflected light, the music and sound effects will be drowned out, and they’ll probably be standing still on hard concrete instead of sitting on a couch or chair. They’ll probably play for a minute or two and move on to the next oasis of stimulation. Any feedback we get from them will be limited to the most immediate concerns, and not to the long-term experience of playing the game over the course of half an hour.

But that’s not what trade shows are for, anyway. The main goal of the show is advertising and building some kind of buzz beyond the reach of this blog. Hopefully we can at least achieve that.

Enjoyed this post? Please share!


202020767 comments? This post wasn't even all that interesting.

From the Archives:

  1. Thomas says:

    I love Arvind’s t-shirt, it’s a great character design and the screenshots look gorgeous.

    If you find yourself with some spare time somehow at some point another Good Robot posts would be great. Although obviously not burning out and making the game are more important.

    Either way, after development has finished maybe you could do some post-mortem posts on the whole experience?

  2. Da Mage says:

    So good robot is finally getting some ‘in-the-wild’ testing with people that neither know the developers, nor anything about the game. Should be really interesting to see the follow-up post for all the unexpected actions that players took, and how you may address them.

  3. 4th Dimension says:

    So what you would want is comments + play video? Since video of somebody playing would be waaay too big, I wonder how complicated would be an implementation of a simple stat logger (how often the player is able to heal, how often he shoots, his DPM, average speed, what caused most of the deaths) or even replay recorder. A replay system (for devs, so it’s okay to be bare bones) would probably be too much but I do wonder at the difficulty.

    • Alexander The 1st says:

      There are a few minor problems with a replay system for something as placement important as a shump; the main two that come to mind are de-synched input, and screen resolution changes.

      The latter I know tends to come up when you go from playing a game widescreen to viewing it windowed, or vice versa – at least with the Touhou games, if you play the replay from one to the other, it causes the character to not be in the same place anymore. Which makes the replay less accurate of a replay.

  4. Ranneko says:

    One idea is to try to see if you can set up a pool of people who have webcams and something like Open Broadcaster Studio to record themselves playing the game. You just need a good set up guide for that aspect.

    Once they have finished playing, you can ask them to upload the videos to YouTube, share them privately with you and review the footage.

    It won’t be quite as good (or as standardised) as a local playtesting environment, but I suspect it would be cheaper and easier to set up.

    • 4th Dimension says:

      Thought about that too (check my post) but file size, knowing how to record the game etc would be necessary, and most people wouldn’t care to do that.
      What you need to do is simplify things for them so they can send the replay or stats by clicking one button.

      • Ranneko says:

        Oh definitely if you can build in a replay system then that will probably be better anyway, more lightweight and less difficult for the user.

        The downside is you then need to spend time developing a replay system, testing it, ensuring the bugs you are seeing are not due to that.

        Though recording (via provided instructions) has its own downsides as it:
        – Amplifies requirements (OBS is lightweight, but not that lightweight)
        – Can introduce its own bugs
        – Requires more technically savvy users.

        • Ranneko says:

          Additional point for anyone wanting to pursue this idea further. You can set up OBS as a portable application where the settings are stored in it’s the same folder structure as the executable. This means you could potentially do a generic configuration for your game (and possibly the webcam, unsure cannot test how generic the set up for those is right now as I have no webcam attached to the PC), package that up and provide that to your testers.

          That reduces some of the initial set up burden, obviously a certain amount of documentation and Skype calls may be needed to get everything going past that.

      • Steve C says:

        Their own video feedback system would be insane feature creep. Na. What Ranneko said is exactly what I was going to suggest (along with every 3rd comment on this post)– Get people to do their own Let’s Play with picture-in-picture webcam and upload it to a private youtube channel for Pyrodactyl’s viewing. Lots of people do those sorts of Let’s Plays so it’s got to be doable. A lot of them are broke teenagers too so it can’t be that hard or expensive. If you’ve ever done it once then you are set.

    • Joe Informatico says:

      A thought occurred to me thanks to this comment: With the growing number of indie developers, is there a market for third-party QA and playtesting? This already exists at the AAA level, but has anyone rolled out a more affordable option aimed at indie devs yet? Anyone know if devs who work through, say, PSN’s Develop or XBox’s ID@Xbox programs get access to 3rd party QA, or is it the dev’s responsibility?

    • Lazlo says:

      So what about this as an idea: advertise “Hey, the next 20 people who sign up will get Alpha access to the game, a free copy of the released game (assuming you like it), and a $10 Amazon gift card in exchange for playing the Alpha for at least an hour on Twitch.” That seems like it’d be a win for everyone. It’d cost $200 which, even for an indie, shouldn’t be a bank-breaking investment. But in return, you’d get to watch someone play your game for an hour in their most comfortable setup. People who like to play games would get to play games that no one else can play, and even if the play completely abysmally sucks, they can drown their sorrows in a knickknack from Amazon. And other people who might possibly run across the twitch channel that it’s being played on get your game subtly advertised to them, for free. Sure, it probably wouldn’t be great for a game where the main draw is the slow revelation of a narrative-heavy plot, but for something like what I *think* Good Robot is, it should be perfect.

      • mhoff12358 says:

        The problem with the $200 idea is that this sort of playtesting needs to be very iterative. You want to start doing this as early in development as possible so that the cost of changing things is lower. But at the same time, you need to test every new feature or attempt to fix a fault you found in the last round of testing. So it would be maybe $200, times the number of testing rounds you need to do, plus the loss of revenue when you release the game, as most people looking to do this sort of thing probably already showed interest and were somewhat likely to buy the game. So I could easily see this ballooning into costing thousands of dollars, if not directly then indirectly.

        That said, there’s got to be a way to get this sort of thing to work out. And it sounds like it could be valuable enough to be worth some amount of money to the developer.

    • Zak McKracken says:

      why youtube? If the video fits through your internet connection, you could just upload it onto whatever bitbucket, dropbox, public Spideroak share, or whatever ftp upload area Shamus might create on his server as well, and there it is.

      • Ranneko says:

        All of those are viable options.

        I suggested YouTube only because:
        a) Everyone is going to be familiar with it
        b) It is pretty easy to upload videos and the user gets feedback if the file is broken

        But any method of sharing large files could work.

    • Mike says:

      Either that, or have the playtest version generate a log of the random seed that the game starts with, plus the player input state at every point where the inputs are checked (eg: up key, right key and left mouse button are pressed, mouse is at x,y position, etc). Each log entry should be about 10-20 bytes (maybe less, depending on the number of keys tracked), meaning an hour of “video” at 60fps would be about 2-4mb. Replay would be as simple as swapping the “inputs checking” function for a “log replay” function.

      This assumes that the game is state-based (check inputs, calculate updated state, render frame, repeat) rather than interrupt-based.

    • Groboclown says:

      The idealized way to set this up is with a User experience lab, where all the variables are controlled, which gives you the synchronizing the user face capture and screen, and thus captures eye tracking with the screen elements. Here, we’re talking more about a Amazon Mechanical Turk style data gathering.

  5. Why not just ask new playtesters to stream the game and a feed of them on a webcam over twitch or something?

    • Daemian Lucifer says:

      Currently,the problem with that is that there are plenty of bugs that can happen due to recording/streaming software,as can be seen in numerous spoiler warnings.So youd need to have someone who knows what are the common problems with doing so.Yet the people who know it the best,currently are the ones who stream (semi) professionally,and most of them arent that interested in being play testers.The profession of play test streamers will arise sooner or later,no doubt,but currently you wont find many of them.

    • Duneyrr says:

      I would totally be willing to do this :D
      I wouldn’t be streaming it, but I could record both the game and a webcam

  6. MichaelGC says:

    Brum’s not far from me – it’ll take some sorting out, but I’ll try and toddle along! Although…

    Pros
    Good Robot

    Cons
    Deafening
    Crowded
    Sensory overload
    Marathon
    Hard concrete

    Yep, still worth it! :D

    • Zak McKracken says:

      Gaah! And I’ve just been told that I’m not allowed to plan anything this weekend … Me sad again!

      (goes and checks ticket prices anyway):
      Oh, there are only some regular day passes for Friday and Sunday left, everything else has sold out already…

      => Whoever wants to see Good Robot in action, be quick!

  7. WWWebb says:

    For business software, we can use something like Morae to take video of the app and user. Then you frustrate the developers by making them watch people clicking everywhere to find that command that you put RIGHT THERE in an eye-catching toolbar button (and in the menu, and in the right-click dialog, and live-linked in the tutorial, and mapped to a hotkey for the pros).

    For a game, you’d probably want one of the fancy implementations that includes eye tracking so you can actually tell whether or not they noticed those health resources or even that their health was getting low.

    • TMC_Sherpa says:

      Business software… The best part about writing it is no one can tell you what they want while you’re designing the dang thing but everyone remembers once you get to deployment.

      PS
      I don’t social media so this is all I got. Happy Birfday Mumblo!

      • WWWebb says:

        Dangerous discovery for a small business thinking about improving their UI:

        Finding and hiring a good UI designer/programmer is probably more expensive than hiring a new college grad to travel around the country full time doing on-site training.

        • TMC_Sherpa says:

          In a previous cycle I did Crestron programming (basically A/V automation).
          I worked on a couple jobs for Chicago city hall and no matter what other room we were doing we got dragged back to a “finished” job to add a button, then take it off, then put it back. That was fun.
          I also was involved on a couple different campuses for a lawyering college. On campus two the touch panels got redesigned to have a better layout which was rejected because they were worried that the staff (who worked at both) would get confused. I didn’t have access to the original layout so they faxed pictures of the old style pages. I was feeling a little cheeky so as a test I used the old layout with the new iconography because the old one was kinda pants. They liked the new panels enough they wanted to contract us to go back and change campus one. I don’t know if that ever happened.

  8. Christopher says:

    God Robot.

    Even if there’s no time to blog about the development, if would be interesting to just watch Josh play a session of it or something. Honestly, anything is better than seeing nothing. Looks like things have changed a lot since the last trailer I saw on Greenlight.

  9. Benjamin Hilton says:

    I think there should be an official mod of Rutskarn voicing the inner monologue of the Good Robot.

    But that’s just me.

  10. Galad says:

    How legally or otherwise complicated / in the realm of possibility would it be to put up an ad wherever is most convenient about looking for a one-time playtesters (say 1 hour of playing, and half an hour of feedback) for X amount of people?

    • Alexander The 1st says:

      Having done user testing at university, the major blocks would be in the privacy policy for how to disseminate the data – figuring out who can see the stream itself, and/or who can see the results recorded from the stream.

      Not including any NDA’s involved.

  11. Blue_Pie_Ninja says:

    Any chance of heading to PAX Aus? Would love to try out the game for you there, looks great from all the screenshots, and from whatever videos I have seen of it in past posts. :)

    • Lachlan the Mad says:

      I wouldn’t generally expect indie devs to come all the way to Aussieland, because flying here from the USA or England or other countries full of game devs is super expensive, but Arvind is from India which is relatively close to here!

  12. Daemian Lucifer says:

    You know…You seem to be on the verge of “play testers renting stream” business,an office where a bunch of people have the only job of trying out new games for the first time and streaming it/sending the recordings back to the publishers.Sooner or later,this is gonna happen.

    • Ranneko says:

      I am pretty sure that outsourced playtesting services do exist, at least within some kind of publisher structure.

      I remember Brad Muir from Double Fine talking about it on one of the post-backer release Massive Chalice teamstreams. Comparing the experience of being able to watch people streaming the game to the playtest information they got from working with publishers.

  13. topazwolf says:

    Just going to put this out there, if you want players to test the game, record the video of themselves playing, and send said video to you, I would totally do that. Shoot em ups aren’t quite my thing, but if it would be of service to you I would totally like to help. And I’m sure that you have plenty of readers with the same level of technical knowledge and motivation to do that for you.

    EDIT: Forgot to mention that you could probably get your readers to do so for just a thank you.

  14. boz says:

    I may be the only person with this problem but for a moment I thought game was named “God Robot” instead of “Good Robot” thanks to the logo.

  15. newplan says:

    It sounds like what you want is what a twitch stream generates – footage of the game with a camera on the player’s face as he experiences it.

    Depending on the game being streamed some streamers look for entertaining games to play in downtime caused for whatever reason (unwinding after completion of a session of the primary, between match / round / whatever downtime).

  16. tzeneth says:

    I don’t think this gets said enough but Shamus, you are good at coming up with good captions for the pictures you use on your blog. Most of the time I get at least a chuckle and I’m glad you put in the extra time to add them.

    From all your readers who want a laugh,
    Thank you.

    • Eskel says:

      I agree that the captions are great. That said I personally prefer static caption under image rather than a title tooltip. Tooltip s incovenient to read on mobile devices and I tend to forget it’s there and miss the fun :-)

  17. Zak McKracken says:

    Quote:
    They’ll just send an email back saying, “It’s too hard.” And then when six people all say that, we end up nerfing foe damage and making the player more powerful…

    no, no, no! The only appropriate response to this complaint is “you’re not good enough!” Unless you learn that, how will you ever make a truly great game?
    (also, how do I properly add a quoted box? “quote” tags are ignored

  18. Nidokoenig says:

    Regarding standing on concrete, as someone who once worked a job that involved a lot of standing on concrete, I can tell you that a little foam kneeling pad to stand on will make all the difference. Either have two side by side or one horizontal to rest your ankles on. They’re usually available from hardware stores or garden centres.

  19. Decius says:

    This caused me to have a strange idea. What if there was a convention that was literally nothing but playtesting?

    Or a program that recorded input, screen, and camera of the player’s face? Is it unreasonable to send test builds out with the expectation that people will record a stream and send it back to you?

    In both of those scenarios, you will get only a harder audience than the general one.

  20. Diego says:

    This might be hard to implement, but how about you program your game to record relevant data into a file? the playtester attaches this file with his report. This file could contain stuff like the level layout, a heatmap for deaths, relevant commands (at this position he used the bomb, for example) and stuff like that. Depending on the size you’re willing to make it could even provide enough information to reproduce his play session completely on your pc.

    • Lanthanide says:

      Yes, some kind of replay-recording functionality for the game could be useful.

      However, you’re only getting 1/2 or less of the actual feedback. You see the player do something stupid, then some other stupid thing, but you have no idea why. If you were actually watching them as they play it, you might see that they’re looking frustrated, or maybe they’re looking bored. That extra context is the really valuable part.

  21. Frank says:

    While by no means perfect, User Testing is not a bad start to get some videos of people testing your software.

  22. Anthony says:

    Good! Mr Shamu, I wanted to ask. This line from the Steam Description:

    “Are you a Good Robot? Let me put it this way:
    Do bad robots come with a 14-day money back guarantee*?”

    Are you really going to include that or is that a joke? Because if it is a joke, I think its best to remove it. You do not want to deal with entitlement issues or open the door to a complete jerk to ruin your launch.

  23. Warclam says:

    Am I the only one who thinks of this when reading about Good Robot?

  24. You can avoid the need to record someone playing if you use experienced testers.

    Someone with QA or reviewer experience would be able to point out things like missiles to fast or slow or similar or missing health pickups or explain why a boss fight gets boring halfway through.

    Are there Crowdtesting sites out there though?

  25. Zak McKracken says:

    So? how did it go?

  26. Fists says:

    If you’re ok with the game being shown publicly in it’s current build you could approach up-and-coming youtubers and ask them to do a blind playthrough for you in exchange for a link-back from the blog. They’ve already got the Rube Goldberg recording system set up, you get some raw playtesting video and they get a bit of exposure.

  27. This exchange is a lot like that nasty-ass Honey Badger video on YouTube.

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *

*
*

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>