{"id":49094,"date":"2020-02-02T06:00:01","date_gmt":"2020-02-02T11:00:01","guid":{"rendered":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=49094"},"modified":"2020-02-02T11:03:22","modified_gmt":"2020-02-02T16:03:22","slug":"video-game-chart-party-2-chart-harder","status":"publish","type":"post","link":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=49094","title":{"rendered":"Video Game Chart Party 2: Chart Harder"},"content":{"rendered":"

Two things happened last weekCorrection: More than two things happened last week. However, I only care about two of them right now.<\/span>. First, I published that chart dump of video game data<\/a>. Second, I got another delivery. Late last night someone threw a brick through my window. Once I finished sweeping up the glass, I noticed it wasn’t actually a brick. It was a MySql database, wrapped in a note that said, “Don Data sends his regards.”<\/p>\n

This data is a lot more complete than the last batch. It has information on publishers \/ developers, which might be useful if I knew what to do with it. But whatever. Let’s make a new version of the charts from last week:<\/p>\n

<\/div>
<\/div><\/p>\n

Here I’ve limited the chart to the twenty years from 2000 to 2019, since the data outside of that range is too sparse to use. Also, this chart just shows the range between 60 (worst game ever) and 80 (best game possible).<\/p>\n

Last week some people expressed concern that my methodology was off. Specifically, what happens if Shoot Guy 4: Shoot ’em All<\/em> gets a 100% from one thousand critics, and if the slightly obscure Punch Guy 3: Knuckles of Doom<\/em> only gets one review of 0%. If those are the only two games that year, then won’t that result in an average of 99.9% for the year?<\/p>\n

Short answer: No. Long answer: <\/p>\n

I don’t have individual votes. I only have the aggregates. In this scenario, the average would be 50%. Having said that, I can’t guarantee that this information is correct or useful. Maybe it’s inaccurate. Maybe it’s accurate but I imported it wrong. Maybe I imported it right, but then I messed up the pivot table in the spreadsheet.<\/p>\n

Moreover, I don’t have a background in statistics so I’m not an expert at handling data like this. Last week people asked for the mode and the mean of these scores. I can manually make the charts that people ask for, but in the long run it seems like it would be more efficient if I just pass along what I have and let everyone see the raw data for themselves.<\/p>\n

I’ve spot-checked a few games in the list and they seem to match what I find online, but there are over 4,000 games in this list and there’s no way I could check them all.<\/p>\n

Let’s Talk About Metacritic<\/h3>\n

Every time Metacritic comes up in conversation, there’s always this back-and-forth where people speculate how it works and we try to find a signal within the critical noise. Also, stuff like this does not inspire confidence:<\/p>\n

I can't give away our secret, but we came up with the idea of using MATH on DATA. Our engineers are very clever.<\/div>
I can't give away our secret, but we came up with the idea of using MATH on DATA. Our engineers are very clever.<\/div><\/p>\n

First off, they advertise the site as a service to “help you find stuff you’ll love”. That’s… that’s not how people use the site. I’ve been in this game critic business for a long time, and I’ve never heard of anyone<\/strong> scrolling through Metacritic to find games. The site doesn’t even contain information required to make that a worthwhile thing to do. Like, if I wanted to find top-down turn-based fantasy RPGs based on the Dungeons & Dragons license, then Metacritic can’t help me. Sure, it can help you look for “action games”, but that’s like saying you need a restaurant guide to help you find “food”. If you’re looking something up, then you probably have something specific in mind<\/strong>.<\/p>\n

Sometimes people consult Metacritic to guide purchasing decisions for games they’ve already found elsewhere, but that’s it. For the most part, Metacritic is part of the post-launch conversation where we try to figure out what the public thought of it.<\/p>\n

But fine. Metacritic either doesn’t understand how people use the site, or the leadership is trying to re-position the site within the market. Whatever.<\/p>\n

The more concerning thing is that the site brags about their “proprietary” system for calculating the metascore. That’s alarming. I always assumed<\/strong> the metascore was just an average of critical reviews. Which of these things is true:<\/p>\n

    \n
  1. Metacritic is trying to sound smart by claiming that taking the average of a bunch of numbers is a “proprietary” technique.<\/li>\n
  2. Metacritic isn’t actually taking an average. Instead they’re weighting certain scores (how?) by certain reviewers (who?) to make a more accurate (according to whom?) number. If the site is massaging the numbers according to unknown criteria, then how do we know they’re not outright manipulating scores to make money? Without transparency, there can’t be any trust.<\/li>\n<\/ol>\n

    If they’re doing #1, then they’re dishonest. If they’re doing #2, then they’re being REALLY dishonest.<\/p>\n

    I’ve always wondered why Metacritic seemed like such an awkward site. Then I glanced at the ToS page to find:<\/p>\n

    <\/div>
    <\/div><\/p>\n

    That explains it. Metacritic is owned by an Old Media dinosaur. That probably explains why they think having an opaque “proprietary” system with no accountability is a selling point. 30 years ago, this would impress people. But to tech-savvy gamers, this probably triggers instant suspicion and distrust.<\/p>\n

    An average of all critic scores is data. Metacritic’s number is that data, plus or minus an unknowable random number. I really don’t see what value that random value adds to the deal.<\/p>\n

    But whatever. Here’s one more chart I pulled out of the data:<\/p>\n

    <\/div>
    <\/div><\/p>\n

    This is the average number of ratings each game received for the year in question. This isn’t how much they liked a game, this is just a measure of how often they voted<\/strong>.<\/p>\n

    It might be tempting to look at this as a measurement of the popularity of Metacritic itself. On the surface, this suggests that Metacritic peaked in 2011 and has been gradually declining in relevance since then. However, remember that my data is limited to PC releases. This downward trend from 2011 roughly coincides with the Great Indie Deluge on Steam. Let’s have another look at this chart from last week:<\/p>\n

    <\/div>
    <\/div><\/p>\n

    That’s the number of PC games on Metacritic. Instead of seeing the previous chart as a drop in the relevance of Metacritic, it might be more accurate to see it as an increase in the number of tiny games that get very little attention.<\/p>\n

    Maybe we can control for this by looking at other numbers, but I feel like I’m running into my limits as a statistician. I figure the best way to deal with this is to just crowdsource it. I know there are plenty of people with a lot more experience at this, and we’re likely to get better results if those folks can see the data for themselves. So here it is.<\/p>\n

    vg_data_02-02-2020.zip<\/a><\/p>\n

    That’s a simple comma-delimited text fileI mean, the file is inside the zip. Just be glad I didn’t stick it inside a .7z file.<\/span> with all of the 4,331 games in the database. The fields are in this order:<\/p>\n

      \n
    1. Title of game<\/li>\n
    2. The platform, which is always “PC”.<\/li>\n
    3. The release date.<\/li>\n
    4. Publisher<\/li>\n
    5. Developer<\/li>\n
    6. Metacritic’s proprietary bastardization of the aggregate critic score. (Hopefully this is something close to the average.)<\/li>\n
    7. The number of critics that submitted scores.<\/li>\n
    8. The averageWe assume.<\/span> user rating.<\/li>\n
    9. The number of user ratings.<\/li>\n<\/ol>\n

      Note that there are holes in the data. Sometimes publisher and developer fields are blank. Also, games like Total War: THREE KINGDOMS – Mandate of Heaven<\/a> don’t have any user ratings. This shows up in the list as a -1.<\/p>\n

      You could do a lot more with this data if we had numbers for Playstation \/ Xbox \/ Nintendo titles for comparison, but this is what we have for now. If you come up with some cool info, then leave a link to your results on your blog, Pastebin, imgr, Instagram, GitHub, PornHub, or wherever people upload this sort of thing these days. Leave the link in the comments belowI’m kidding about PornHub. I’m 99% sure my spam filter will eat any such links.<\/span> and we can all have a look. Maybe we’ll learn something new. Maybe we’ll just waste each other’s time and get lost in pedantic arguments about mean vs. mode. I don’t know. Assuming we learn something, I’ll create a follow-up post later.<\/p>\n

      Hopefully Don Data will have more data for us in the future.<\/p>\n

      EDIT: In the comments, Lino expressed an interest in having the genre info. So here is a dump that includes it:<\/p>\n

      vg_data_02-02-2020-wgenre.zip<\/a><\/p>\n

      I don’t know how accurate it is, and I’m willing to bet it isn’t complete, but there it is.<\/p>\n","protected":false},"excerpt":{"rendered":"

      Two things happened last weekCorrection: More than two things happened last week. However, I only care about two of them right now.. First, I published that chart dump of video game data. Second, I got another delivery. Late last night someone threw a brick through my window. Once I finished sweeping up the glass, I […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[120],"tags":[],"class_list":["post-49094","post","type-post","status-publish","format-standard","hentry","category-videogames"],"_links":{"self":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/49094","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=49094"}],"version-history":[{"count":21,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/49094\/revisions"}],"predecessor-version":[{"id":49164,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=\/wp\/v2\/posts\/49094\/revisions\/49164"}],"wp:attachment":[{"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=49094"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=49094"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shamusyoung.com\/twentysidedtale\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=49094"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}