There’s also a bit of information that can come from any of these sources: The url for the game’s official website might be handy, and we also need to get the publisher, developer, and release date from one of these places.<\/p>\n
Of the three sites, it seems like Metacritic is the best one to start with. It has games listed by platform, which is necessary in a structural sense. For the purposes of our database, it’s possible for the same game to have vastly different information depending on platform. For example, maybe a game is released on the Playstation 3 in 2010 by Beloved Developer, but then a year later it gets ported to the PC by Shovelware Games. Metacritic is the only place where we can get this information reliably. Steam obviously isn’t going to have non-PC data, and Wikipedia entries aren’t guaranteed to have all the per-platform data in an easy-to capture locationIt might be in the info box on the right, or it might be buried in the article text (good luck capturing THAT) or it might not be listed at all.<\/span>.\u00a0<\/p>\n Metacritic even has a handy index page that you can go through:<\/p>\n You can see that I can choose a platform by changing the bit where it says So the plan is:<\/p>\n Start at page 0. This page will return a list of 50 games. On this index page, all we get is the title, publication date, and a link to the full metacritic page for the game. We grab the names and the URLs, then move onto the next page. If we find a page with no games, then we’ve run past the end of the list and it’s time to stop.\u00a0<\/p>\n Once we’re done, we have the base info for these games: Title, platformJust PC-only for this first run.<\/span>, and release date. The latter is important to avoid confusion over same-name sequels like Tomb Raider, Doom, Sim City, etc. So to uniquely identify a game in our database we’ll need all 3 pieces of info.\u00a0<\/p>\n Once we read the whole index, we can go back and load the Metacritic page for each game and get the more detailed info: Developer, publisher, critic score, user score.<\/p>\n That’s the plan, anyway. Now all we need to do is start scraping.<\/p>\n The thing is, it’s very important to make sure your bot is well-behaved. If you’re careless, it’s possible to – completely by accident – launch a denial-of-service attack on a website by simply building an ill-behaved bot.\u00a0<\/p>\n It’s not that my humble residential connection is any threat to the mighty Metacritic, but that’s no reason to be careless.\u00a0<\/p>\n Now, technically<\/b> you’re supposed to have your bot read the robots.txt file. That’s a plain text file that tells robots how to behave. It tells the bot how often it’s allowed to make requests, and it tells the bot where it is and isn’t allowed to go.<\/p>\n Normally, I’d be a Good Citizen and follow the rules. My problem is that a full implementation of robots.txt compliance can be fairly complicated:<\/p>\n 1) Read and store all of the directories and files you’re not allowed to scrape. It’s not hard, but it would be time-consuming and it would end up making the project a lot bigger. In a practical sense, it would be better to MANUALLY view the robots.txt file, see what it says, and then scrap the entire project if it’s too restrictive. <\/p>\n Making a well-behaved bot in this case would mean writing tons of code that, if it ever got used, would mean the entire project was pointless and I shouldn’t have bothered with any of it. <\/p>\n But most importantly, if Metacritic says bots aren’t allowed to crawl for publicly-available information, I’m going to do it anyway because this sort of prohibition is stupid<\/strong>.<\/p>\n I realize this comes off as uncharacteristically Renegade of me. I’m usually very Lawful Good about this sort of thing, but there comes a point where I think it’s important to draw the line.<\/p>\n Let’s say I’m walking down the street and I see a sign on a building that says “KEEP OUT!”\u00a0<\/p>\n Okay. Cool. I’m going to stay out. This is someone else’s property, and they haven’t given me permission to enter. That’s fine. I’ll piss off.<\/p>\n Now let’s say I’m walking down the street and I see a great big sign, which is brightly lit and easily visible to anyone walking by. I read the sign, and then at the bottom it says that I’m not allowed to tell other people about the sign. Or maybe I can tell them about it, but not take a picture of it. Or perhaps they don’t want me reading it aloud or writing it down.<\/p>\n Some people see this as a perfectly reasonable request, but I can’t help but see it as an encroachment on my<\/b> freedom. You can’t put something in my head and then insist I’m not allowed to tell other people about that thing. Your public-facing, reachable-by-Google page is a giant lit up sign facing the street, and you have no right to tell me I can’t use my camera in public and you certainly don’t have any right to inhibit my speech by demanding I keep your sign a secret.\u00a0<\/p>\n If the sign is facing the public street, then presumably I’m allowed to read it. (And If I’m not, then it’s your fault for putting your sign there.) But see, I’m a slow reader. So I’m going to have my friend here read the sign for me. The fact that my friend is a robot is beside the point. He’s a friend. The point is that he’s helping me to read and remember what the sign says, and if I have permission then he does too.<\/p>\n The information on Metacritic is visible to all. I’m just building a robot to look at it for me. It’s true that I can’t encroach on your property, but you can’t tell me what I can and can’t do with my robot.<\/p>\n Some people have a totally different mental model of all of this. To them, going to a website is like going INSIDE someone’s building. Certainly you have the right to ban photography within your own building. You should be able to prohibit drones. Demanding people not tell others about what they see inside is a bit iffy without an NDA, but I think most people would agree you have rights over me while I’m in your home that you don’t have if we’re standing on the street.<\/p>\n Using this mental model, the “no bots allowed” demand makes a lot more sense.\u00a0<\/p>\n The problem is that neither of these mental models are correct, because the internet is very much its own thing. We try to map it to familiar ideas so we can import our existing collection of moral assumptions, norms, and etiquette. That works most of the time, but sometimes the novelty of this system is inescapable.\u00a0\u00a0<\/p>\nwww.metacritic.com\/browse\/games\/score\/metascore\/all\/pc\/filtered?view=detailed&sort=desc&page=0<\/code><\/p>\n\/pc\/<\/code> and I can choose a page by changing the number at the very end. This particular index will only list games with a rating of 30 or above, which will filter out a bunch of dross that we’re not interested in right nowThere’s another index that lists all games alphabetically with no filter. I might switch to that someday if I’m curious about all of the bottom-of-the-barrel stuff.<\/span>.<\/p>\nAsk for Permission, or Forgiveness?<\/h3>\n
<\/div>
\n2) Every time you need something from the site, compare the prospective URL to the deny list to make sure you’re not inside of any of the forbidden zones or grabbing a forbidden type of file.
\n3) Create fallback behavior for when you can’t get to something you need.
\n4) Set up a test on my own webserver to prove that the system works as intended, or else all of the above was a waste of time.<\/p>\nIt’s Just Me, and My Robot Friend<\/h3>\n
<\/div>On the OTHER Hand…<\/h3>\n
<\/div>