T w e n t y S i d e d

My bot is nowWell, not RIGHT now. This series was written after the bot was completed. downloading pages from Metacritic, one at a time, at the rate of a page every couple of seconds. This would be painfully slow if we were trying to read something large-scale, but right now we’re just scraping for PC games that scored above 30 over the last 19 years. That’s well under 1,000 games.

Of course, downloading these pages isn’t useful unless I can pull information out of them. Much earlier in this series I mentioned I’m using the Html Agility pack. This library can parse HTML for me and return the bits I’m interested in.

One of the funny things about this project is that I’m so far out of my comfort zone / area of expertise that I don’t even know what I don’t know. Not only am I likely making lots of hilarious blunders, but I don’t even know that I’m making them.

This is strangely liberating. When I know what I’m doing, then every cut corner makes me feel vaguely guilty. But when you don’t know what you’re doing, you’re free of the obligations to do things the Right Way(tm) because you don’t know what the right way is! As far as I know, I’ve just written the best web scraper in the history of scrapingDespite the lack of proof, I’m fairly confident that I have not actually written the best web scraper in the history of scraping..

Continue reading ⟩⟩ “Scraping Part 4: THE FINAL CHAPTER”

Shamus Young is a programmer, an author, and nearly a composer. He works on this site full time. If you'd like to support him, you can do so via Patreon or PayPal.

So now I’m done messing around and being silly. It’s time to actually scrape the web for stuff. There are three different sites I’m interested in:

1. Metacritic, for critic scores.
2. Wikipedia, for credits regarding director, writer, producer, composer, etc. This information is spotty and I can’t think of how it might be useful right now, but I’m going to include it as part of the exercise. Also, Wikipedia often notes what franchise a game is from, which might be handy if I want to do a search that includes “all Resident Evil games” or somesuch. 
3. Steam, for PC -specific info like DRM, controller support, multiplayer, etc.

There’s also a bit of information that can come from any of these sources: The url for the game’s official website might be handy, and we also need to get the publisher, developer, and release date from one of these places.

Of the three sites, it seems like Metacritic is the best one to start with. It has games listed by platform, which is necessary in a structural sense. For the purposes of our database, it’s possible for the same game to have vastly different information depending on platform. For example, maybe a game is released on the Playstation 3 in 2010 by Beloved Developer, but then a year later it gets ported to the PC by Shovelware Games. Metacritic is the only place where we can get this information reliably. Steam obviously isn’t going to have non-PC data, and Wikipedia entries aren’t guaranteed to have all the per-platform data in an easy-to capture locationIt might be in the info box on the right, or it might be buried in the article text (good luck capturing THAT) or it might not be listed at all.. 

Metacritic even has a handy index page that you can go through: Continue reading ⟩⟩ “Scraping Part 3: A Well-Behaved Bot”

Shamus Young is a programmer, an author, and nearly a composer. He works on this site full time. If you'd like to support him, you can do so via Patreon or PayPal.

So there are thousands of webpages that have information we want. When faced with this problem, ancient civilizations used to go to these pages using Internet Explorer 6 and copy the data into Notepad. We don’t know what they did with it after that, because they got eaten by Woolly Mammoths or conquered by Mongols or whatever. I’m not a historian so I might be slightly off with my timeline, but you get the basic idea: The past was hard.

But now we have these newfangled web scrapers that can surf the web for you and harvest whatever data you like. The problem is that putting the data into Notepad isn’t terribly helpful. Great, now you have an enormous text file of random facts. Are you going to sit down and read it manually? Probably not. So what do we do? Write another program to read that file? You need to turn this text into data sooner or later, and to do that we need to put it into a database.

Continue reading ⟩⟩ “Scraping Part 2: Full Control”

Shamus Young is a programmer, an author, and nearly a composer. He works on this site full time. If you'd like to support him, you can do so via Patreon or PayPal.