T w e n t y S i d e d

So now I’m done messing around and being silly. It’s time to actually scrape the web for stuff. There are three different sites I’m interested in:

1. Metacritic, for critic scores.
2. Wikipedia, for credits regarding director, writer, producer, composer, etc. This information is spotty and I can’t think of how it might be useful right now, but I’m going to include it as part of the exercise. Also, Wikipedia often notes what franchise a game is from, which might be handy if I want to do a search that includes “all Resident Evil games” or somesuch. 
3. Steam, for PC -specific info like DRM, controller support, multiplayer, etc.

There’s also a bit of information that can come from any of these sources: The url for the game’s official website might be handy, and we also need to get the publisher, developer, and release date from one of these places.

Of the three sites, it seems like Metacritic is the best one to start with. It has games listed by platform, which is necessary in a structural sense. For the purposes of our database, it’s possible for the same game to have vastly different information depending on platform. For example, maybe a game is released on the Playstation 3 in 2010 by Beloved Developer, but then a year later it gets ported to the PC by Shovelware Games. Metacritic is the only place where we can get this information reliably. Steam obviously isn’t going to have non-PC data, and Wikipedia entries aren’t guaranteed to have all the per-platform data in an easy-to capture locationIt might be in the info box on the right, or it might be buried in the article text (good luck capturing THAT) or it might not be listed at all.. 

Metacritic even has a handy index page that you can go through: Continue reading ⟩⟩ “Scraping Part 3: A Well-Behaved Bot”

Shamus Young is a programmer, an author, and nearly a composer. He works on this site full time. If you'd like to support him, you can do so via Patreon or PayPal.

So there are thousands of webpages that have information we want. When faced with this problem, ancient civilizations used to go to these pages using Internet Explorer 6 and copy the data into Notepad. We don’t know what they did with it after that, because they got eaten by Woolly Mammoths or conquered by Mongols or whatever. I’m not a historian so I might be slightly off with my timeline, but you get the basic idea: The past was hard.

But now we have these newfangled web scrapers that can surf the web for you and harvest whatever data you like. The problem is that putting the data into Notepad isn’t terribly helpful. Great, now you have an enormous text file of random facts. Are you going to sit down and read it manually? Probably not. So what do we do? Write another program to read that file? You need to turn this text into data sooner or later, and to do that we need to put it into a database.

Continue reading ⟩⟩ “Scraping Part 2: Full Control”

Shamus Young is a programmer, an author, and nearly a composer. He works on this site full time. If you'd like to support him, you can do so via Patreon or PayPal.

You might remember a couple of months ago I posted a bunch of charts of video game data. The obvious question that went unanswered in those postsTo the genuine annoyance of some. was, “Where did this data come from?” So let’s talk about that.

Actually, before we talk about that I should make it clear that this is a programming project. I should note that that this project pre-dates that crazy stuff I was doing with BSP loading a couple of weeks ago, but I’m posting them in the opposite order. For some reason.

Maybe reading yet another programming project sounds fun, but this isn’t a game-focused project with cool screenshots to show off my project. This is pretty dry and you’ve already seen the end result. I’d talk you out of reading more, but we both know you’re going to read this stupid thing no matter what I say. So Let’s just get this over with.

Continue reading ⟩⟩ “Scraping Part 1: Easy Mode”

Shamus Young is a programmer, an author, and nearly a composer. He works on this site full time. If you'd like to support him, you can do so via Patreon or PayPal.

In the previous entry I ended with an abrupt half-joke where I said I got raytracing working. The idea was that I spent days struggling to get simple old technology working properly, but then casually mastered cutting-edge tech in a single sentence. Sadly, it’s not totally true. I got access to raytracing, but I think it’s a stretch to say it’s “working”.

When I left off, I presented you with an image that looked more or less like this:

That’s what the program looks like when you leave it alone, but if you move the camera even slightly then it looks something like this: Continue reading ⟩⟩ “Adventures in Raytracing”

Shamus Young is a programmer, an author, and nearly a composer. He works on this site full time. If you'd like to support him, you can do so via Patreon or PayPal.