{"id":49806,"date":"2020-04-23T06:00:12","date_gmt":"2020-04-23T10:00:12","guid":{"rendered":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=49806"},"modified":"2020-04-23T09:45:31","modified_gmt":"2020-04-23T13:45:31","slug":"49806","status":"publish","type":"post","link":"https:\/\/www.shamusyoung.com\/twentysidedtale\/?p=49806","title":{"rendered":"Scraping Part 1: Easy Mode"},"content":{"rendered":"

You might remember a couple of months ago I posted a bunch of charts of video game data<\/a>. The obvious question that went unanswered in those postsTo the genuine annoyance of some.<\/span> was, “Where did this data come from?” So let’s talk about that.<\/p>\n

Actually, before we talk about that I should make it clear that this is a programming project. I should note that that this project pre-dates that crazy stuff I was doing with BSP loading<\/a> a couple of weeks ago, but I’m posting them in the opposite order. For some reason.<\/p>\n

Maybe reading yet another programming project sounds<\/strong> fun, but this isn’t a game-focused project with cool screenshots to show off my project. This is pretty dry and you’ve already seen the end result<\/a>. I’d talk you out of reading more, but we both know you’re going to read this stupid thing no matter what I say. So Let’s just get this over with.<\/p>\n

<\/p>\n

For years, I’ve been wondering about the stuff we’re always discussing \/ arguing about in gaming culture. The division between fans and critics. The difference between platforms. The changes to the industry over time.<\/p>\n

The problem is that we never have any numbers to work with. We just sloppily take our anecdataAnecdotes extrapolated into “data”.<\/span> and project it onto the industry as a whole. Just about everyone realizes this isn’t a scientific way of going about things, but we don’t really have any alternatives. It’s either guessing based on personal experience, or we chow down on the PR slop the various publishers feed usOr should we read quarterly reports aimed at shareholders, and swallow THEIR slop?<\/span>.<\/p>\n

Do particular DRM schemes impact audience reaction or sales? Do console generations impact PC sales? Do single-player games with tacked-on multiplayer actually sell \/ score higher than games without those features? Does review-bombing impact sales, or is the practice just a harmless but cathartic way of expressing outrage? It feels like critics and consumers have been drifting apart in terms of what they say about games, but is that perceived gap reflected in the review scores?<\/p>\n

I suppose at the root of it was a general curiosity about the decision-making happening at the big publishers. We can’t see what game budgets are, we don’t have access to reliable sales figures, and without those numbers we have no way of even guessing about how much particular games are making or losing. Sites like VGCharts<\/a> and SteamSpy<\/a> give us some estimates to play around with, but for the most part we’re stuck in the dark.<\/p>\n

However, it seemed like there was some<\/b> data out there. We can’t answer all our questions, but maybe we can fill in a few more blanks. Wikipedia has a lot of information on game features and developers. Steam has information on DRM and system requirements. And of course Metacritic has the key information regarding critical reception.<\/p>\n

So the obvious question is: If there’s a bunch of data available to the public, then why don’t we just round it up? (Preferably without having to do it by hand.)<\/p>\n

How Do You Do That?<\/h3>\n

The process of having a program load web pages and pull out desired information is called Web Scraping<\/a>. I’ve never written a web scraper before, but I’d always wanted to try it out. It just seems like a fun idea to have a program surf the web for you and bring back a great big haul of information. Maybe, deep down, this project was more about my desire to write a web scraper than to study the resulting data. But this project seemed like a fun way to satisfy both of these curiosities.<\/p>\n

As I discovered, the process of building a web scraper is pretty easy. For a project at this small scale, I’d even say it goes from “easy” to “trivial”. All told, this whole project was much less than a week of work. If you handed this project off to someone who knew what they were doing, they could probably finish in a couple of days.<\/p>\n

In the old days, I would have done this with C++. But now I’ve spent time time with Unity<\/a> and learned just enough C# to be dangerous. Since that project I’ve wanted to play around with C# apart from Unity so I could get a feel for what C# is “really” like. The environment that comes with Unity has a ton of game-specific features, and it’s not always clear to a newbie which things you’re using are “standard C#” and which bits come with UnityI’d sort of assumed that Unity-specific stuff would have Unity-specific includes, but it’s also possible Unity comes bundled with some third-party things and conventions.<\/span>. In Unity projects, the engine controls the loop. Tens of thousands of lines of invisibleInvisible to the game developer. I’m going to assume people working on the engine can see their own code.<\/span> code might be run before Unity gets around to reaching the bits of the program you’ve written. In vanilla C#, program execution begins and ends with your codeOkay, there’s probably a little bit of stuff the program does that’s invisible to a regular C# programmer, but that’s NOTHING compared to the gargantuan task that Unity does when it creates a window, launches a rendering pipeline, initializes the sound system, loads assets, and a thousand other things.<\/span>, and I wanted to get a feel for how that worked.<\/p>\n

The Hardest Thing is Realizing how Easy it is.<\/h3>\n

The biggest thing that held me back was my learned habits. I’m used to the C++ world where you need to do everything by hand or spend time trying to figure out how to make alien code<\/a> work with your program. Want to parse some text? Write a text parser. Want to read web pages? You’d better know how to implement your own HTTP stack, including networking, DNS lookups, HTTP requests, and a dozen other things I also don’t know how to do. (Or you could import a library that might not do what you want, or might not have documentation, and might not even compile.)<\/p>\n

I kept assuming tasks were going to be hard. I’d get half an hour into writing something from scratch, and then I’d realize there was already a tool for it that was effortless to import and completely intuitive to use. A lot of this project was less about programming and more about learning how to find out what (if any) programming needs to be done.<\/p>\n

The best example of this is when I tried to write code to parse web pages. At first I did the naive thing:<\/p>\n

    \n
  1. If you’re a new programmer that learned to code on a very high-level language with lots of convenience features, then the naive assumption is that there’s a library out there that will do all the work for you, and all you need is to copy a couple of lines of code from StackOverflow.<\/li>\n
  2. If you’re a dusty old greybeard with knowledge of the Old Ways and ANSI C, then the naive thing is to assume you’ll need to do everything by hand, painstakingly juggling small blocks of memory and writing dozens of lines of code to accomplish simple things.<\/li>\n<\/ol>\n

    I was the second kind of naive. I wrote a text parser that would take the contents of an entire webpage as one big string and look for fragments I was interested in. For example, maybe I’m scraping data from Metacritic and I want to get the title of the game from the webpage. By inspecting the raw Metacritic HTML manually, I’ve discovered that\u00a0the title of the game is contained in a <div> tag with a class of “gametitle”It’s more complex than this in practice, but this works as an example.<\/span>. So the HTML code might look like:<\/p>\n

    Shoot Guy IV: Shoot Harder<\/div><\/pre>\n
    So my program downloads the\u00a0page, loads it into memory, and I have it search the HTML for “gametitle”.\u00a0 Then I look forward for the nearby closing bracket “>”. Then I’d search for the next opening bracket “<“. In theory, the title of the game should be between those two points.<\/p>\n
    The problem with this sort of approach is that it’s incredibly fragile. If the website suffers a redesign, then it could lead to chaos in my code. Maybe in the new design, the “gametitle” div is a container for the title of the game, plus the cover image, some publisher info, and some random branding logos. There’s no telling how my parser would handle that, and the odds are extremely high that it would extract a random block of HTML markup \/ CSS as the title of the game.<\/p>\n
    I knew this wasn’t the “Right” way to do it, but I was anxious to get the thing up and running before I began learning the “right” way to do things, which I assumed would take a long time.<\/p>\n
    The next day I came back to the projectAnd perhaps to my senses.<\/span> and started looking for something to help me parse these web pages. I realized I was going to have to make different parsers for all the different websites I might need to deal with, and rather than making three or four parsers, it would probably be smarter to just bite the bullet and use someone else’s library.<\/p>\n
    The Lazy Way is Also the Right Way?<\/h3>\n
    This is exactly what it looked like when I worked on this project, except I'm a man, I'm twice her age, I'm not in a wheelchair, I wasn't using a laptop, my office is never this bright, and I'm not a stock photo model. Okay, so this picture has nothing to do with the project. I just wanted to break up this wall of text.<\/div>
    This is exactly what it looked like when I worked on this project, except I'm a man, I'm twice her age, I'm not in a wheelchair, I wasn't using a laptop, my office is never this bright, and I'm not a stock photo model. Okay, so this picture has nothing to do with the project. I just wanted to break up this wall of text.<\/div><\/p>\n
    As an old-school C \/ C++ programmer, my expectation is:<\/p>\n