Site Trivia

By Shamus Posted Thursday Apr 2, 2009

Filed under: Random 46 comments

  • The first post on this site is dated Sept 1st, 2005.
  • There are 1,869 published posts. (1,870 once I hit “publish” on this one.)
  • There are 93 drafts. (Posts I started but never finished for various reasons. I really should clean that out someday.)
  • There are 61,546 comments.
  • There are 58.7MB of images on the site. (That doesn’t actually seem like very much. I keep meaning to use more images. My goal is supposedly at least one image per post, but I forget so often that calling it a “goal” is sort of pathetic.)
  • The comments take up 43.2 MB in the database. The posts take up 8.4 MB. That means that together, we’ve written over fifty-one megabytes of text in four and a half years. It also means that comments make up about 84% of the readable content of the site.
  • The database export file for the site and all its settings is 55.2MB.

This last number worries me. Most import operations are capped at a measly 2MB. If I ever needed to move to a new host, I’m not sure how it could be done. I often import a copy of the site to my local machine and work on it there. (It’s kind of confusing if I have the local copy open in one window and the real site open in another. I’m always worried I’ll get mixed up and do something destructive to the live one, thinking it’s the local one. I try to use IE for the local one (because the site looks different in IE, even with the same theme) to reduce the chances of this.) I have to alter the settings to my local webserver to be able to import the site data. I need to raise the limit on how much memory PHP is allowed to use, and how long a PHP script is allowed to run. Not all webhosts give you access to those settings. Changing hosts at this point would be a major undertaking.

I feel the site is a little sluggish these days when compared to how fast it was in the beginning. Even when I’m editing a post, refreshing the page takes about four seconds. How much of this is due to the extra traffic, how much is due to the larger database, and how much is just plain old net congestion? I have no way of knowing. My fear is that the performance will continue to erode as the site grows. Eventually the slowness will drive people away. I know there are many sites I’ve stopped reading, simply because they were too sluggish to navigate.

 


From The Archives:
 

46 thoughts on “Site Trivia

  1. Joe Cool says:

    About how many characters/words do the combined writings of you and your commenters equal? What I’m after is about how many printed pages would be filled if you printed off the entire website to a book? Are we at encyclopedic length yet, or merely unabridged dictionary length?

    An it’s all been written in four and a half years. Imagine if you and your readers used your powers for evil.

  2. Maldeus says:

    We could…Write an evil encyclopedia?

    Also, you’d have to be running a really slow site to drive me away, Shamus.

  3. Snook says:

    I haven’t noticed a slowdown since I started reading 2 years ago. Changing hosts now seems a tad silly, unless the slowness bothers you that much…

    I use Chrome if it matters to load times. :p

  4. onosson says:

    I have a stack of floppies somewhere that you are welcome to use when it comes time to transfer…

  5. SPG says:

    @1 Joe Cool: According to first result on Google(pages per megabyte), the average number of pages is 662 for straight text. Assuming that number holds:
    51.6MB * 662 = 34,159.2 pages total … Huh. Pi? Is that you?

    By contrast, if I parsed Wikipedia correctly, the Encyclopedia Britannica’s Micro- and Macropedia compose ~29,000 pages.

    The 662 estimate holds in a few results, others give another number. There are no guarantees that this is completely accurate.

    (First post on this site. Keep up the good work Shamus. If your site slows, I’ll just get a cup of coffee while it loads.)

  6. RichardB says:

    FWIW I’m not noticing any slowness here.

  7. Fenix says:

    As far as sites go, I would never say this site feels sluggish. At least your site doesn’t load up like 100 flash objects before letting me read anything.

  8. JMcNeely says:

    The speed of the site is actually what kept me coming back after I finished reading your DMotR and the D&D Campaign. It’s usually the first site I come to in the mornings and it’s easy enough to check back occasionally without wasting a boatload of time. Of course, I have a tendency to go through and open about 7 tabs every morning before reading anything, so even if it was slow it wouldn’t make a difference.

  9. Mari says:

    What constantly astounds me is that 84% of the readable content is comments and yet they stay (for the most part) so civil. It’s just astonishing to me the community you’ve built here. I love your posts and value your insights but I have to admit that a fair bit of the appeal of the site would be diminished without the comments of your readership.

    And I wouldn’t say the site feels sluggish at all. Compared to a great many sites I load in the mornings, this one loads very quickly. For the record, I use the “Morning Coffee” Firefox add-on that loads up all the blogs and webcomics I read in one fell swoop when I get up. Your tab opens first and loads third (out of an average of 15 tabs per day).

  10. Ludo says:

    Let’s approximate the total content of this site to 35,000 pages. How many bibles does this represent ? (only as a numerical point of reference of course)

    @#8 Vladius : I’m sorry, I didn’t understand ;)

    @#10 Mari : I just love your red hair :) (and I totally agree with you on the community)

    @Shamus : I’m loading yout site from over the Altantic Ocean and the response time is Good to Very Good. Kudos to the host. As for your fear of migrating, 2 MB is only a concern if you use phpmyadmin and you have absolutely no control over your server. I trust the customer service of any respectable host to upload a specific sql file on a given account without much fuss. Or to allow you to do it by yourself if need be (ssh is your friend).

    A little bit of sunshine : you’re shining outside of the USA, keep up the good work :)

  11. macrovore says:

    Well, let’s hope the tubes don’t get any more clogged. Else we wouldn’t be able to see the site.

  12. RodeoClown says:

    I work on a web-based program for my day-job, and I understand COMPLETELY the problem you have with distinguishing the two sites (local and production).

    To solve the problem, I had everyone at work install Greasemonkey for firefox, and an equivalent for IE, then wrote a quick script that changed the background colour to bright red if the site being viewed was the production one.

    Makes it REAL obvious when you are on the real site (and you can turn off the script when you aren’t doing work).

  13. LanceWithAbee says:

    After some period of time you could stop archiving comments. As thought provoking and civil as your readers can be, they are just window dressing. Your thoughts are the star of the blog.

  14. Robyrt says:

    @Ludo: The Bible has 774746 words. At 500 words per page, that’s 1550 pages of straight text. Formatting usually increases it to ~2000 pages, but these comments will also require plenty of formatting.

    As far as pure wordcount goes, you’re looking at 11 Bibles.

  15. 55mb is tiny…and as Ludo said, do not use phpMyAdmin to backup or restore a whole database. That’s what mysqldump and the mysql program itself is for.

  16. Blake says:

    I’m in Australia and I’ve always thought of your site as one of the faster ones.
    As for the civil comments, it’s always amazed me too that there’s no flame wars especially when there are often disagreements.
    Either you got very lucky or you’ve been doing something very right :)

  17. Strangeite says:

    TwentySided would have to fall into a blackhole for it to seem slow to me since I am a reader of Giant In The Playground.

    Nice touch that you did not post an image in the post that you state it is your goal to post one image per post.

  18. Jabor says:

    @Lance: That sounds like a bad idea. Most of the joy of reading back through old posts is reading the comments on them (some of the commenter’s(sp?) are just as funny as Shamus!), and it would be a real shame to lose all that.

    Though I could see disabling comments on older posts and just keeping them as static HTML pages as an option for lowering the database size.

  19. Mari says:

    Strangeite – Well thank goodness I finally have confirmation of that one. The only people I know who read GitP live under the same roof as myself so I’ve always had this niggling worry that maybe the issues we have accessing the site are unique to our ISP or network. Now I know better. But yep, that’s actually the last site to load with my “morning coffee” if it loads at all (it seems like that’s only an issue, though, when Rich has a new comic up).

    Oh, and Ludo – if you’re an anime fan, you should definitely check out Android Announcer Maico, from which my avatar is shamelessly lifted.

  20. Dev Null says:

    My goal is supposedly at least one image per post


    (though to be fair, what image do you include in a blog post about the content of blog posts?)

  21. Zel says:

    Well, I don’t mean to go against everyone else here, but I did notice some slowdowns recently.

    The RSS page takes a few more seconds to load in Sage (Firefox RSS add-on), and sometimes doesn’t load at all. Pages seem slower to load but I’ve got less of these annoying black boxes hiding comments at the end of the page (I don’t know if I’m the only one experiencing this…), so less refreshes needed to see everything.

    Editing comments (something I do a lot to fix spelling) takes much more time since the new comment editing plugin was introduced : the old one would load my text instantly and was nicely integrated in the page. Now I have to wait for the plugin window to take over the tab, then it downloads my text (few seconds), and when I’m done it takes a few seconds to save. I’m also pretty sure you could scroll the page while editing before, now you still can but the page is blacked-out with a huge window in the middle so it’s unreadable (evil theme).

  22. MuonDecay says:

    Funny anecdotal consideration: a large number of people in the US have noticed that their DSL speeds have been on a visible decline in the past couple of years.

    I would not be surprised if net congestion is beginning to bottleneck in some of the nodes in the country’s ISP networks.

    The exaflood or whatever it’s called now was likely a chicken little thing, but I do believe that people will likely begin to experience some amount of that pinch when some of the more localized routing centers for networks get close to being overwhelmed. You might have some neighborhoods repeatedly losing connectivity, or a whole county here or there.

    I’m no expert, though. I reserve the right to be naively incorrect.

  23. SatansBestBuddy says:

    I’m at about 3.5 seconds for a page to load enough that I can read it, and 7-9 seconds for a fully loaded page.

    I’m a pretty paitent guy, though, I tend to wait about 30 seconds before I continue on and do something else.

  24. Zerotime says:

    MuonDecay: More likely, that’s just simple overselling by their ISPs.

  25. Volatar says:

    I open about a dozen tabs whenever I get on anyways. I then open each post I have not read in a separate tab and close the main one.

    I do this for other blogs, forums, and Slashdot too.

    After I have somewhere around 30-40 tabs, then I read them. :)

    I never worry about speeds.

  26. Julian says:

    This site is one of the fastest ones I read daily (I use Morning Coffee. Your tab loads fourth, right after Rob & Elliot, xkcd and Darths and Droids. You are closely followed by Darths and Droids.)
    HOWEVER, there are some times when the pages just. won’t. load, but they’re rare, and they go away after 20 minutes.

  27. mark says:

    The site has always loaded fine for me here in the UK, with an 8Mbit line, shared with 4 housemates.

  28. JMcNeely says:

    Just to throw in my 2 cents on the GiantITP issues… The main page loaded to a readable state in approximately 5 seconds, though the sidebar .gifs took about 45 seconds to all load in correctly. The comic page, navigated to from the home page, took about 3 seconds to load.

    Also, this is the first time I have visited that site on this computer so no cookies, history, or any other associated crap has been pre-loaded, pre-cached or what have you.

    Now this probably won’t be similar for anyone else because this is on a fresh install of Vista x64 using Firefox with Ad Blocker on a 6MB line that no one else is currently using. So take it with a grain of salt and call the doctor in the morning, or whatever. But there it is for whatever it’s worth.

    (edit) Trying to browse back into the archives seems to be causing a server timeout… I guess this is what others have been experiencing?

  29. nilus says:

    Well instead of changing hosts you could just become unpopular. If you just start acting like a jerk and posting unpopular things your traffic should die down. Either that or it the whole plan would backfire and it traffic would skyrocket. Either way it could be fun.

  30. Stuart says:

    I can’t believe how small your image are… Even including DM of the Rings? :-)

  31. maehara says:

    Importing that database, should you ever need to: I’ve found BigDump invaluable in the past for working around upload limits…

  32. Helm says:

    I’ve never seen it be slow TBH GiTP is a ‘mare a lot of the time but like your site it’s funny and my life isn’t so sad I can’t spare a minute for a laugh

  33. xbolt says:

    Is it just me, or does this post change the theme to ‘True neutral’? I usually use Lawful good, and that shows up everywhere else, but this post shows True neutral. Very odd.

    EDIT: Heh, as soon as I comment, it goes back to normal. I guess my internet had a hiccup.

  34. vrittis says:

    Shamus, perhaps you could do some changes to the way the system works by freezing older posts. I guess there’s a sweet point in time after which no comments are added to a post (3 months, perhaps 4)
    Why not freeze these posts, and display them as static html? Imagine a table frozen_content in which you could dump older posts and comments; it would fetch the main area content from this table, and save you some trips to the database in order to bring back posts and comments.
    There could be problems regarding gravatars, i’m not sure, but apart from that, i think it could help you reduce your footprint at best, and would also help with the “sluggishness”

    And i agree about the community being incredibly civil and pleasant. I had the luck to play on the L4D server and found the same kind of pleasant and civil mood: lucky!

  35. roxysteve says:

    Well, I used to be a rather brilliant DBA (I’m now a novice Solaris sysadmin and perl groo).

    If getting stuff out of a database is becoming a tad onerous I might be able to suggest a few things, depending on the tech/engine you are using (I’d bet on MySQL but that’s a guess).

    But I have a database neutral suggestion that will cut the storage by quite a bit: Remove all the vowels from all the posts older than three days. Then remove some of the “y”s.

    Ths cn b dn wth smpl rglr xprssn.

    :o)

    I’ve not noticed sluggishness other than in the commentators (Owowowowow!).

    As always, sincere regards and best wishes to the brilliant creator of DM of the Rings – which it turns out my latest DM has never heard of. This speaks volumes for his gamosphere accumen. I mean, who could be :

    a) running a RPG
    2) an avid intarwub user
    #) iggerint of the Great and Powerful Shamus Young, game commentator without portfolio and DM of the Rings, singly the most relevant commentary on D&D in the universe ever made in the entire history of time, and a damn good laff to boot?

    It beggars the imagination. It also explains why he was taken by surprise when his city guard demanded our weapons and we all said in unison “you mean my walking stick?”

    Steve.

  36. Inwards says:

    Shamus, don’t take this the wrong way, but you have tiny database. Even if the site were to increase by 10x, you’d probably see very little appreciable difference in performance (assuming that you haven’t forgotten to build indexes or something) My own pet site has a 3.2 gigabyte database and most page loads are sub 1-second.

    I wouldn’t worry very much about the 2meg memory limit, either. As long as you let the DB do the work, the size of your db should have very little impact on the amount of memory that your PHP scripts consume.

    PS; if your ISP really starts to suck, I’m sure I could accommodate you on the ZP box.

  37. Hawk says:

    The site’s pretty fast, kudos. I’m posting from NoGameistan (measly bandwidth) and your site pops right up, unlike most other sites I try to access. Keep up the good work.

  38. Tacoma says:

    I’m willing to put up with a lot of bugs and slowness if the content really rocks. For example, Giant in the Playground is an AWFUL website for usability. It often doesn’t load and takes more than a minute to load sometimes. The fora are all but unusable. The dude hasn’t updated his articles in a long time so I’ve read them all and gleaned what I needed from them. But there are two comics there are rock and so I’m willing to check in once a week or so.

  39. Cybron says:

    You’d have to be pretty damn slow to get me to leave. Hell, I still read Order of the Stick (its load speed is terrible), and you’re more entertaining than that.

  40. Using wget, I made a local backup mirror of your site in case you ever disappeared from the face of the Earth and took your website with you. I hate it when that happens with other people, and I want to be prepared in case it happens, since you have some great stuff here. I update my little mirror about once a month. It only downloads the changes (i.e. new posts), and I make it go very slowly (one request per ~10 seconds) so it doesn’t strain your server.

    Let me play with your data here.

    I count 2372 individual pages.

    A quick estimate piping the whole thing through lynx, to strip HTML, then into wc shows about 8 million words (posts + comments). It is 50.8 MB of “lynx -dump -nolist” text (very close to your count) and 143MB of HTML. In other words, HTML bloats your content by a factor of ~3. I assume it is not stored as HTML, which explains these numbers pretty well.

    I count about 56MB of images.

    Some information theory: the lynx dump can be compressed with gzip to 17MB, bzip2 to 13MB, and lzma to 11MB. I won’t bother with paq8l which would take about a day to complete. So there is about 10MB worth of pure, non-image information here.

    I can pack a static mirror of your website, images and all (HTML version), down into a 67MB bzip2 tarball.

    Note: My current mirror is a couple weeks old now, and I have no quick way to divide comments and posts and stuff for further analysis. The mirror only includes material under “/twentysided/”.

  41. Possibly interesting word counts:

    The top ten words are boring, common words: “the”, “to”, “a”, “of”, “and”, “i”, “in”, “that”, “at”, and “is”.

    “game” and its variations almost make it into the top ten, with around 60,000 appearances.

    “Shamus” is ~50 with around ~20,000 appearances (but it appears on every page, too, so subtract ~2000).

    “DRM” is the ~500th most common word, appearing ~2000 times.

    “tabletop” is at number 325

    “accordion” only appears once. Make that 2 times now.

    “fuck” appears only 250 times.

    1. Shamus says:

      Chris Wellons: Thanks for gathering up that data. That was great.

  42. Sitte says:

    I don’t know if I’d call it slowness – the page either loads for me, or it doesn’t. Of course, it *doesn’t* about 20% of the time. Usually hitting refresh a couple of times takes care of that, but sometimes I have to come back the next day (for instance, last night I was unable to get it to load after many tries).

    I read 20-S at the same time as a few other sites, though, so it doesn’t bother me much. I load all the pages at the same time, and if this one doesn’t load on the first go I just periodically switch over and hit F5, then go back to reading other stuff.

  43. Ralph says:

    There are lots of solutions to uploading big databases to un-cooperative hosts. I used bigdump a couple of weeks back to upload 70Mb of phpmyadmin SQL export. Upload the exported database, fill in the database settings in the script and run it, it will then use AJAX to import the database in little chunks a locked down PHP can handle.

    http://www.ozerov.de/bigdump.php

  44. Wollollo says:

    Concerning loading times: With all the good writing on every page on this site, the loading/reading time ratio is ridiculously low, which for me would have made quite bad waits perfectly bearable. Not that I have ever had to wait noticeably.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply to Julian Cancel reply

Your email address will not be published.