T w e n t y S i d e d
I’m pretty frustrated with the performance of this website. The recent links from Slashdot and Reddit have shown that the thing is fragile and will fold up under a wave of visitors. In general, new visitors are the worst ones to lose. Regulars will check back later if the site is down right now, but a new visitor is going to surf by without a second look if they click a link and don’t get a timely response. Life is too short to wait around for feeble websites.
I use Hosting Matters, and they host blogs even larger than mine without difficulty. I still soak up more bandwidth than even their largest plan has to offer, but the site isn’t anywhere near large enough to warrant a dedicated server or anything. So I can only conclude that the site is creating some sort of performance bottleneck. I also suspect that the problem isn’t bandwidth, but CPU cycles. PHP can keep the processor pretty busy. Things like Wavatars , the dice roller, and some of the other little gimmick plugins are no doubt creating a load that more text-based blogs don’t have to deal with.
Some wordpress plugins are CPU intensive, either due to their scope or inefficient coding. In some cases a seemingly innocuous plugin can bring down an entire machine, all by itself. I really think that web hosting companies need to give you a way of seeing how much CPU you’re using. You can see how much bandwidth and harddrive you’re using, but there’s no way to gauge CPU use short of calling up your hosting company and asking them, “So… how’s it goin?” It’s clear this site caused a lot of problems yesterday. Not just for me, but for anyone else using the Adams server at Hosting Matters. But site admins and plugin authors don’t usually have any motivation or tools for making well-behaved sites, because you can’t judge how well something works until it blows up.
Some hosts offer a little green light / red light to let you know how the machine is doing, but that only lets you know that there is a problem, not how bad it is or who caused it. Site admins would do better if we could see how many CPU cycles we’re consuming. This would make people aware of which plugins cause problems, which will lead to more efficient plugins in the future. CPU use wasn’t an issue back in the days of static HTML pages, but with all this fancy-pants PHP stuff it’s moved to the forefront. Webhosts will disable your site if it begins crushing the shared machine hosting it, so it’s a bit like a world with an enforced speed limit where nobody owns a speedometer.
Yesterday I upgraded this site to WordPress 2.7.1, and installed the Super Cache plugin. In theory, this should greatly reduce CPU load by serving static pages to most visitors. Only myself and people who have left comments (less than 1% of visitors) will see dynamic pages. Again, this should help, but I have no way of knowing until the next big rush of visitors. Let me know if anything seems screwy. I know on the backend, everything has been moved around to the point where I’m constantly getting lost. But the site itself should work exactly as before for you. I do trust you’ll let me know if this is not the case.
Civilization VI
I'm a very casual fan of the series, but I gave Civilization VI a look to see what was up with this nuclear war simulator.
Grand Theft Auto Retrospective
This series began as a cheap little 2D overhead game and grew into the most profitable entertainment product ever made. I have a love / hate relationship with the series.
Mass Effect Retrospective
A novel-sized analysis of the Mass Effect series that explains where it all went wrong. Spoiler: It was long before the ending.
Free Radical
The product of fandom run unchecked, this novel began as a short story and grew into something of a cult hit.
What is Piracy?
It seems like a simple question, but it turns out everyone has a different idea of right and wrong in the digital world.
Thanks for the update. I’m not sure why I care, but you manage to make this sort of technical update matter to me.
Interesting. I don’t seem to be able to edit my comment any more. That was one of my favorite features, and I’m sure it’s one you didn’t mean to lose with the update.
Here’s what it looks like from my end:
1) I wrote a comment, as usual, and then hit “submit,” figuring I’d read it over and revise it (as usual).
2) The new page loaded with my comment, but without the usual blue border and the countdown allowing me to edit my comment for 12 minutes and counting.
3) Nor can I click and then edit the comment.
– Chuck
Can it really be the case that nobody has commented on this post yet? Maybe I just can’t see them all. In which case half of them are probably nigh identical to this one.
Websites do tend to forget about me so maybe the site does not identify me as a previous commenter, and is serving me a comment-less page?
Maybe slice out the wavatars? they’re kinda cool but slicing them out to save CPU seems worthwhile to me.
I’d really prefer to keep the dice thingy, it’s been a staple of the site for quite a while
My word it seems there really were no comments last time I checked. That’s quite unusual. Nvm then.
Hm, i can’t do the edit thingy either.
That makes me sad, i liked doing that.
You might be interested in the tips at Coding Horror on Reducing Your Website’s Bandwidth Usage.
A mid-range option for you might be a Virtual Private Server. You’d probably want to know a thing or two about Linux and/or Apache to make the most use of it, but it would give you a lot of the benefits of a dedicated server without the costs. I’ve used http://www.hostmysite.com/ for VPS, which runs about $30 a month. You may find it cheaper elsewhere, but I can definitely recommend HMS if you care about the speed and responsiveness of customer service.
The The Coral Content Distribution Network can help you awhile here, at least temporarily. You can use it by adding “.nyud.net” to the URL, so yours would be:
http://www.shamusyoung.com.nyud.net/twentysidedtale/
And it takes the load off your servers. However, your current setup had some issues with it.
First of all, you use that idiotic WordPress plug Bad Behavior that will block IPs and user agents at the drop of a hat. Somewhere it (incorrectly) decided that using coral cache was bad behavior and blocks it. You would need to not use this plugin to make it work.
The other issue is that most of the links on your site are hard links, containing the “www.shamusyoung.com” rather than more relative links. This isn’t a big deal because if you wanted to host through coral cache during heavy loads, you can write a .htaccess file to do the proper redirects anyway.
Turn on gzip encoding!
http://www.dnsqueries.com/en/check_http_gzip.php
http://www.samaxes.com/2008/04/20/htaccess-gzip-and-cache-your-site-for-faster-loading-and-bandwidth-saving/
Not useful for some image types, but I use it for html, php, javascript, and css files
The CPU impact of mod_gzip is minimal (gzip is a tried and true “fast” algorithm) and the bandwidth save/cost of serving a given object is greatly reduced.
Contact me if you’d like some examples in detail
Shamus: If you’re having that kind of CPU usage issues, that might actually be exactly the kind of thing that calls for a dedicated (or virtual dedicated) solution. I’ve been with Linode for a while now, and they’re pretty fantastic – unmanaged, so I’m responsible for keeping everything in working order – but at the same time, I have full control over my Apache, PHP and MySQL configurations and can check and see whether any sort of resource usage is going through the roof.
For twenty bucks a month, I get 360 megabytes of RAM to play with, and 200 gigs of bandwidth. It’s hosting my service (which, without going into overtly advertisement-filled details, consists of a lot of dynamic PHP pages and an IRCd) without any issues.
It looks like the Linode plan has a lot more system resources available than the hostingmatters plans – as in, by more than twice. Since you seem like you know what you’re doing, I bet you could get LAMP set up with no issues – with an Ubuntu or Debian base system, you could probably be up and running in an hour.
With regards to wordpress, you aren’t the only person I’ve read with responsiveness issues; but it ought to be manageable, even if that means throwing more hardware at the problem. I’d be surprised to see any shared hosting platform capable of resisting the Slashdot Effect ™ but that’s really neither here nor there.
Something extremely funky: when I click on one of the links up top (“Webcomic”, “D&D Campaign”) the spacing of the links shifts around. I’m using Firefox.
EDIT: I can in fact edit, but now it’s a button inside the comment box rather than a header, and I only have 5 minutes to edit. And it’s counting down as I type. MUST TYPE FASTER
animeblogger.net used SuperCache a lot before the migration to WPMU. It does help, especially versus CPU consumption. That said, it’s not without problems: other plugins driven by requestor data get confused due to being run in a different context. CopyFeed obviously fails like that. Not sure about postfactum editing, that might be affected too.
Have you considered a host that allows you to have a dedicated virtual server? That would give you much more control over the workings of the thing, as well as give you the ability to track things like CPU cycles. Depending on the host you’d also be able to do things outside of your web root, like monkey around with PHP configs and stuff.
It’s less expensive than owning your own physical box, but you get most of the same advantages. Most of the enterprise computing world is switching to VM because it scales so well.
Forget about needing a server with 128GB of RAM and 24 processor cores…just get 5 small VMs and a load balancer.
Can’t offer any help, but congratulations on becoming so popular that you have to deal with such headaches.
Comment editing re-enabled, time limit re-set to 15 mins.
$20 a month for a server is indeed a deal. Thanks for the heads up on Linode. I think that’s too much server for me. (I’d get lost in the deep end if there was trouble. My knowledge of the Linux shell was rudimentary ten years ago, and I haven’t used it since.) But I had no idea that sort of thing was so affordable. I’ll have to poke around and see if I can find some sort of happy medium.
I’m not sure about your server permissions (and I’m assuming it’s Unix running apache etc.), but in php, something along the lines of eval(“uptime”); should give you server CPU load stats. I think.
Amirite?
Shamus, for what it’s worth, i’ve read in many places that it’s very very very hard to absorb the wave of interest from a high traffic site like digg or reddit.
You may be interested on how to absorb the digg effect: http://www.ndesign-studio.com/blog/updates/the-digg-effect/comment-page-2/
I wont put up an exhaustive list of links, but at least the above link may guide you to interesting articles (there’s one aimed at wordpress specifically)
Good luck
Be aware that having a vserver means the same need for maintaining and administration like a “real” server. You need to keep it uptodate and secure. With a normal shared hosting account, your provider does that for you. :-)
Out of interest: How much bandwidth are we talking about? And do you know what uses it?
Yes, gzip will help a lot for text content.
You are using PNG a lot, even for content that would work well with JPEG or a limited colour palette. Examples:
http://www.shamusyoung.com/twentysidedtale/wp-content/themes/TSGood/images/dice.png
Your file (PNG): 222 Kilobytes
JPEG at 85-90: 30-40 Kilobytes
http://www.shamusyoung.com/twentysidedtale/wp-content/themes/TSGood/images/d20.png
Your PNG: 32 Kilobytes
JPEG: 5 Kilobytes
http://www.shamusyoung.com/twentysidedtale/wp-content/themes/TSGood/images/logo.png
PNG: 50 Kilobytes
JPEG: 10 Kilobytes
The “wavatars”:
Half the size if you use PNG with a 256 colour palette. A third the size if you use JPEG.
Same for the little dice next to comments.
So if the images are the bandwidth eaters, you can easily erase 2/3! The visible difference is none or so minimal that no-one would notice unless he compared PNG and JPEG next to each other.
If you want so help on this, just send me a mail. This kind of stuff is very easy to do in a batch in Linux. ;-)
Might I suggest Dreamhost? I may be somewhat biased since it’s run by guys who graduated from my college, but it’s still one of the best colleges in the country and, unsurprisingly, they set up one of the best hosts on the internet. I seriously suggest you check them out, because they actually can withstand being slashdotted, even on their shared hosting, and they’re cheap.
Well as a reader since the second installment of DM of the Ring, I must say your site has (almost) always been here for me. Everything looks good from my view. Sorry if nothing more to say, other thank keep up the good job.
For what it’s worth, I don’t care about the Wavatars, and in fact, they seem to have some kind of problem loading for me… usually some of them are sort of chopped off at the top. They get chopped off in a different place, or not at all, if I scroll them off the screen and then back on. I might be in the minority here, though, since I use Opera.
I think most of us wouldn’t mind the fat being trimmed. We come here for the content, and that content isn’t the Wavatars or the dice rolling.
I’m a little sad now that you’ve upgraded, because my previous “cool” avatar has been replaced by a different one (since the wavatar implementation at gravatar.com uses a different email address mangling algorithm than yours did).
Maybe now I’ll have to consider actually setting a real avatar :) (Although this one isn’t too bad…)
(Incidentally, my website uses Hosting Matters too.)
What is this dice rolling you are talking about? I can see that the page loads something every couple seconds (because my browser tells me so), but I cannot see any change…
Current Opera.
Edit: Also, my Wavatar changed! :(
So, now that you’ve made your updates, all you need to do is testing. That means you need to write something witty or interesting enough to get on the front page of slashdot. Now get to it!
Write “How hardcore sensibilities destroy our favorite past time” or “Why DRM will eat your babies” or “What color paddle should we use on fanboys”
There’s new wavatars? I wonder if my new one is still as bad as my old one.
@Octal, they’re sometimes chopped for me too using Opera, but today they seem fine.
Edit : New wavatar : I don’t know if that’s awful or awesome… I can’t decide!
If you keep an eye out, you can find some real dedicated and virtual dedicated deals out there from time to time. I recently picked up a virtual dedicated server with 1 TB/month bandwidth for $5/month, and I’ve had a lowish-end dedicated server for several years for less than Hosting Matters’ G3000 shared hosting package. A good place to monitor is Web Hosting Talk Forums.
Whatever you do, do NOT go for Dreamhost. They might be nice for a couple of weeks, but once things go bad they will lie and/or be incompetent. For example I had them tell me the reason why a site had a “speed” of ~40 Kilobytes/s was “because the distance from the USA is so much and hey, webbrowsing sucks with a ping of more than 200 anyways”…
(This being my second post wrt Dreamhost, I’m starting to feel like a shill–I wish I got paid for this.)
@31 Hannes: That’s interesting… out of the many people I’ve heard of using Dreamhost (and, again, remember that these guys come from my college, so I both know a lot of people hosting through them, and know a lot of people biased towards them), you are the first complaint I’ve ever heard with regards to them… it really makes me wonder what the actual cause was, though, since that does sound like a terrible cop-out (not just a cop-out, but a terrible one).
Heh, looking back on my support history I laugh. Those transfer speeds were always confirmed with people in other (mostly european) countries.
I moved to Dreamhost in mid 2006 and was completely happy for the first months. For that price, what I got was insanely nice. Paid 2 years in advance because it was much cheaper.
Jan 8th 2007: Transfer speeds of ~100 Kilobytes/s.
Dreamhost’s answer “These speeds seem normal, coming from the fact that they are traveling long distances, depending on different type of systems from hop to hop, possibly losing packets through each transfer. I’m not sure if there is much we can do here.”
Time until it was better: >24 hours
Jan 27th 2007: Disk quota of the file server exceeded (not me, the whole server).
Feb 19th, 2007: 20 Kilobytes/s
Support: “DreamHost does not limit, cap or restrict connections speeds in anyway. I would recommend talking to your ISP about this issue as they provide the physical connection between your computer and our servers.”
May 24th, 2007
Server is so overloaded that I cannot even get a normal (apache) directory listing.
Support: “We sincerely apologize for the problems you experienced due to our DNS issues. Site downtimes, including mysql and webmail, email sending and receiving and delays due to mail server issues are now considered resolved. Please note, mail still may be delayed a bit as our servers catch up.”
May 25th 2007:
uptime: 13:29:08 up 1 day, 23:07, 7 users, load average: 17.87, 17.64, 17.18
May 30th, 2007: Still the same issues, after some “yes, blahblah” support answers, this gem:
“After looking at the ping and traceroutes, this is not an issue of server slowness, but of the connection slowness from DE to the United States, and as such there is nothing I can do to assist you in this matter.”
And moments later the site was fast again…
On my reply how I don’t feel that a ping of ~200ms from DE to the US affects webbrowsing that much (and that this is rather unrelated), the answer is “Hm, that’s rather odd. I personally get annoyed with the internet at more than 85ms.” ;-D
Jun 5th, 2007: The problems reappear. I can ping my site, but neither do I get any HTML output (from a perl script) nor can I login to SSH.
Jun 18th, 2007:
After uptime of “01:42:51 up 11 days, 51 min, 10 users, load average: 13.09, 16.19, 15.94” the server is not responding anymore. Very fast reply this time though and the server is fixed 5 minutes later.
Jun 25th, 2007:
Misconfigured Apache, “The requested URL / was not found on this server.”. No websites can be accessed.
Jan 11th, 2008:
Load usages of >10 again.
May 9th, 2008:
Unaccessible everything. Fixed 4 hours later (other user was eating all resources), not too bad.
Oct 17th, 2008:
Load average >100… Fixed ~3 hours later.
And that’s when I decided I could not reliably and successfully run a website with Dreamhost. They oversell/-load their servers. They are cheap, both the price and service.
Concluding I had 10 times where my site was no usable or accessible (that I know of) with no guilt on my end. One time this lasted more than a week and it was generally not nice to use the site for about a month. Their support responses were not satisfying most of the time, downright insulting/stupid occasionally.
If you ask around, not many Dreamhost customers will recommend Dreamhost if you want a reliable website. At least that’s what my woes showed me.
Now I am back with my german host, pay the same but only get 10% of the webspace (still more than enough, 20GB), support answers in <5 minutes 24/7 and the server is fast like a frightened dear. :)
Does your host use CPanel? Mine does and it lists the CPU utilization in the main CPanel page.
Well, regarding monitoring your cpu usage and such, I’ve always maintained that real hosting services give you a unix shell account to manage things.
I use rahul.net’s DirectAdmin services, which give me control over everything as well as a shell account (which I use for developing PHP stuff). After having used CPanel for a couple years, I much prefer DirectAdmin. They aren’t an ISP reseller either, they have their own peering agreements with backbone providers.
Pricing is reasonable too. 1 GB of disk space with 50 GB monthly traffic for $17/month, and they are meticulous about keeping backups.