Do Not Panic

  By Shamus   Mar 3, 2013   169 comments

Yes, the site theme has changed. No, this is not a permanent change. I’m just mucking around trying to figure out why the site is so ridiculously slow. I’ve fussed with the database, I’ve turned off all my plugins, I’ve checked the error logs, I’ve scoured the WordPress help pages, and I’ve said swear words in a loud, authoritative voice. I’m basically out of ideas at this point.

SOMETHING is making it incredibly expensive to show pages. It gets bad when lots of people are commenting. It gets horrible to the point of uselessness when I add a new post. We’ll see what this does.

EDIT: Initial test: Inconclusive. I’m going to let this sit for couple of hours and watch performance as comments appear and I edit posts.

EDIT: Well, I stripped the site down to the bones. Default theme, no plugings. (And then, just plugins to rebuff spam.) The site loaded fine, but posting comments still takes over a minute. Actually, the comment goes up right away, but the person leaving the comment gets stuck waiting for the page to refresh. The same thing happens when I dd a new post or update an old one. Basically, submitting a form results in an epic page load for no reason in the world. It could be the form. It could be modifying the database. It could be rebuilding the affected pages. I don’t know.

There are 235,000+ comments on the site. That’s a lot, but in the grand scheme of things they shouldn’t clog up the works that badly.

I really am at a loss. There’s no reason for this. I contacted my web host and the machine itself seems fine. No rogue processes eating CPU/memory/bandwidth. It’s just another stupid WordPress blog.

EDIT: On the advice of the tech support guru at my host, I dropped the Super Cache plugin and switched to W3 Total Cache. The difference is pretty stark. I’m hopping around the admin pages with a speed I haven’t seen in over a year. Comments seem to be working well. We’ll see how this goes.

Thank you for your continued patience.

A Hundred!2020209Many comments. 169, if you're a stickler


1 2

  1. Indy says:

    Faster but not by much. Nothing to say that it was a change in theme that improved it.

    EDIT: This post took 80 seconds to go through.

  2. Zukhramm says:

    Too late, I’m already panicking.

  3. MrGuy says:

    “Sheriff, his is no time to panic.”
    “This is a perfect time to panic!”

  4. Daemian Lucifer says:

    Whoa,that was jarring.

    Ok,lets see if this helped.Timing the comment posting now.

    • Dragomok says:

      I know this is not actually related to site’s performance, but up until now I have thought that the demon in your avatar is male.

      • Daemian Lucifer says:

        Yeah,I chose a succubus way back when HOMM5 was about to be published.Before that,I had an ifrit.This way,I support the notion that all demons are actually genderless,no matter how they appear.

  5. Shamus says:

    Testing this for myself.

  6. newdarkcloud says:

    Testing.

    EDIT: Took about 2-3 minutes.

  7. X2-Eliah says:

    Best. Theme. Ever.

    Edit: Eh, still quite slow. No change, even. Imo.

    Wat a bit and see if it gets slower as the comment count grows, maybe?

  8. EagleEye says:

    Take a picture of this and add it to the post. The site looks hilarious like this.

    Also, what WordPress plugins do you have installed?

  9. AJax says:

    Can anybody edit the comments after posting?

    • Indy says:

      The user who posted them (briefly) and moderators i.e. Shamus.

      • Indy says:

        Oh, you meant now that it reset. Nope. And I can’t edit my post to remove this wrong information.

        • MrGuy says:

          I just edited a post (the post below, from one dot to two.) So post editing is working for me (or, at least, was working a few minutes ago).

          Or it could be that, in playing around with the site, Shamus might be turning features like “post editing” on and off looking for the slowness…

          Edit: If you can see this, editing posts is STILL working for me…

  10. Aquarion says:

    You might find handy http://www.tracelytics.com/. I’ve used it at work, basically you install an Apache mod on the back end and some inserted JS on the front end, and it watches the PHP code’s execution model to see where it spends most time (rendering, database calls, whatever) and then gives you pretty pictures about what’s wrong with your site.

  11. Brandon says:

    I am happy to time a comment. It will be a fine excuse. I take it this isn’t a Gravatar/Wavatar thing like the last time you went hunting for a bottle neck? I guess that was page rendering, not comments, so never mind.

    –And about 1 minute to post.–

  12. Wulfgar says:

    hey, it can stay forever. i like this look.

    over 60s to post a comment

  13. vamphri says:

    Posting has never really been an issue, but I should mention that for me the die cast auto plays. Im not sure if that is a quicktime plugin setting on my end or if its universal though…

    EDIT: 40s to post and 20s to edit.

  14. Shamus, I don’t know if this is relevant, but when I visit this old page showcasing the first episode of Spoiler Warning, I see the old theme. Maybe something in your archives is to blame for the slowdown?

    If not, I think the solution is to post more SW and Diecast episodes until the problem clears up.

    • 4th Dimension says:

      You need to clear your cache or Ctr+F5 reload that page. You are getting the cached version.

      • I did, though now it’s irrelevant because he’s changed it back.

        Though it does give me another non-progammer idea: What if something like those old Viddler links is doing this somehow? Some little bit of code that’s phoning home or having a hard time getting the zeroes and ones it thinks it needs?

  15. Tomato says:

    Looks like the mayans were right after all.
    I know adding more plugins on top of this mess might not be a very bright idea, but have you tried WP-Optimize? I’ve been told it’s pretty good for cleaning up the inevitable garbage that piles on a long-running WP install like yours.

  16. Mike Riddle says:

    FWIW.. The initial page display was MUCH faster than before..less than 2 seconds versus 20 seconds

    comment took 44 seconds to post

    first edit took 1:45 to post

  17. Attercap says:

    What’s the DB backend for the site? Have you tried any database cleaning and/or optimization?

  18. Tsi says:

    I’m from France and it took only a few milliseconds…

  19. slipshod says:

    The front page loaded much, much faster.

  20. Ravens Cry says:

    It seems a significantly faster, all things considered.

  21. Thomas says:

    So much white, what is this? The ending to Human Revolution?

    EDIT: That comment went through smooth

    EDIT EDIT: The edit, not so much

  22. Ciennas says:

    Really scared me for a second there with the everything disabled button. Thought the site had been rpossessed by hobo squatters, like they do elsewhere.

    Instantaneous loading, though. Just swap the trolley picture with something familiar.

    Anyway, to test comment posting in this mode.

    57 seconds from my end.

  23. Tony Kebell says:

    Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaahhhhh!!!!!!!!!!!!

    (1:30, approximately)

  24. ClearWater says:

    For a moment I thought I’d ended up on the wrong site.

  25. KMJX says:

    Is this the twentysided blog? Am I in the right place here?

  26. wheals says:

    Whoa, yet another theme

  27. Mintskittle says:

    Test comment is testing.

  28. Even says:

    Welcome to alternate testing reality.

    By the way, is it correct that http://www.shamusyoung.com/twentysidedtale/?page_id=16473 and http://www.shamusyoung.com/twentysidedtale/?page_id=16458 show the original theme with ads and everything? Because that’s totally happening in my Firefox at the moment.

    The latest Dishonored post shows the previous site theme with the picture of a tram and the one before that still rolls the original theme.

  29. bloodsquirrel says:

    IT’S THE ENDTIMES! REPENT! GIRD YOUR LOINS! SMOTHER YOUR CHILDREN IN THEIR SLEEP TO SPARE THEM THE HORROR!

    Also, are you guys doing another Diecast? I was perfect happy in life before them, but now I feel angrily entitled to more.

  30. Also testing. The blog just looks kind of zen now. Makes me feel a little more at peace.

  31. krellen says:

    The page still loaded pretty slowly for me. Since the front page seems to load just fine – it’s only subpages that are really slow – I suspect the comment database is to blame.

    Incidentally, this page seems to be loading with the old theme (I tested it in three different browsers, so I’m pretty sure it’s not a cache problem on my side.)

  32. Johan says:

    The moment I saw the site today I PANICKED

    But then I saw your post

    Slow site is a common theme, thought it is always interesting to see followup posts on WHY it’s slow, I seem to remember one time it was the Date and Time eating up everything?

    Either way, good luck

  33. ShamusLoggedOut says:

    Yet another test.

  34. ShamusLoggedOut says:

    Posting comments is still taking in excess of 5 mins for me.

  35. anaphysik says:

    “Hilariously” (and I do use that term lightly here) the current theme breaks youtube embedding. I wonder what quirks the other test themes may produce.

    • anaphysik says:

      (Oh, and Shamus followed @DisclosureAlert. *SQUEEEE*. Now if only Aldowyn would edit the eps like he should have a week ago, so that we weren’t stuck with me figuring out ways to produce outtakes <_<)

  36. ShamusLoggedOut says:

    Hmm. That one was less than a minute.

  37. ShamusLoggedOut says:

    Another test.

  38. Charles Henebry says:

    I hadn’t noticed it running slow.

  39. According to a website speed test ([url]http://www.webpagetest.org/result/130303_WD_EQF/1/details/[/url]), 4 of your swf previews eat most of the speed.

    I had a similar problem with my site, when getting data from twitch.tv took 21 of the total 23 seconds loading time. Maybe look into that?

  40. Helpful Lurker says:

    I’m a lurker, but the page has never been slow for me. However, I have adblock (turned off for your site) and Stop Autoplay for Firefox. I’m commenting now to see if I experience a delay, and also if culling the slow from the not-slowed might help us figure out the cause.

  41. Stacy says:

    Testing comment function.

  42. Stacy says:

    Testing comment function again.

  43. Bryan says:

    But, but, but… no large friendly letters on the cover! No cover to speak of! How am I supposed to not-panic in these circumstances?!

    Ahem.

    Not knowing much about the system that processes an HTTP POST and spits out HTML, the fact that normal page loads are running at a reasonable speed, and the DB entries that users’ posts turn into are showing up if the page loads in another tab, I have to assume that something in the POST handler that runs after the DB insert is causing grief here. But I don’t know what it would be (never run php myself).

    Maybe grab a backup copy of the scripts, then add some logging to wp-comments-post.php? (Debug by printf, basically?) I’m assuming that’s the one that’s taking forever to return the redirect with the fragment-id of the new post in it. And since the post shows up on other-tab loads of the page, I assume (without knowing for sure) that the post-redirect GET is taking a reasonable amount of time to finish. Especially if that’s the same URL (except for the fragment-identifier) as a “normal” page load, though now that I say that I don’t know that it is. Hmm… going to try something with this one, hang on…

    • Bryan says:

      …Nope, same URL after the redirect after making a comment (except for the fragment identifier, which doesn’t actually get sent to the server) is the same as before the comment. So yeah, it has to be the wp-comments-post.php script before the redirect gets flushed to the client, and after the database insert, I suspect.

    • Bryan says:

      …Nope, the URL after the redirect after making a comment (except for the fragment identifier, which doesn’t actually get sent to the server) is the same as before the comment. So yeah, it has to be the wp-comments-post.php script before the redirect gets flushed to the client, and after the database insert, I suspect.

  44. File this under “spitballing and hoping something sticks.” A few searches for this kind of problem suggested disabling something called “subscribe to comments.”

    If that helps, huzzah. If not, well… the dumb suggestion was free, right? :)

  45. Also, isn’t your site keeping track of how many posts everyone’s login name has made and displaying them as dice? What if you shut that off? Maybe it’s hanging having to go through the entire site’s comment DB to get those totals.

  46. newdarkcloud says:

    One thing I’d like to add is that sometimes I’ll post a comment, and the comment counter on the home page will go up, but not on the permalink for whatever article I commented on.

    Then, after some seemingly random time period, the comment will be posted and the comment counter will be what it should be.

    Refreshing the page doesn’t seem to help, so I’ve no clue what’s causing it.

  47. This might be a dupe-post, but the first attempt didn’t go through.

    Could the problem be the comment counter for each user? If the site has to go through the DB and tally how many comments we’ve made before our posts go up, could that be the source of the bottleneck?

    As further “evidence” (though it’s only anecdotal), when the comment counts have been missing their graphics or only appeared to count the ones in this post, the site seemed to load a LOT faster.

  48. Jace911 says:

    Am I the only one who’s worried that Shamus is going to start going through the site’s source code line by line and find OBEYOBEYOBEYOBEYOBEYOBEY stuck in there somewhere?

  49. JPH says:

    doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof

  50. Jokerman says:

    Just posted a comment on the last SW episode and after about 5 mins i get a message saying…

    “Duplicate comment detected; it looks as though you’ve already said that!”

    So its trying to post multiple comments, at least when you post with Chrome.

  51. StashAugustine says:

    … and keep a towel with you at all times.

  52. Deadfast says:

    The whole site is now slow as hell for me Shamus. It takes up to 20 seconds for the server to tell me that die*.png doesn’t exist.

  53. Erik says:

    Have you tried using XHprof? (https://github.com/facebook/xhprof)
    Its originally designed by facebook and gives you great insight in whats taking howmuch time, what paths are executed and howmuch memory is used.

    Helped me out a lot when trying to find performance issues in 6 million lines of code :)

  54. Rax says:

    Is it okay to start panicking now?
    This is the state of the site on my end as of about 2 minutes ago:
    Screenshot on imgur
    (note that it claims the gw2-post and the “do not panic”-post have finished loading)
    Those still working tabs at the bottom have been like this for more than 5 minutes (without posting any comments obviously). Other sites load that number of tabs fine so it’s probably not a problem on my end.
    Between doing the screenshots and writing this two of those tabs have resolved to tell me they couldn’t connect to the databse, the other tabs seem to have loaded fine (except for the comment-counter-picture-thingy and avatars)

  55. Rax says:

    The following comment was written about 20 minutes ago, about 10 minutes after hitting “Post Comment” the site collapsed completely, I really like this “error”.
    It seems to work again, so let’s try that comment again, everything in it still holds true:

    Is it okay to start panicking now?
    This is the state of the site on my end as of about 2 minutes ago:
    Screenshot on imgur
    (note that it claims the gw2-post and the “do not panic”-post have finished loading)
    Those still working tabs at the bottom have been like this for more than 5 minutes (without posting any comments obviously). Other sites load that number of tabs fine so it’s probably not a problem on my end.
    Between doing the screenshots and writing this two of those tabs have resolved to tell me they couldn’t connect to the databse, the other tabs seem to have loaded fine (except for the comment-counter-picture-thingy and avatars)

  56. Hieronymus says:

    I’ve noticed that when your site is loading, it locks-up Firefox for a couple of seconds (most noticeable by opening new (empty) tabs). I’d been assuming it has something to do with the embedded videos.

    I’d try two things:
    1) Disable objects (or however you have videos/ads set up to load).
    2) Take a look at how your MySQL database is set up (or have someone who is more familiar with them do it, I HATE MySQL). If everything in your database is stored in the same chunk of table, then things would become exponentially slower for every addition. If this is the case, consider making several, smaller tables to increase efficiency.

    • Neruz says:

      Holy crap what did you do?! The latest post has vanished into thin air and comments aren’t nested anymore, huargh.

      -EDIT-

      Took about 2 – 3 seconds for the post to happen.

      -DOUBLEDIT-

      Okay now everything is back to normal. What the hell is going on?!?

  57. X2-Eliah says:

    Site takes about a minute or more to just load an article page (no hot comment-on-comment action).

  58. X2-Eliah says:

    Also, hey, these tools may or may not help. Just in case you weren’t using/checking these stats out.
    http://www.webpagetest.org/result/130303_EW_J2K/
    and
    http://tools.pingdom.com/fpt/#!/LWe7C0paf/http://www.shamusyoung.com/twentysidedtale/do-not-panic/

    (there’s more tool-pages like these, ofc)

  59. Even says:

    Enter the testrix.

  60. MelTorefas says:

    Page took a very long time to load for me, much more so than usual. Now, lets see how this comment goes.

1 2

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *

*
*

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!