T w e n t y S i d e d
Yes, the forums are down. Yes, I realize you can’t edit your own comments. Let’s talk about that.
On my Linux-based webserver, there is a user account linked to me. This “shamus” account owns all the files: All the PHP scripts to drive the blog, all the scripts to run the forums, and all the images and other random files that makes the site operate. Under normal circumstances, the entire file structure is designed so that only my user can upload, delete, and modify files.
However, you need to make some exceptions. For example, I run a WordPress plugin that makes weekly database backups. This plugin needs to be able to save these backups, which means that I need to make the backup directory writable for all users, not just the “shamus” userPHP, MySQL, and other processes are owned by the root user.. Otherwise, the backup plugin would run but it wouldn’t be allowed to save the resulting backup to disk.
So I need to make a few spots on the machine where processes not owned by me can put files. This alone isn’t enough to compromise the security of the machine, although it’s often considered something to be avoided if you can help it. The danger is that it may provide an attack vector for potential hackers. If there’s a vulnerability in either WordPress (the software that runs the blog) or PhpBB (the software that runs the forums) then they would be able to write files to these directories.
Here is a ficticious example of how something like this could work: Let’s say the forum offers a feature where users can upload their own profile image. You’re supposed to upload a JPG or PNG image file. These files end up in /forums/profileimages/. In order for this feature to work, I need to set the permissions of /forums/profileimages/ so that anyone can write to that directory. Let’s say the people who wrote the forum software didn’t do their job and the forums don’t make sure that what the user uploaded was actually an image. Like, maybe they uploaded a PHP script. This allows them to put new pages on my site, and those pages can do all sorts of nasty things.
Now, they can’t just put those pages anywhere. Those pages can only end up in /forums/profileimages/, and only the attacker will know about them. Once the upload is done, the attacker can then manually type in the URL like so:
shamusyoung.com/forums/profileimages/badpage.php
This will cause the script to run and do whatever it’s supposed to do. This doesn’t give the attacker full control over the machine. (They can still only put new files in directories I’ve had to leave open.) They can’t re-write the blog or attack visitors directly, but this is still an alarming situation that allows them to see a lot of stuff they shouldn’t.
This is a very simplified explanation. The actual method of attack is a lot more complex and to be honest most of it is beyond me. But this is the idea in broad strokes.
A couple of months ago PeterHe doesn’t comment often so you might not know him, but Peter has been providing technical and hardware support to this site for a long time. and I discovered some files on the site that were not owned by the “shamus” user. Files like this:
lprvpluh.php
pvkmnwoj.php
onrvyxwg.php
ukwwtgwx.php
Always the same pattern: A PHP file with a gibberish eight-character name, probably generated at random. These files contained highly obfuscated PHP code and were not part of the normal file structure of either WordPress or PhpBB. More importantly, they are obviously malicious in nature.
Link (YouTube) |
Peter and I have been battling this mess for the last month or so. We deleted all the suspect files, tightened up directory access, and then hoped we’d fixed the problem. Then a few weeks later the mystery files would show up again and we’d have to start over.
Last week the files showed up for the third time, and so we went to maximum paranoia level. We wiped WordPress clean and started over with a fresh install. We uninstalled the forums completely. This machine is now as locked down as we can make it. There are no directories with write access. This would break several of the WordPress plugins I use, but since I haven’t installed any plugins that’s not a problem yet.
If the problem returns, then I’ll need to contact my host and have them wipe the machine clean and start over. I’d hate to do that, since it would result in a ton of downtime. (The blog has about 1.2 gigabytes of images, and I don’t have a very fast upstream connection. That would be a long upload. Not to mention the time required to restore the databases and re-install everything.)
I’ve deliberately left out a lot of details on the off chance that the attacker actually reads the blogThis is unlikely. These kinds of attacks are often done by bots.. So if you’re thinking of asking, “Why don’t you guys just X?”, then keep in mind we probably did X but I’m leaving it out of this explanation.
So that’s why the forums are gone and all of our quality of life plugins are missing from the blog. It’s a known issue. We’re still investigating. If all goes well, then we’ll eventually get back comment editing and all the other little plugins we’re used to.
Footnotes:
[1] PHP, MySQL, and other processes are owned by the root user.
[2] He doesn’t comment often so you might not know him, but Peter has been providing technical and hardware support to this site for a long time.
[3] This is unlikely. These kinds of attacks are often done by bots.
Why The Christmas Shopping Season is Worse Every Year
Everyone hates Black Friday sales. Even retailers! So why does it exist?
Lost Laughs in Leisure Suit Larry
Why was this classic adventure game so funny in the 80's, and why did it stop being funny?
Deus Ex and The Treachery of Labels
Deus Ex Mankind Divided was a clumsy, tone-deaf allegory that thought it was clever, and it managed to annoy people of all political stripes.
D&D Campaign
WAY back in 2005, I wrote about a D&D campaign I was running. The campaign is still there, in the bottom-most strata of the archives.
Netscape 1997
What did web browsers look like 20 years ago, and what kind of crazy features did they have?
You should still contact your host and ask them at least what are they doing about the hardware vulnerabilities that have popped up recently.Specifically the spectre and meltdown vulnerabilities that were disclosed these days.Theres not much you can do about it,but at least youd be informed.
I mean, it’s not like THEY can do much either at the moment. At the very best, they might have a patch for Meltdown and one variant of Spectre, but I don’t think the chipset makers have even put out the microcode patches that Spectre requires just yet.
Although I’d say these vulns are so new the probably of them being both 1) actively exploited in the wild and 2) Shamus’ site getting whammied with them this soon is pretty damn small.
They arent that new,they are just newly leaked to the public.But yes,there probably arent many bots floating around that exploit the them.Still,its nice to know what to keep your eye on,and at least be prepared to burn down everything if the unlikely happens.
My understanding of them is that they can only be exploited by people who are physically at the computer.
This is emphatically not the case. Meltdown and Spectre are specifically most dangerous for computers shared by multiple users, such as web site hosts.
The actual impact of these exploits is that the attacker learns all information in the compromised computer’s memory, including everything the operating system would normally keep private (including the data owned by other users active at the time). The exploit does not allow the attacker to modify the target, though it might reveal sensitive information such as passwords if they are stored in memory, which could then be used to make changes.
Also, run updates. These are very fundamental issues with hardware design, but there are software countermeasures which significantly reduce the attack surface.
Weird, the spam checkbox is gone. It makes sense given this post, but still odd to see.
Nope. As long as the attacker is able to run code on the computer, they can in theory run the attacks. But for the average home user there isn’t much point: there are much easier vulnerabilities to exploit. It is more a problem for cloud computing services where you can use it to spy on data in processes that other people are running.
In theory it can be exploited from Javascript, which means it potentially threatens nearly everyone. Browser makers are shipping workarounds that hopefully render that path irrelevant. (In Firefox’s case, I gather they’re making the various timing functions less accurate as the attack relies on very precise timings.)
Isn’t the implementation very specific to the hardware involved?
Quite the opposite, for Spectre anyway. All x86 (and x64 by extension) systems are vulnerable. There’s talk that arm could be in trouble too.
Intel is more vulnerable to specific version of Meltdown (which itself is a subset of sorts of Spectre), but while AMD is a bit safer, it’s still not safe.
Part of what makes it so dangerous is that Alan is right. So long as you can run javascript on the system (through pretty much any browser for example), you can get access to data. Depending on the specific attack it might be very slow (everything needs to go through cache, and reading/writing from it for every bit of data is slow as hell), but that’ doesn’t save the victim much pain.
Problematic is that there’s no real patch that could fix it. It’s down to the way CPUs are built these days (running code before it’s needed if they expect to have to run it later, for example). There’s workarounds for current CPUs, but there’s more overhead, making them (marginally) slower, and it’s not exactly a real fix.
Shamus, I’m pretty sure i escaped the italics. but it doesn’t seem to work
I definitely escaped both italics and bold in this message. if the current sentence isn’t regular, the escaping doesn’t work. If it does, just ignore it. Hope it’s user error in this case.
That’s apparently a bit complicated; any code can use the exploits to load protected memory into the cache, but then they have to get that information from the cache to somewhere useful to them. Some of the methods they can use to do that do require physical access, but most definitely not all of them.
OK, but, like, why don’t you guys Y? :p
Y is so last year.In 2018 Z is all the rage.
No, we went grecian about Three days in. Theta Delta 9 is the true solution for all of lufe’s problems!
Shamus, y’all should go hexadecimal and implement option 0x19.
Execute “End of Medium”? That sounds ominous. I’d recommend 0x1D, “File Separator”. At least that keeps all the suspicious files separate. :)
I really hope everything sorts out. Best of luck to you and Peter!
Same, hope it works out.
Your host doesn’t offer FTP backup? If it comes to nuking the site from orbit, it’s probably time to find a better host.
Actually it’s more about how the web server is managed. If the host is going to force you to set world-wide writable directories in order to make WordPress function, you’ll never be rid of this problem. Linux has group level controls. As long as your web server process and the user that is executive any background processes are in the same group and the directory is owned by that shared group (regardless of user), then it’s writable by both without being worse by the internet at large.
If your host can’t help you set that up, it’s definitely time to switch hosts.
This is true. However, unless you’re running a dedicated server and have full root access, this isn’t the sort of tweaking you can do as a user normally. My Dedibox lets me create users and do basically anything I like (I run VNCServer with a desktop environment and WINE for certain programs I can’t run locally right now because I’m between homes and reliant on tethering). But my hosted accounts (where I have my sites) only let me set permissions per-folder using CHMOD, nothing deeper.
I don’t remember ever coming across a host who’d actually go to the trouble of setting up specific Linux users and other backend stuff before, but I suppose if you’re paying for a managed solution that should be something you can do? I’m on opposite ends of those poles, with both an unmanaged hosting package and a fully dedicated server, nothing in-between, so I’m not sure what that grey area in the middle of my own experiences is capable of ;p
Doing a lookup, I’m guessing this is the host – http://www.peak10.com – and presumably Shamus is using one of their managed services.
I suspect (and really, really hope) that the “writable by everyone” bit was just an oversimplification for brevity and the sake of the non-techie readers. Actually granting 777 to a website directory would be madness.
Wait, who’s Peter? Have I missed a person appearing? I feel like I’m in one of those “you remember the new guy, right?” TV show episodes where they try and convince us a new character has been around since the start.
As Shamus said, he doesn’t post much. I think I’ve seen him directly involved in a conversation maybe twice in the last three years :)
Ahh, somehow I missed the footnotes to this article I in first read through.
I’m going to do it anyways, just because I’ve seen _exactly_ those symptoms before. So even though you’ve probably already done it: did you disable XMLRPC? Like, down at the .htaccess level?
Yeah, thought so. But I had to ask; there was some canned push-to-hack script running around a few years back that exploited xmlrpc and generated filenames that looked exactly like that…
Carry on, and good luck.
Have you tried turning the forums off and on again? How about unplugging your PC? Yeah, that’s the limit of my expertise in coding.
I mean, I can do a couple of “If… Then” lines in basic, but somehow I don’t think it’d help.
I see your satirical hacking-related video and raise you a better version.
I was going to post that exact video! :D
I only recently found out about DoorMonster, and they are everything I never knew I needed in my life, and nothing more than that. In their 5 years’ worth of backlog, I don’t think they ever released a video (not counting community or announcement/organisation videos) that didn’t make me chuckle.
The context and quotation marks made me read this as “Sham-us”.
Also, I now feel completed to point out that I am not a robot.
I also am not a robot.
Though I may be an android.
Ok then,lets test the two of you:
THIS STATEMENT IS FALSE
Now,eaglewingz:youre in a desert,walking along the sand,when all of a sudden you look down and see a tortoise.You reach down and flip the tortoise on its back.The tortoise lays on its back,its belly baking in the hot sun,beating its legs,trying to flip itself over,but it cant.Not without your help.But youre not helping.Why is that,eaglewingz?
Oh, Maker! My life is a lie!
Don’t think I’ve ever had a client with a wordpress which didn’t get hacked at some point. It’s very popular with script kiddies I assume
Gosh darn it. I really miss the edit button. I mean, if things stay like that, I may have to start actually checking my comments for typos before hitting the ‘post’ button. And that would just be awful. That would almost be like work.
There have been studies. Changing the formatting of something makes your proofreading more accurate. Basically, when it’s the same formatting, your brain assumes “oh, I’ve already read this, let’s just skim.” A new font or background tells your brain “this is new text, better read carefully”. That’s why you don’t spot the typos until after you hit “Post”.
Interesting. I should implement the Comic Sans proofreading technique.
This must explain why, no matter how many times I reread my comments before posting, I almost always find typos after posting. Often multiple times in a row. I once got myself flagged as a spammer on a website for editing the same comment to remove typos too many times in a row.
If you really need to be sure a document ends up with no typos, print it out and proofread a paper version. Whilst it remains electronic the pesky 0s & 1s are free to jump around when you're not looking, but if you run off a hard copy they know they have to settle down and behave, else you'll have them rang to bights. Or … I'm pretty sure that's why the method twerks, anyway.
I can absolutely confirm this in my own experience. When writing on my own blog I can reread something in the editor multiple times and not notice anything, then spot multiple typos as soon as I hit the preview button. The ability to preview before posting something is a wonderful, wonderful thing.
Since I tend to deal with the lots of files problem somewhat often, may I recommend instead of using your personal computer/net connection for the site backup/restore process, instead use a VM somewhere else? Using (something like) Amazon to ssh/ftp/scp/rsync into your site means that you can leverage their storage and their connections (and glacier, so their backup) which could significantly reduce site downtime and increase options. And once you’re done with the instance, you don’t need to keep it around.
Hmm! That’s pretty darn clever actually.
I use Hosting Matters and have been hacked at least once. A directory full of Canadian pharmacy ads appeared on my machine. I deleted it and complained and nothing (that I know of) has happened since. But I just run static web pages with no DB, so not much to disturb there.
Hi. I don’t have much time before I have a ridiculously small amount of sleep left tonight, but here’s some advice from a sysadmin with programming, security and webmastering experience:
* almost always it’s some automatic worm that’s compromising your site, not a dedicated hacker;
* almost always it’s either vulnerable (outdated or unsupported) plugin you use (like 90%), vulnerable (outdated) version of WordPress (like 5%) or a vulnerable (outdated or unsupported) library in your theme;
* it’s counter-intuitive but your limiting write access to WordPress directory actually greatly increases the probability of a successful hack, as WordPress can’t upgrade itself automatically with these constraints;
* you will not be able to keep up with WordPress proper / Plugins / Theme security updates manually – you really should allow it to update itself and it’s plugins automatically;
* you have to clean your site very soon, or browsers will start to refuse to load it;
* you can check the modification dates of the malware files and compare them with HTTP server logs – more often than not the URL used for compromise will be there and can point you to the name of the plugin which was used to compromise your site;
* check your list of users and force a change of all their passwords – it could be overwritten by a hacker to ensure future access;
* if a site was compromised and a worm was able to run any code on site then it’s very hard to be sure what was changed by it to maintain access to the site – but based on these random named files it looks quite primitive, which is good;
* install your operating system security updates regularly and reboot the server regularly;
* install Wordfence – it will alert you if there are suspicious files in your WordPress installation and can check all your WordPress files if they are unchanged.
“you have to clean your site very soon, or browsers will start to refuse to load it;”
Not just that, Google will also delist it. Then you have to go through a mildly annoying process to get them to list your site again once you’ve cleaned it.
“WordPress plugin that makes weekly database backups”
If you are able to run CRON jobs on the server I’d suggest getting a “standalone” wordpress backup script (could be written in PHP even) that is run by the CRON job service and just get rid of the plugin (which is dependent of running when a user triggers it right?).
Once thing to make sure of (tat is easy to forget) is to place config/password stuff needed, outside the webroot (in other words, make sure hat the webroot is a subfolder on the server). That way someone can’t download a config file etc.
Sure you can use .htaccess to “lock” certain folders or files but it’s easy to forget/screw up.
Yikes! Chances are it’s small scale stuff, but you can never be too careful. I hope the latest measures manage to halt this stuff in its tracks, Shamus.
I can’t believe you put up with that shit. I couldn’t do it. I’d have lost patience years ago and thrown my monitor out a window.
If I had to host a blog I think the most complication I could deal with is throwing up a simple blog on something like GitHub Pages (e.g., Jekyll) and manage my comments with a 3rd party solution like Jeff Atwood’s (https://www.discourse.org). Just wash my hands of the hard stuff.
It can also be worth checking your own (plus those that contribute) computer(s) for any viruses. Came across this once where malicious scripts (javascripts) would be added to some free text form plugins in WordPress once. We found a pattern where the scripts would re-appear every time a specific user had done changes to the site. After cleaning up that computer we didn’t see it any more.
Would agree though with some of the other comments here that this is more likely than not bot(s) creating users and uploading files and then triggering them. A good thing to do with static resource folders is have them being served directly by Apache with mime types added (only image), that way the PHP files will simply download, not run.