T w e n t y S i d e d
I mentioned last week that this site caused a ruckus for my host. Incoming traffic in the form of a deliberate attack (unlikely) or waves of attempted spam had slowed the webserver to a crawl. This wasn’t just a problem for me, but for everyone else hosted on the same machine.
In an attempt to block the attack, one of the techs at my host started banning IP adresses. Well, ranges of IP adresses. Actually, huge blocks of them. Check it out:
 |
This seems… really agressive. Entire countries are blocked by this. I’m not talking about the usual hotspots of trouble like China or Russia. I’m talking about New Zealand and large parts of Austrailia as well. Big parts of the English speaking world.
When you see it blocking something like 203.0.0.0 to 203.255.255.255 it means nobody with an IP adress starting with 203 can reach me. There are about 16 million possible addresses in the range, although not all values will be used and often many people will share a single address. I’m not knowledgeable enough to say how many people might be behind a particular range, but the number is likely in the millions. And there are a lot of those sorts of entries in the list above.
Millions upon millions of people blocked, in order to keep out what might be a couple hundred troublemakers. (Not that I have millions of readers, my readership isn’t even a hundred thousand. I’m just saying that if any of those people try and visit, they won’t get through.) Each of those ranges is like a door. Behind the door is perhaps a hundred or so real readers, several million people who might stumble on the site someday, and maybe a farm of spambots that will cripple the site. Locking all of these doors is overkill, but I don’t want to run around unlocking them at random and get attacked again. When that happens, nobody can reach the site or a number of other sites on the same machine.
I’m still mulling over what to do about this. I’m pretty sure my host went overboard in blocking IPs, but without knowing what I’m doing, un-blocking them all will just invite another attack. If that happens, my host will just block them all again and I’ll be right back where I started.
Is it really that hard to hunt down these spammers and have them all killed?
Gamers Aren’t Toxic
This is a horrible narrative that undermines the hobby through crass stereotypes. The hobby is vast, gamers come from all walks of life, and you shouldn't judge ANY group by its worst members.
What is Vulkan?
What is this Vulkan stuff? A graphics engine? A game engine? A new flavor of breakfast cereal? And how is it supposed to make PC games better?
Self-Balancing Gameplay
There's a wonderful way to balance difficulty in RPGs, and designers try to prevent it. For some reason.
Bethesda’s Launcher is Everything You Expect
From the company that brought us Fallout 76 comes a storefront / Steam competitor. It's a work of perfect awfulness. This is a monument to un-usability and anti-features.
The Disappointment Engine
No Man's Sky is a game seemingly engineered to create a cycle of anticipation and disappointment.
“really aggressive” is putting it mildly. After 15 yrs of working in high tech and managing junior technical support staff for about 5 yrs, I can tell you that what you have here is a classic shotgun/fly situation. Whoever did this had no idea how to diagnose it so they brought down the big hammer.
Granted, this type of thing *is* tough to properly diagnose but throwing an m80 into a beer can is probably overkill.
Is this a person blocking these IP ranges or a piece of software? Perhaps the software is going bonkers…
What you could do is unblock these people, and have a redirect that tests whether they are spam candidates by IP number, and verifies that they don’t have the correct cookie set, and takes them to a static HTML page with a form where they need to “type the security word shown in the picture” which will then take them to a little CGI script that sets the cookie, and redirects them to your site again. Now they have the cookie set and won’t be redirected, problem solved.
This causes a bit more CPU load than a simple denial, but since you’re serving a very short static HTML page with the form on it, your web server can serve this effortlessly ““ unlike serving pages of your blog.
I run the Emacs Wiki. We have a list of denied IP numbers (can’t access the site), banned hosts (they are only banned from editing pages), we have a list of banned URLs (can’t save pages linking to URLs matching these regular expressions), we have a list of banned regular expressions (can’t save pages where any part of the text matches these regular expressions).
The category “may not even read the site” has very few numbers:
The list of “may not edit” is also quite short. [1]
The list of “banned URLs” is quite long. [2]
The list of “banned regular expressions” is quite short again. [3]
The point of this multi-layered defense is to stop spam…
I also have a counter active that kills the script with a “503 Service Unavailable” if there are too many processes running in order to prevent server load reaching the thirties and fourties, bringing it all to a crawling stop.
That would explain why I can’t access your site from home. My TLD as resolved from there is 4th in your table – yes, it’s Poland. Curiously, I have no problem with browsing it at work, since our external address resolves as .org.
A shame, and I do enjoy the site.
I was having a discussion with my organization’s webmaster last week. She was telling me that they pulled down all the captchas on the site because the cracking attempts were causing more headache than the spam.
Instead of captchas now they use a hidden form field. A robot won’t know that the field is hidden, so it will try to fill it out. To make it tempting they label the field as something like “website”. Human visitors cannot see the field, so they can’t fill it out. If any data is entered in there, it must be from a robot/spider/spambot, so the submission is trashed and the IP is banned.
Was there a corresponding jump in spam getting through your site on the day the server was getting slammed? If so it could be scripts trying to decode your captcha. I doubt yours would be hard to break, but the programs may be inefficient.
Didn’t you read the comments last week? Someone from New Zealand/Australia mentioned that you were linked by their “stumble-upon” link of the (a local phenomenon like Digg or Slashdot) that resulted in you getting millions of new visitor from millions of new IPs. It wasn’t an attack it was fame… and your server is now a smoldering pile of slag.
“Fame” is on the internet is like going from 0 to light-speed in a second and it ir rare that a mortal server can survive.
Oooh, I like that approach, Phlux!
Yes, that list of bans is crazy. Someone panicked.
Class C bans should be fine. Class B is overkill, class A is simply lazy (on the part of your sysadmin).
Get the list of spamming IPs and do some tree analysis on it, I bet you can find intelligent ban ranges very quickly.
Phlux: nice approach, I’d guess they’re using a ‘visibility: none’ rather than an actual hidden input tho?
Zack: I’ve been Slashdotted before. I’ve been slashdotted AND FARKED on the same day (years ago) and this problem, whatever it was, was worse.
It couldn’t have been fame – there was no corresponding rise in comments whatsoever.
Alexis: It looks like that’s what I need: A list of offending IPs. I don’t have access to that, though. The spam wasn’t making it through and I don’t have access to the log files. I don’t know how to tell where the spambots were coming from on my end.
Or… maybe you just need a firewall that’s better at detecting an attack???
Although IDS/IPS can be pricey…. but there are inexpensive options.
Good to see that I can access your site again, I was getting very, very sad.
Wow! This is very, very restrictive. They have banned all of Russia, Poland, China and bunch of other country specific tld’s that I do not even recognize. Ouch…
This is not even killing a fly with a shotgun. This is like killing a fly by carpet bombing the whole country with thermonuclear warheads.
I don’t know what to tell you though. I’d get rid of the most broad of those filters and see what happens. Maybe ask your ISP to be more discerning and selective when banning IP ranges and don’t ban whole countries. Doesn’t mean they will listen but if you tell them your revenue stream is hurting because of their banning policy they might be bit more careful the next time around.
I suppose this was the reason I haven’t been able to connect to the site for like a week.
Ah, this would explain why I (a NZer) was unable to access your site for about a week, yes? (Even though the site was only down for a short time.)
[SNAP! Vegedus]
Indeed, I’m also in NZ and couldn’t access the site for about a week. But it’s better now.
FYI, my current IP is in the 203 range.
Yeah, time to dump the lazy host. Blocking ip ranges means he doesn’t want to do any actual work. So I send the link to my friend in Denmark and he gets blocked. Great job.
So _that’s_ why I would sometimes get the 403 message when trying to read your posts from home! (I am myself in Moscow). Works from my girlfriend’s, though…
I’m sure we could hunt down the spammers if we organized.
The Spamhaus project http://www.spamhaus.org/ does a good job of tracking and listing (by name) the worst spammers and organizations in the world.
I’m picturing something like an A-Team style group of dedicated anti-spam tech nerds who go around the world destroying the worst of the spam operations.
“If you think you can make an easy buck by being a huge jerk online, you just might get the attention of the Spam-Team.” (Catchy 80’s style TV show music goes here…)
Bloody hell, they’re banning half of the EU, and significant bits of the Asia Pacific and Latin America/Carribean regions.
(and also at least one of the ranges reserved to IANA, apparently)
‘Excessive’ doesn’t even begin to describe it, really. But hey – if they carry on like this soon their servers will be completely secure. Although disconnecting them from the internet entirely would have a similar effect.
The blocking ranges are way to wide, I’d have to agree on that. Blocking entire A-nets is just pure wrong. They should at least narrow it down to specific ISPs. :)
At work I’m on the 213. network and cannot access the site at all. Luckily I can still access it from home.
Well, I have actually blocked Asia-Pacific in the past for customers of mine… Although, IIRC, the only time I did that was for a company that *could* only do business (legally) with people in the US, and I stressed to them that this would block all of their US customers while they were traveling, and that it might be a minor consideration, but they legally could do business with US customers in Guam, which gets its IPs from APNic.
So, the things that are wrong with this (well, some of them) are:
1) your potential “customers” can be (and are) anywhere that anyone could read the English language (or get it translated). If you’re a car dealership in Topeka, chances are very good that anyone coming from an address in that block list is not there to do business with you. But for a blog? That’s nuts.
2) They’re blocking top-level domains using reverse DNS. That’s a horrible idea. If you really want to block those countries, block the netblocks assigned to them, which are fairly static, or block AS’s registered in them, which is even more static, but reverse dns hardly means anything. So many ISP’s don’t set it up at all, and if I were a russian ISP, when I set my reverse DNS up, I’m not sure I’d have it say .ru.
3) They’re blocking top-level domains using reverse DNS. That means, they’re doing a reverse DNS lookup on every page hit. That just sucks. And they aren’t just doing it for logging, where you can delay the log write until the name resolves while serving the page immediately – they have to get the name resolution back before they can serve the page. So that’s going to slow things down for everyone, especially those unlucky enough to have an ISP whose reverse DNS ends up on a server that’s broken (I know some fairly major ISP’s where that’s the case)
4) use of something like TOR will sometimes evade it, and sometimes get blocked by it. Both of which are, IMHO, undesirable results.
So yeah, that does kind of suck.
If I were you, I would, at the least, demand some statistics. Get a copy of your logs during the attack, and see exactly where it was coming from. If it really was coming from all of those places, then it was almost certainly coming from a botnet, and if so, it was coming from everywhere, so the only really effective block list is 0.0.0.0/0.
Friend of mine in China couldn’t email her mother half the time, because one of the ISPs between her and her mother decided to block all email from China. (Because, apparently, only spam comes from China.) So on the days that her emails got routed over that company’s servers, her emails just vanished.
She’s more technical than I am (I last worked in networking when ISDN was the wave of the future) and was able to track down what was happening. Ironically, at the same time there was a big fuss about Chinese censorship in the US media, but apparently private censorship by businesses was OK. Her mother remarked that at least the Post Office didn’t just discard letters from certain countries without warning…
Well I’m in Australia, and I was blocked at work last week, but this week I can get through again. Maybe your host has scaled back on the blocked IPs?
They should do a rotating dropping of the IP blocks until they pin down the spam source country, then maybe start from there.
Ah, so that’s what happened. I’m from Australia and have been cheerfully clicking at work, but can’t access from home which was strange but now explained.
I’ve been gettting a weird message when I post a comment from work. I’ll give it a shot from work tonight and see if it’s related. I’ll see if I can get my work IP as well.
I’m in NZ and was never blocked, so it looks like you were only blocking some of our ISP’s – I’m on 60.234
Is it really that hard to hunt down these spammers and have them all killed?
they need to revive letters of marque and reprisal for spammers and span irradication.
honestly.
I can’t get e-mail through my domain (adrr.com) because of random “hopeful” spam (all four letter domains apparently get 70-100k a day in spam addressed to likely addresses that might exist on them. I had only one functional e-mail address, but the spam collapsed my host’s server and they closed the account, even though I didn’t see hardly any of it. Sigh).
But, if letters of marque and reprisal would issue for the hunting down and terminating of spammers, that would create an economy. I toss in $5.00, you toss in $5.00 and pretty soon there is $100,000 or more in the kitty to reduce some spammers to mouldering ruin.
/Sigh. Not sure that is exactly what we want either.
well i’m in china, but have no problem accessing you through a proxy
didn’t even notice for a while, because i’m usually running a proxy anyway to get to sites the chinese government bans, most notably wikipedia
and if there’s chinese censors reading this, don’t worry, i’m australian, i’m not being corrupted by any data i couldn’t just get from home anyway if i really wanted it… :P
Okay, I just tested it (the editing feature is very nice!), and I’m getting the 403, but only after I post a comment. And the comment gets posted, and I can hit back then reload to see (and edit) the comment if I like. I don’t know what my IP is at work. I suppose Shamus can look it up if he likes.
Finally, I can at least access TwentySided from work! Did admins unblocked some regions? We read your blog in Russia too, Shamus! :)
Aha!
That’s why I haven’t been able to visit (I’m in Australia). The RSS feed has been coming through my feed reader so I could read one paragraph – it was quite confusing and frustrating.
Er, obviously it’s working now.
I’m in Peru and my IP is in the 201 range, and I was blocked for a while, now everything is working perfectly. Funny thing is when I was blocked there were times when I couldn’t get the RSS feed, but I could access TwentySided! Something’s going on with those admins.
Hurray, I can read it again!!! After of two months of 403s… Please do not do this again. :(
Looks like I’ve been blocked. I keep getting a 403. I got here through anonymous proxy.
I can still see your entries through google reader but they only contain the first few sentences.
Yes, someone should really hunt down and talk really severely to whoever is spamming you.
(I wonder if I can comment through proxy. Yes, I can!)
I was blocked for two weeks too. Not fun. :(
I’ve also been getting 403’s when I comment, but if I refresh the page comes back and my comment appears. I’ve never had trouble accessing the blog.
Arghh, in Poland I am being blocked at home (403), but not at work (I work for American company, that may explain it).
my IP address is in the range of 83.175.xxx.xxx
My ISP address ends in ghnet.pl
I hope it can be put on the whitelist or something.
Actually APNIC should be blacklisted. The Austrailians can’t obviously be trusted to do any real work to stop spam in Russia, China, Pakistan and India. Australans are too lazy to even try. APNIC (Australian’s Pretty Much Not In Control) needs to be shut down and a group that will at least take the 1 second of time it takes to validate registrants email addresses. If the people who get addresses from APNIC want to access anything outside of their communist or muslim countries, use a proxy server from a country that at least cares about spam laws, and Internet decency. And don’t even ask why Guam chose to enlist in APNIC as it is neither Asia or Pacific, but apparently communist leading.
Hi
I’m really really sorry for commenting on this 2008 post, but I am a reader of your website since 2009 and now I see why I almost always get a 403 error when I try to access it. You see, I’m from Brazil, and to come here I must use a proxy – granted, it’s more like a hassle than a real problem, except that I can’t watch videos, it’s slow and sometimes it doesen’t even work at all. Anyway, if you end up reading this on the forgotten corner of your blog and want to help a fellow DM/gamer/game programmer, is there a way to solve this?
thanks =)