So when I mentioned that I was out of work, a lot of people emailed me. It really made me aware of just how many people read this blog. Even more importantly, it made me aware of just what a wreck my entire email system is.
Some of this is my own fault. Back in 2000 when I registered this domain, it seemed like a good idea to make a catch-all email address and use it for registering on other sites. If I created a forum account at wombats.com (for all your Australian marsupial needs!) then I’d register with [email protected]. And for a couple of years, this solution worked really, really well. But at some point spammers began dictionary attacks where they would just spam [email protected], so instead of getting one spam I’d get dozens. Then this domain went from being an obscure little thing to something popular. The links from Slashdot, Make, Digg, Stumbleupon, FARK, etc all told the bots, “THIS SITE IS IMPORTANT. YOU SHOULD CARPET-BOMB IT WITH SPAM.”
At some point I finally switched from using Outlook Express to Firefox. That helped a great deal, and the adaptive spam filter that came with Firefox did a great job of culling most of the chaff. The levels of spam rose over the years, but the spam filter ate enough of it that I wasn’t too concerned. Eventually checking my email took a long time, though. I have broadband, but I’d end up sitting there for almost a minute just like when I was on dial-up. I was getting hundreds of emails a day, but the spam filters were eating 95% of them.
And then there was the additional idiocy that (at the time) AOL, Hotmail, and the other free email services couldn’t seem to get their act together and put a stop to this. I realize that stopping spammers for good is impossible, but this is the easy stuff. New users sending out thousands of emails with image attachments and obvious keywords in the subject line? That’s a spammer! Ban them! Morons! This was another bit of damage from the dot-com boom: That technology services run by idiots had enough cash to keep rolling even when they actually inflicted overall damage on the net and provided terrible service.
And then the final outrage that Outlook would show me these stupid images as I tried to dig through and find my legit email. (Firefox has a button to “show me the images in this email”.)
It was such a cyclone of destructive stupidity. Lots of people had to be very incompetent for a very long time in order to make a mess this big.
I’ve also had my email address – [email protected] – on my website here, which probably added a good bit of fuel to the fire. I was eventually forced to abandon my catch-all address. Now if I want to return to my account at wombats.com I’ll need to create a real wombats@shamusyoung email in order to recover my password. Ah well. Hasn’t been too much of a hassle so far.
But even with the catch-all gone, the levels of spam rose and rose, and I think at some point the adaptive spam filter in Thunderbird began to malfunction. I’ve actually known about this for a while now. I knew I was losing the “occasional” email from strangers, but I didn’t realize how bad it was until I started seeing comments from people “did you get the email I sent?” Uh… no?
And now that I’m looking in the spam trap, it looks like the filter is worse than I’d feared. I’m not getting as much spam as I used to. I just did a count for yesterday, and I got 3 legit emails, (Wow, only 3? Okay.) and 158 spams. But it marked one of the legit emails as spam and I had to manually deal with about 50 of the real spams. So yesterday my spam filter missed a third of the spam, and ate a third of my legit email. Those are lousy, lousy numbers on both sides.
Maybe that catch-all address just broke the filter. Maybe the media stuff I get confuses the filter. (I get a lot of press release-style stuff from marketing firms that are 50% boilerplate text.) I don’t know. But this is unacceptable.
I have a private email I use for business and family, and that never gets any spam. But I need a public email so that people can contact me through this site. I really hate to ditch [email protected] because I still get a lot of important email there.
The upshot to all of this is that if you emailed me and still don’t have a reply by the end of today, I haven’t read your email. I apologize. I don’t know why this has to be so hard. I’m working on it.
EDIT: And yes, when I said “Firefox” above I meant “Thunderbird”.
The Witch Watch
My first REAL published book, about a guy who comes back from the dead due to a misunderstanding.
Another PC Golden Age?
Is it real? Is PC gaming returning to its former glory? Sort of. It's complicated.
My Music
Do you like electronic music? Do you like free stuff? Are you okay with amateur music from someone who's learning? Yes? Because that's what this is.
Trusting the System
How do you know the rules of the game are what the game claims? More importantly, how do the DEVELOPERS know?
Quakecon Keynote 2013 Annotated
An interesting but technically dense talk about gaming technology. I translate it for the non-coders.
Shamus, did you get my reply?
/joke
:-)
Good luck, seems like managing a high-profile email adress is a big lot of trouble..
I laughed.
(well, before you purge your Junk Mail folder: I DID send you a reply, however. I just don’t expect you to answer back. :o) )
Shamus, did you get the gold I emailed you?
I never got a reply, either time I emailed you. Should I just try again?
As a suggestion, try changing your email link to something like this:
Mail to <a href=”mailto:[email protected]?subject=PASSWORD (add your subject here)”>me</a>
Then you can change your spam filter to NOT delete anything with subject containing PASSWORD.
All I can say is that the spam for adrr.com hit 100k a day before I gave up on my e-mail address there. Carpet bombing every possible name combination was going on.
I would really like to see ping back verification initiated. It would require three e-mail pings for every one, but would immediately stop spam (every e-mail that comes in has to contain a ping back code to verify the sender, and only gets through the filter once a valid ping back response has occurred. That can be automated).
There should also be delisting of urls connected to spam.
http://www.youtube.com/watch?v=z_tMm_3xobE
Even Better:http://www.youtube.com/watch?v=6CIv0QGs4y8
I know that trusting your data to a third party can be a bad idea sometimes, but I’ve found that Gmail’s spam filters are downright excellent. I’ve only ever had to fish out a legit e-mail on a few rare occasions. For public use where you won’t have any critical data at stake, it’s a great choice. I’m just not a fan of Google having access to everything I say or do…
Gmail’s filter is good to the point that I am utterly shocked when something slips through. Though when something gets through, it usually brings its friends. Seems like I’ll have a week or so where there is spam every day. Then Google fixes it and no spam for months.
I use gmail too, and I receive about 20 to 30 spam a day, it catches them all. I think it learns from other users too, so if someone mark a mail as spam other won’t receive it, if it’s identical.
It’s free, has tons of space (7,5 gb), and it’s really fast. And it shows little text google ads (if you check on the web), so you won’t be distracted by all the images and useless ads.
I’m really happy with it.
We use it in our company. It serves our “corporate” email – they are not @gmail.com addresses, mind you- you can set a @yourdomain.com email address served by gmail.
They do it for free and, as others are saying, it works great. I used thunderbird before, until I noticed that the gmail web interface was actually simpler to use – and it saved me several GB on my hard disk.
And we receive no spam at all (mostly because being work addresses we don’t really use them to register onto pretty much anything)
Do this. Now.
I sign up with my gmail address to nearly everything on the internet, and these days My inbox gets about one spam email a month, while Gmail’s junk folder gets 20+.
There is no good excuse for not using Gmail.
There are, in fact, MANY good reasons for not using gmail. I do use it, but some don’t and they have their reasons. Sheesh.
I am looking into Google apps, so I can Google-filter my shamusyoung.com addresses. But I can tell this is going to take time, a resource of finite supply. Like, I’m not able to “Do this. Now.” Thank you.
what everyone else said…
I’m amazed how well it works, actually – spam is just not something I think about *at all* anymore
Sometimes you can forget that Gmail is even catching spam, especially when the spam folder auto-empties every week or so.
You could move to a white-listing approach, at least for your public email address. When someone not on your white list of addresses sends you a message, the message goes to a spam folder, and an auto-response says “Hey, I get way too much spam. If you really want to send me the message, reply to this email and I’ll get it.” and include some kind of code in it. When you get that reply from that address, it comes through, and you can read it, and add that address to your white list. Spammers won’t bother replying to that kind of thing, so you’ll never see their email. It makes emailing you for the first time a pain, but it prevents you from getting any spam, and also prevents real messages from getting lost. There are white listing systems out there. They work great in some ways. On the other hand, if you expect to use that address for any kind of automated responses, say from amazon or what have you, you might have a problem unless you know the exact address they’re coming from.
The “prove you love me” email systems work great if you only ever expect to get emails from human beings. The instant you use them to sign up with a system that you ever want to receive email in the future from, you’re done, because a web app ain’t gonna reply, and even if it did, it probably would only say “Sorry, this isn’t a monitored email address” without your magic cookie code included. And whitelisting the first email address that comes when you sign up isn’t reliable because the next mechanism down the line is 60% likely to mail with a different return address on it. So, Shamus ends up having to keep everything that traps anyway, just in case he needs to go rummage through it to find out whether something he signed up for just stopped or if it changed the address it sends from.
I have a spam trap address. Anything that shows up in both that address and my regular one is deemed spam and processed by the bayesian filters for future reference. The only problem is that it’s hacked together and I don’t know where you’d find a real, distributable one.
Seconded. I’m guessing his host provides Box Trapper for free as part of his hosting package (I do for my clients) and it’s pretty easy to set up. Legitimate remark from Peter below about bots, but just use a different address for them. Spam may still come in, but’ll never miss anything from a real person, which is the important thing anyhow!
I got it.
have the filters look at all the emails on the blog-post-responses, and use that to generate a white-list. Inform everyone that if they want to send you an email, they should respond to your blog (even just a little bit).
I am a gmail user, but I can understand why not to. number one reason: they stood up to china and came out with an even partial victory. they stood up to a government that (nominally) controls and represents 1/6 of the worlds population.
thats danger if I ever saw it.
Do you mean “Thunderbird” when you say “Firefox”?
Yes, he does. Confused me too. :)
Has he been watching the archives of a certain UK television puppet show recently?
Thunderbird is Mozilla’s email client. It’s generally paired with Firefox. There’s also a wacky extension you can get that randomly shuffles the names of those things in the…bar at the top of the window that tells you what it’s called. (what is the name of that bar)
I’m typing this in Mozilla Dreamskunk.
“Title bar”
Yeah. My bad.
As someone has already commented, you could use GMail for its excellent spam filter and nice web interface. You’d have two options:
1. Forward all mail to a GMail account; or
2. Set up GMail to download all your mail via POP.
The only bad thing about this is that you’d need a Google account if you’re one of those who’re against their innevitable world domination :)
There’s actually a third option: set up a Google Apps For Your Domain account. I’ve been much happier since I switched my e-mail to a Google Apps account.
Stable e-mail servers with excellent spam filters, for free? It was a win-win decision, for me (I had been hosting my own mail server).
I second the Google Apps standard edition. It’s free, has lots of cool toys, and integrates seamlessly with your domain.
I need to find out more about this. Getting gmail-level filtering for my domain. I would pay for this.
I also use Google Apps Standard Edition, 7Gb of mail space and google’s spam filter! All for free. Looky here: https://www.google.com/a/cpanel/domain/new :)
Also check this piece of code, it might not block spammers but it sure annoys them xD
http://officeofstrategicinfluence.com/spam/
The killing spam idea is funny. Especially that that example-bot downloaded 1 gig of email adresses. Imagine how many adresses that must be… a text file of 1 gig?
But I fear that most crawlers will recognise the site soon. The site should also create an unlimited number of fake sites (or spoof a wrong adress if that is somehow possible) so simply blocking the main site doesn’t work :D
Then Google would say you’re crazy! But yeah, it’s a good way to go, and like others have said, it’s free, too.
I use it for my domain, and I recommend it to my smaller customers. It’s perfect for the situation you have here.
If you want any help setting up or testing it out, feel free to contact me.
You can pay for it, if you want ;) The for-pay service gives uptime guarantees and phone support, and it also lets you set up more than 100 mailboxes (users), but so far, the free service has been extremely reliable for me.
In fact, you could set up POP with [email protected] so you get all that mail in Gmail and then IMAP that to Thunderbird if you want to use a desktop client!
And that way you’ld get both GMail’s spam filtering, and Thunderbird’s spam filtering as a backup! (That’s also what I currently do for my GMail account. I’m dumb enough to run my own mail server for my home account, though.)
(Oh, and ob-disclaimer: My day job is working on Thunderbird.)
Ah, so that was the problem. No worries. We found some other guy to pay six figures to work on our new procedurally generated RPG/Tetris/Comicbook hybrid project.
That’s so incredibly cruel and yet I find it highly entertaining, possibly because of this darned emotional sadistic streak :-P
Pics, or it didn’t happen
Oh, it happened, and it’s amazing. We can’t wait to show it off to you. (Still working out a few bugs, unfortunately.) :)
I know what you need for your problems: pills to M4KE U B1GGR LOL.
Just kidding. You have to wonder who the people are, who apparently make e-mail spam a profitable venture. Someone, somewhere out there, sees an email like that and says “why, yes, I DO want some cheap V1AGA, stranger I never met”.
That is the creepiest thing about spam, IMO. I try not to think about it.
There have actually been a few times I’ve wished I knew who those spammers were so I could toss them a couple of nickels for entertaining me, though. Every once in a while I get bored and read through my spam and while most is moronic I get the occasional entertaining one. Like the one with the subject line “Supersize your Fry” that included an excerpt from a Shakespearean sonnet. Because, y’know, Shakespeare was HUNG. Or…something.
That reminds me… I recall one particular spam that quoted a snippet of the Wheel of Time which mentioned Nynaeve’s name. I have no idea what the spam was trying to sell, but I still feel vaguely offended that they’d use the Wheel of Time to try to get through filters.
“Take the blue pill, and find out how deep the rabbit hole goes.”
I still like that one.
Eeew!
That, right there, is a really sad thing. A good deal of people who respond to spam are the same people who call 1-800 credit card and direct marketing numbers. Most of the time, they are desperately lonely people who crave ANY kind of attention from another human being, even if it’s for buying some Viagra or a Scrub-o-matic 3000, or applying for yet another credit card they have no use for.
I’ve some friends who manned call centres and received the occasional call like this; the orders from management are usually to cut the call short to free up the line, but that kind of person is vulnerable enough that they’ll offer to buy something for five more minutes of human contact (or, in this case, some email messages that look like they are from another human being). Terrible thing, really.
It’s people like the guy in the following article.
http://www.uniset.ca/other/wsj_spam.htm
*sigh*
He’s a 45 year old grandfather. Yikes!
45? Um, have first kid at 22, kid has first kid at 22… 45 is not a big deal. GREAT grandfather at 45 is a big deal, yes, but not grandfather.
I’m not sure which I hate more, the spammers or the idiots like Mr. Soto who gives them money.
Congratulations, jerk, for every moment of enjoyment you get out of that spam-habit, there are a hundred people (or more) who get a moment of their life wasted.
I really want to use a word stronger than “jerk”. grr.
Apparently for every moment of Mr. Soto’s enjoyment there are about 10,000 of us spending a minute pissed about spam.
The economics of spam mean that even if your success rate is 0.0000001% you still make money on the basis that if you send enough crap someone will respond. It doesn’t happen with mail because mail has a cost whereas spam is essentially free, either buy or create a botnet, set to email *@*.* at a rate of thousands if not millions of e-mails a second, wait for those hundreds or thousands of hopeless dupes to pay, win. If it wasn’t working no one would be doing it, same concept as every other scam if you try it one enough people it will eventually work. Scummy but I personally dislike those who scam people face to face (faith-healers and such) far more than the robo-scammers of the web.
While I agree with you as for why spam exists I don’t think that “If it wasn't working no one would be doing it” is a very good argument. After all virus exists and there’s not much money in that (that I know of at least). I think part of the reason for why spam is so widespread is because you don’t have to be all that competent to make it.
Very true the low cost of entry makes it very easy to get into. A lot of viruses may be minor variations created by script kiddie trolls looking for bragging rights but there are those who profit from creating novel viruses. Those viruses are the work of those looking to install data harvesting apps or create zombie machines for later use in DDoS attacks and both of those can be profitable through extortion, blackmail or spam.
oh… My argument seem to have been defeated ^^
Well, in all honesty, this site is rather important.
I can imagine why spammers would want to carpet-bomb it.
I don’t know if it’s legit numbers of not, but some rating websites I’ve used calculate about 40,000 uniques/month. I don’t know how much of it is bots, however.
That’s big.
They probably want to try and put a stop to all the horrid puns that come from here.
Understandable really.
If that was the case, I would imagine someone would be breaking the Geneva Conventions all over Rutskarn’s site right about now.
In an odd coincidence, Rutskarn and Jibar are testing a Rainbow Six LP over there. And sure enough, hostages are involved!
FWIW if you run your own email server SpamAssassin does a great job of filtering spam these days. Between its Bayesian filtering and the hardcoded ones I rarely get a spam outside my spam folder and very rarely get a ‘ham’ in there.
I am guessing that the majority of legit email comes from people that have left comments, so the set of all comment email adresses would be a fairly good basis for a whitelist.
Not knowing your setup, I can only point to mine: Self-hosted, using Qmail-Toaster which gives me (among other things) SpamAssassin with some RBL filters tacked on. It’s only set to actually-delete anything with a ridiculously high “spam score,” and tag the rest. Then I let Thunderbird shunt things into a “Spam?” folder if it’s over a certain minimum score.
It ain’t perfect, but I actually “see” a vanishingly small amount of actual spam and nobody’s asked me, “Didn’t you get that email…?”
Just give gmail a try. I had a handfull of address that where getting way too much spam (at peak, one a minute), but then I just forwarded them all to gmail. The only false positives their spam filter has given me over the years were from automatic website registration/password reset emails that I was expecting. Plus the web-interface / search features are so nice. (Note: I don’t secretly work for Google… yet).
Good luck with the job hunt, I feel your pain.
For me Thunderbird sums up most spam in “quarantine reports” which just shows the name of the emails. I can deliver them if I want, and usually a good email slips by. But it does a good job of catching anything thats spam.
Thunderbird’s junk filter worked great for me until one day it just… stopped. Stopped marking anything as spam, no matter how many times I tried to manually “teach” it or switched it on and off.
Same thing happened to me– my gmail account gets none but boy my personal account (the one I used before gmail) gets ton and thunderbird barely notices anything wrong. Todays included “This will give you the male power” which made me giggle. And I have barely any spam compared to Shamus– his inbox is crazy.
By the power of Greyskull… I HAVE THE POWER!
I use a catchall (and have for quite a long time). I use Gmail to process the incoming mail.
On the catchall side, I manually black-list any addresses that start getting spam (turns out a lot of emails given to specific companies leak… somehow). This seems to keep it manageable.
The worst is when someone starts using my domain as a “From” for spam. That’ll up my spam rate in the hundreds to thousands per day – spam in the sense of bounced messages.
Gmail handles it all pretty well, but it does false-positive sometimes, so I still have to check.
** EDIT
Speaking of companies leaking my email address… I just found a fake Battle.net phishing email sent to escapistmagazine.com@mydomain…
I recieved a similar e-mail. That’s the Escapist’s doing?… sigh.
No, it’s someone claiming to be the escapist, knowing idiot’s don’t check that anyway, and hoping to prey on 13 year old who don’t know better, if they do check it.
Could well be. I signed up to the escapist and all of a sudden my junk folder starts filling up with battle.net phishing attempts.
It might not be that they sold it though. They could just have terrible security, neither is a particuarly great outcome.
hmm I got a few of those in my spam folder too.. a couple of weeks ago looks like. Don’t really use battle.net though, so *shrug*
I feel your pain, Shamus. A while ago, I was responsible for sifting through my company’s catch-all, looking for actual clients’ emails that were sent to a misspelled email address. It was nasty, nasty business. I remember I started charting how much spam I received each day. When I started it was about 100 per day. By the time I stopped nearly a year later (because the inbox on the catch-all was so full my email client timed out when connecting to the ISP) it was over 3000 per day.
I remember one spam sender would try to fool the non-existent filter by putting jokes in the emails, in tiny font at the bottom. I don’t remember how I discovered this, but I started looking for those emails and collecting the jokes. The plus side was I did assemble a pretty good joke book, which I still have on my computer, of the jokes that I got from spam emails.
Concerning Gmail, I must provide a contrasting POV to the others who talk about how good it is. Gmail catches about 100 messages a day in my spam folder, but at least 10 get through. I do, however, have a unique problem. For some reason, I think other people forget that they don’t own my Gmail account, and often register for things with my email address. I don’t know why this is, as I don’t have the problem with any other email address. This creates a lot of spam that isn’t caught because it’s not really spam, but legitimate email, just sent to the wrong address. Examples of what I’ve received: shipping receipts (with address); airline and rental car reservation confirmations; break-up emails; pictures of people’s dogs, sent via SMS; and some poor woman from Ohio thinks I’m her son and sends me family newsletters.
You get mail from Aunt Sue, too?
(joking)
This is really ‘hilarious.’ I’ve often thought about whether that happens (imagine someone maliciously flooding your inbox with verification e-mails, etc., by using your e-mail to register places; the horror!), but I’ve never seen something like that in action. This seems particularly bizarre since it seems to have been done absentmindedly.
Could be a similar name. Joe Cool up there might have a deceptively common name, like I do. A few months ago, I had some lawyers practically beating down my phone trying to get a hold of “me”, based on, I’m guessing, their flawed assumption that my name was so rare, no one else in the state could possibly have it.
The person they were actually looking for lives about 50 miles away from me in a small town separate from the metropolis I live in. And I found this out simply by searching for me AND the other guy they were looking for, instead of just me alone.
When I actually spoke to the fellow on the phone, it took me nearly half an hour to convince him I wasn’t the guy he was looking for.
I have a very common name, and there’s some local guy with my name who apparently doesn’t (or didn’t for a good while, anyway) pay his medical bills… or give his medical providers any (accurate) info on where he lives, etc. That’s been fun. :-/
And then there was the time I was pulled over by the police, and they had a warrant for “my” arrest. Luckily, the officer who pulled me over was very professional about it, and checked my drivers license and asked me my middle name, etc, and determined that I was not the guy they wanted… man, I’m glad that was before the handcuffing, booking, etc. I had my kids with me and everything.
I’ve never been so happy to have such a weird name. I went through most of my life thinking I was entirely unique. Turns out according to Google that there are three of us. Three people on the entire interwebs with my first name. And neither of the other two share my last name.
I hate my name, though, and have since grade school when no teacher could figure out how to pronounce it on the first day. Still, after hearing this I’m glad of my weird name. I have enough problems with wrong numbers. Wrong names sounds like a nightmare.
Your problem isn’t so unique, I get the same thing. There’s a guy in Nebraska who keeps giving out my address as his (we have the same name). Either he’s just moved to Toronto, or there’s someone else out there who has started doing the same thing.
The annoying thing is that I know the Nebraska guy’s real e-mail address, and it’s a misspelling of our shared name, so there have been a number of times that someone who wants to e-mail him has noticed the misspelling and “corrected” it, so it gets sent to me instead of him.
I’ve received enough personal information about his activities that I could make things really difficult for him if I wanted. Luckily for him I’m honest.
I like how you add the correction at the end of the post, but don’t actually remove the original mistake.
Your Fable 2 posts from a year ago still contain the Lucien/Lucian confusion.
Just saying.
Also, the mail ostensibly from gmail or hotmail accounts almost certainly has forged headers. Free mail providers rate limit pretty ferociously, as you can imagine.
This is possible because SMTP was designed without any authentication at all; because why would you want to pretend to be sending the mail from a different address? You wouldn’t be able to read the replies!
The same designers presumably went on to implement a RSH server that stored login passwords in cleartext, and habitually left their car keys in the ignition, because it’s more convenient that way.
I had the strangest spam the other day. The header was forged to look like it came from me sent to myself and some other random address.
There was no information about what it expected me to do just information that a cheaque I’d apparently sent was canceled because ti was a duplicate.
Of course it was all US based which means it’s noting to do with me, with companies I’ve never heard of.
Spams to you from yourself were (still are, maybe?) very common… there’s no “from” address to check that would get it caught in a blacklist (I’m assuming that’s the theory, anyway).
I haven’t thought about the spam “problem” since the last time I checked my gmail spam folder. I’m willing to let Google have my private data (and my unborn children) in exchange for not caring about spam.
I use Google Apps. All mail to the domain comes to my mailbox, and there’s also a few[1] @gmail.com accounts that forward to the main account. This way I can easily be wrong on the internet without compromising my anonymity.
[1] 5 and counting. Don’t judge me, bad posting isn’t a crime.
Hmm… I’m using Opera for reading mails, and the learning filter works really well. But I think the scale of the problem on your side requires a server-side solution.
So unless your provider will let you plug a filter into the mail server (or offer ´to do it for you), it’s probably worth going to some mail provider who does that.
I’m no on the US market, so I can’t comment much, but around here pretty much any mail provider has server-side spam filters, because otherwise the systems would simply break down. And anything that goes through this, the filter in Opera can catch for me. I got about two or three spam mails this month that were not recognised right away. You can also add rules by hand. Oh, and you can also have incoming mail sorted autimatically, which would work nicely with [email protected] (I actually had to look that site up, but was somewhat dissappointed, I had hoped it was real), so then you can sort anything that wasn’t sent to your personal adress, or sort them by site and so on. Opera is also usually not displaying pictures, and you can delete an e-mail without even displaying anything of it, if you recognize it as spam by the header.
Yeah, but server-side filtering would probably still be best. Do you run the server yourself or does your provider do that for you? In the former case, there’s certainly something to be had for free that would help you.
Speaking about spam, but otherwise unrelated: I just posted with a wrong e-mail adress, but it still worked … what is the adress actually used for by the comment system? Anything besides gravatars? And who gets to see it?
Anyone with admin access (like me) can see the email. To my knowledge, it isn’t used for anything besides avatars.
I’m with the guys suggesting RBLs – they work. Filters like gmail also have the advantage of being subject to a ridiculously huge number of spam mails, so they can better “train” their filters. I have a strong suspicion that those kinds of services use many honeypot accounts to identify spam, so it can be positively flagged before it even reaches the filter stage.
If you wanted to use a gmail filter without switching your entire system to gmail, you could look at postini.
Another useful tool is greylisting. This will temporarily reject mail the first time an email is received from an unknown sender. For a legitimate mailer daemon, it will see a temporary failure and queue the message for retry a few minutes later. When the greylist filter sees the retry attempt, it will allow it to pass (and possibly whitelist that host/address for future use). Spammers don’t operate traditional smtp servers, and almost never handle failures elegantly, as doing so would not be an efficient use of resources. The down side is that whenever the greylist encounters a new host/address, the first mail from that address will be delayed by one retry cycle (typically 5m-15m for most default configurations)
In handling the same issues, I’ve gone a slightly different way.
I’ve configured my mail server to aggressively discard anything that doesn’t look legitimate. For me, the most important rule is that any e-mail transaction which doesn’t begin with a correct “HELO” or “EHLO” line gets immediately denied. This one rule all by itself, I’ve found, catches about 80% of all incoming spam and I’ve never had it discard non-spam. And the great thing about this is that it allows the server to cut off the spammer before they’ve even sent the body of the e-mail; this results in dramatically lower bandwidth and CPU usage, compared against client-side filtering.
I’ve also configured my mail server to discard anything claiming to be FROM my server, unless that user has logged in and authenticated. This rule catches another 10-15% of the spam. For a while, though, it did break Paypal notifications. For some reason, Paypal used to insist on sending you e-mails with your own e-mail address in the “FROM” line, so they used to be discarded by this rule. But they don’t seem to be doing that any more, so there’s really no reason not to refuse to accept mail that alleges to be from yourself, if you’re not actually authenticated to the mail server. As above, this happens early enough that you can cut off the spammer before they’ve sent the body of the e-mail.
Finally, I pass any still-surviving e-mails through SpamAssassin, configured to use (amongst other heuristics) about six different RBLs, SPF, etc. Anything which gets a SpamAssassin score greater than fifteen gets immediately discarded. Anything with a score greater than five goes into a “spam” folder. Anything else goes through to my inbox.
I find that with these three steps, I end up with only one or two e-mails tagged as “spam” per month, and virtually no actual spam getting into my inbox. Additionally, I’ve never had cause to believe that it has incorrectly discarded an e-mail.
Of course, all of this requires having server hosting where you can configure your own mail server. Don’t know if that’s the case for you Shamus, but if so and if you really want to make a major dent in the amount of spam you receive, denying e-mail from people presenting malformed (or unverifiable) HELO lines has made a huge difference for me, and cutting off anyone who claims to be me (but hasn’t authenticated) made a noticeable improvement as well.
Further discussions on the issues around blocking e-mail based upon HELO/EHLO lines is here: http://faculty.cs.niu.edu/~rickert/cf/bad-ehlo.html
Might changing how the email address is written on this site help? In addition to using whatever other spam reduction methods you use, perhaps choosing a less obvious email address, and writing it differently from a regular address ( “Twenty(at)shamusyoung.com”, for example, rather than “[email protected]”, and possibly using something less obvious than twenty, or shamusyoung). would reduce the incoming spam, making the filters easier to handle.
(Or perhaps it wouldn’t be worth doing, or wouldn’t work, but it’s a suggestion all the same.)
Also looking at the other comments, you guys get way, way more emails than I do.
This won’t help with spambots that go for [email protected], but it might come in handy for lots of forum registrations and the like.
A little-known feature of gmail is that you can alter your username by adding “+” and anything else you want after that, and it will all come to you. So say your main e-mail is [email protected]. You can sign up on different forums as [email protected], [email protected], and [email protected]. Gmail recognizes all of these as [email protected] and so they’ll all return valid on any test. They all come in to your regular itsme stream, but you can apply filters on them if you want to label-and-archive or autodelete or what have you.
You can also use it to see where spammers are originating (that darn MemeMeBaby forum!) or who sells your e-mail address (!), or where Interesting People of the Internet Who E-mail You Out of the Blue have seen you post.
I think it’s an interesting and useful feature, and keep meaning to use it more.
S.
Actually, that’s not a feature of gmail, but a feature of emails in general: http://en.wikipedia.org/wiki/Email_address#Address_tags
Some sites’ e-mail verification routines do not recognize “+” as a valid address character and will bounce it. It’s nice when it works, but too many places puke on it to make it useful for everyone.
One that hasn’t been mentioned is good ol’ Spamgourmet.
Found at Spamgourmet.com They describe themselves as the molitov cocktail of Spam defense. What they do is you create an account with them and never ever go back to the website, unless you like checking the counter they have of how much spam is eaten.
How it works is you link it to a real email, then use your account name instead.
say your Spamgourmet account name was JimBob.
You want to download something from somewhere that you know will spam your arse and you use it like so:
[email protected]
Someword is a word you haven’t used before,
The number is the amount of emails you want from them, max 20
What happens is that they automatically create the address when they receive the email. You do nothing but receive the email. after a set number, all consecutive ones will be eaten.
Of course if the basic protection is not enough, as in someone knows your account name, then they could in theory spam you by directing twenty emails and use a bunch of words. Then the advanced takes it and lets you build either a list, or a keyword that must be in the list to be allowed.
As for myself, my stats put a smile on my face.
Your message stats: 413 forwarded, 83,350 eaten. You have 152 disposable address(es).
Something to think about.
I second spamgourmet.
I wouldn’t use it to receive e-mails that might contain sensitive information (I don’t know how much I can trust the people who run the service), but for most one-time or unimportant uses it’s great.
In fact, the address I’ve used to comment here is a spamgourmet address.
Holy crap! Thank you! I’ve been manually creating new gmail accounts for every bullshit account I’ve had to create. I will definitely be using spamgourmet.
And this goes double to Shell for the [email protected] thing. I did not know that either.
Just tried address+whatever@gmail and it didn’t work as I expected. I have email accounts that all forward mail to a single private account that only I know about. +whatever went to the sub account but didn’t forward it on.
Still useful to know though.
During my Twenty Sided Tale archive binge about a year ago, I sent Shamus an e-mail about the music I’d listened to while reading Free Radical, since whenever I listen to anything I switch my music around to fit the mood. In retrospect, Shamus probably has better things to spend his time on, which is what I assumed was the reason I never got so much as five words in response, but this post makes me think it’s quite possible he simply never received it at all.
This excerpt explains just how profitable such enormous scales can be:
“Using the Storm botnet for 26 days, the scientists were able to send out 350,000,000 emails touting their on-line pharmacy. Due to factors such as invalid addresses and blacklists, 82,700,000 emails made it to computers. Spam filters further reduced this number significantly (though harder to measure). Of those emails making it to a person's in-box, 10,522 users clicked on the link and visited the fake pharmacy. Twenty eight people initiated a purchase averaging $100. At this point, the pharmacy returned an error message, thus preventing the researchers from actually obtaining names and personal credit card information. This came to a daily income of $140 for the campaign. Since the infiltration amounted to only 1.5 percent of the overall Storm network, this translates to a potential revenue of $3.5 million a year for an internet pharmaceutical company using Storm for spam marketing.
In another portion of the study, the researchers used their Storm infiltration to determine how many PC's they could capture to propagate further spam. They sent out 82 million emails advising recipients that someone had sent them a postcard, which could only be viewed by downloading the “postcard” software. Extrapolating their results, they estimate that Storm self-propagation campaigns can recruit between 3500 and 8500 new computers a day.”
Source: http://www.consumerwarningnetwork.com/2008/11/14/spam-or-scam-how-do-they-make-money/
A separate study, showed that sending out spam purporting to be a tip on stock options could influence the stock markets and make large revenues for the spammer.
And this is why I posted what I did up above about hating the spammers but also hating the people who RESPOND to spammers.
Have you people realised how much spam is now a nuisance, a natural part of our lives, integral part of the internet culture. It’s not something we can be outraged about, simply a silly bother that we have to clear every day.
I’m wondering how that cultural part of our lives is going to progress. Am I going to be teaching spam-avoiding tricks to my grandchildren? Is the moral jugement required to avoid spam will have to be transmitted from a generation to the next?
15 years into the mass internet only. What will it be at 45?
I’ve sent a few mails to you over the years I’ve followed your site. I never got a reply. I’m not so in love with myself that I need validation for every random email I send. Should I have expected a response? They could have been TL;DR or just not that interesting to you. They weren’t important emails. I don’t respond to that kind of stuff from my own family, let alone strangers but I’m trying to figure out what’s normal for you.
Do you normally reply to that sort of stuff? I have no problems if you thought it wasn’t worth responding, but it’s rather annoying to think a filter ate it without being read.
BTW very few contained random pictures of genitals.
In the interest of science, how many is “very few” and can you forward those on? ;-)
If you want to keep hosting your own email, and you want to retain more control than gmail, you might consider Tuffmail. They’re very good – they even offer greylisting!
The only real solution I’ve thought of for this would be to implement some kind of email charge… which is crazy at the consumer level, but might work at the server level. That is, servers charge each other to receive email.
So, server A sends 100 million emails to server B this month. Server B sends 100.2 million emails to server A this month. Since the net is below some threshhold (say, 1 million), it just goes away.
But if it’s over a million, net, well… a penny each, maybe? Sure, $.01 each is peanuts, but businesses that make $140 per 350 million emails would be out of business (YAY!).
There are some serious practicalities to be worked out, there, but it would work. (Well, it wouldn’t stop zombie networks, but those guys are committing some serious crimes already, so it would be good to go after them for that, anyway.)
I used to have a system set up where I’d type the magic command “mailregister wombatscom” and it would set up forwarding for [email protected] to my main one for a week, and then automatically delete it. Then I did just like you describe, using a different address to register for every site – wombatscom to register for wombats.com, etc. so they were easy to derive later. Because lets face it, you probably don’t care about anything they’re going to send you unless you lose your password, in which case you can just register it again. And in the meantime, that address sends you no spam at all…
This doesn’t, of course, help you with the problem of spam from addresses you actually wanted to USE, but it helps cut down on the crap from the ones you aren’t.
(Also, I don’t really get the reference to MALL time in the title. Obscure joke? Or typo?)
I don’t know if this is an option for you Shamus, but given your association with the escapist, have you considered piggybacking on their IT solution?
If they use some sort of big evil spamkiller device(not going to plug the one my company supports, and it’s probably well over what you’d want to invest in just for your blog), perhaps you could have them add your domain to that device.
It would increase your dependency on escapist, though. But maybe something to consider?
I guess that explains why you never responded to my mail then. :p
Oh well, I forgot what it was about.
Shamus, did you forget the first rule of Email?
THE FIRS RULE OF EMAIL IS TO NEVER POST YOUR AN ADDRESS IN PLAIN TEXT ON THE INTEWEBS
No, seriously, you should probably remove [email protected] address from this post, or mangle it somehow. The same people who carpet-bob you with spam, also likely crawl the website with a parser that tries to ID email adresses to add them to the spam list.
Oh and also:
s/Firefox/Thunderbird/g
There, I think that fixed it. :)
I do the same thing as you Shamus, I have a domain and I create a new email address for everything, I now have around 300.
One of my addresses is published on a website, other than that, they are not publicly known. I get very little spam, maybe one or two a day.
The absolute FIRST thing you should do is go through all your pages and obfuscate your published email addresses:
http://www.fingerlakesbmw.org/main/flobfuscate.php
It will not stop all harvesters, but probably over 99% of them. Also this will not stop spam coming from people who already have your address. But over time, if you do not load any external links from the emails (images, links, etc, this lets them know the address is active and its value goes up considerably, getting you on even more lists), you will greatly reduce your spam. Thunderbird helps you with this with the “Show remote content” button.
Apparently there is a WordPress plugin:
http://coffee2code.com/wp-plugins/obfuscate-email/
Also, when using Thunderbird and it classifies an email incorrectly, make sure you use the “This is spam” and “This is not spam” buttons. Thunderbird uses Bayesian filtering to learn what kind of spam you get, the more you train it, the better it gets.
http://en.wikipedia.org/wiki/Bayesian_spam_filtering
Oh, and I am one of the admins at a 200 staff government department in Australia, I have kept track of how many spam emails we get for about the last 2 years now, lately we have been averaging around one million spam emails per month, that’s 35,000 per day.
Now I know why my email didn’t get a response.
Cheers for the fantastic information covered here in your site, below is a little quiz for your blog website readers. What person cited the following quote? . . . .The best job goes to the person who can get it done without passing the buck or coming back with excuses.