Email

This morning we had a common help ticket request. A customer had received an email telling him to pay up or his secrets would be revealed. The secrets concerned his sexual behavior. He was disgusted and wanted to never get such an email again. I don’t blame him.

The email was almost certainly a shot in the dark, hoping to hit someone unwary. This is what they do. Since few of us have the resources of Jeff Bezos to deal with extortion attempts, we will have to find a more practical way to deal with them than counter-attack.

Following is my response to this customer:

Other than throwing the email away, what did you want to do about it? If you have no reason to believe that the email is anything other than spam, there is nothing more to be done about it.

What the criminals do is send thousands of emails like the one you quoted. Perhaps 1 in 5,000 believe it and send them money. That keeps them sending more.

Email was not designed to run on the Internet. It was designed to be used in a single hospital. There was no thought given to potential abuse because everyone knew everyone else. Bad behavior would result in a person being regarded as a misfit and might even get the person fired. From there, email spread to universities. It was found to be a very useful collaboration tool. It was a collegial environment where misuse would result in ostracism, just as in hospitals.

Universities would sync with each other over phone lines nightly using a protocol called UUCP. With some modification, that evolved into the US defense department network called ARPA net. Eventually, that evolved into the Internet. In 1994 – 1995 the Internet became widely available and was flooded by people who had no concept of how things work and no concept of acceptable use. What it had been until then was lost.

Email is a holdover from those early days.

Many attempts have been made to replace it with systems which include control over who can use what email address, who can contact who, validity verification and more. All have failed. The way email works is too entrenched to be altered in any meaningful way. People dislike change and disruption.

We throw away or refuse more than 95% of the email coming to our mail servers. We can’t get that number to 100% without accidentally discarding wanted emails.

The only remedy is to understand that email is flawed. Don’t waste time hoping for it to be something else. Throw the spam away and forget it. It doesn’t deserve more than one second of your time.

I was asked today if some email is not getting through the spam filters on our mail server.  In general, only spam is filtered out, but there are some uncommon cases where wanted email is rejected.   The more email is filtered, the more opportunity exists for mistakes.  Once in a while email does get classified as spam when it is not.  This is called a false positive. There will always be some false positives because mail servers are configured by humans and email is created and sent by humans.  Humans make mistakes.  It’s only a question of when and at what level false positives are acceptable.

An example would be if 2 people want to have an email conversation about certain kinds of pharmaceuticals.  Some of their emails might seem to disappear.  The thing to realize is that putting up with losing that kind of email also means not having to delete hundreds of spam emails daily, many thousands of emails per year.  Most people would say that the inconvenience is worth it.

We are now doing grey listing.  When a sender has not been seen before, a deferral response is sent (a 400 series response, not an outright refusal).  It’s accompanied by a message, “Please try again in 1 minute.”  Properly configured mail servers will try again because they understand that a deferral response is not  a refusal.   Spammer mail servers seldom try again because they need to send as much email as possible before they get black listed.

False positives can happen when the sending server is misconfigured.  Some servers fail to differentiate a deferral from a refusal.  They don’t try again.  Not to mince words, the person or people in charge of the server do not understand what they are doing.

Grey listing also helps us spot email being sent to web harvested addresses.  When we see the same email going to an architectural firm in Dubai, to a home tutoring service in Des Moines,  a car repair shop in San Diego and a rare antiques shop in London it’s a good bet that it’s spam.

We are blocking thousands of spam emails per day using these 2 techniques.

We have also started doing sender verification.  This is done by testing whether a bounce message to the sending address would be accepted.  If the sender address is fake, it’s reasonable not to take mail from it.  This is controversial because it puts load on other servers which are innocent of spamming and can be employed by spammers as an attack on those servers.  However, Gmail, Yahoo, AOL, Hotmail and others do this to our servers.  Fair is fair. But again, false positives can occur due to badly configured servers. The email standards documents say that bounces must be accepted when sent to live addresses.  Some servers refuse bounces because of user complaints that they are getting email returned they didn’t send.  There are solutions to that problem, but this isn’t it.  Some mail server administrators simply do not know that it is possible to tell the difference between verification and a real bounce.

Many people expect that email should be perfectly reliable and run their business partly based on the assumption that it is.  Unfortunately, that assumption is unrealistic.  Without filtering, email is essentially unusable because of the volume of spam.  The entire system is imperfect because it’s run by an unpredictable collection of imperfect humans.  It is flawed. To say it more plainly, it’s a mess.  Nobody in his right mind would design email to work the way it does on the Internet as it is today.

On balance, false positives are relatively rare.  For the vast majority of users they will never be a problem.  All the same, it’s a good idea to remember that they are possible.

At Deerfield Hosting we work hard to reduce the spam (unsolicited email) our users receive.  About 95% of email arriving at our servers is discarded because we can identify it as spam.  None the less, the remaining spam can still be a significant annoyance.  The problem with further filtering is false positives.  We can’t be throwing away important emails.  It’s a hard problem.

We have been noticing for some time the same from address sending to many of our users in different and unrelated domains.  Most often this is due to web site scraping, email addresses harvested from web sites.  To identify this kind of spam, we have started tracking inbound from and to addresses and generating statistics in real time.  When a particular from address exceeds more than a few unrelated domains, subsequent email from that address is blocked.

We welcome feedback on this.  If you are noticing that you are getting less spam or if you can see no measurable difference, we want to hear from you.

 

Your email account with us may be affected by our improved anti-spam systems.  If it is, you need to know what to do about it to minimize the inconvenience.

The number of spammers making use of snowshoe spamming has recently increased dramatically.  The practice is called that because the load of getting out for example a million spam emails is spread out over thousands of compromised computers.  It is easier to evade detection because each individual machine is sending a relatively small number of emails.  They stay under the radar so that the machine owner remains unaware.

Our first line of defense is blocking IP addresses of compromised machines.  If you get a message when you try to send an email which begins “Blacklist Reject”, it means the IP address your computer is using is on a list.  Your machine may have been compromised or your local service provider may have recently assigned to you an address of another machine which was.  What you need to do is check for a compromise or change your IP address.  A phone call to your service provider may  be necessary.  Other than providing advice, we can’t help with this.

The second line of defense is based on tracking the bounce rate of every email account.  We track bounces as a percentage of all sent emails and if it is too high, there is about a 99% probability (not an exaggeration) that there is some sort of compromise. The most reliable indicator of spamming is higher than normal bounce rates. Normal bounce rates are under 1%, if people are paying attention. The block threshold we use is 3%.

You need to receive and read your bounce messages to avoid getting innocently blocked.  If your email client (Outlook, Windows Mail, Thunderbird) sets the “Return-to” address to an address you do not check or you discard emails from “Mailer-Daemon”, you won’t have a clue when there is a problem.  If your account gets blocked, the bounce message will include a link to the mail server where you can remove the block.

A recent security study found that 37% of computers with Internet connections are being operated by remote users. That is, they are under the control of criminals. Often, user logins with passwords are sold by these people. With the new system, we have so far identified more than 5,000 computers which were logging in to our servers with valid passwords and sending spam.

Please understand that it is essential to protect the integrity and reputation of our mail servers, even if it means occasionally causing inconvenience. The purpose of the unblock webpage is to mitigate the inconvenience.   In a sense, you should be glad if you get inconvenienced in this way. It is an early warning that you have a serious problem.  A compromise could turn into problems such as identity theft and worse.

The good news is that we actively pay attention to keeping your email safe and secure.