Email

This morning we had a common help ticket request. A customer had received an email telling him to pay up or his secrets would be revealed. The secrets concerned his sexual behavior. He was disgusted and wanted to never get such an email again. I don’t blame him.

The email was almost certainly a shot in the dark, hoping to hit someone unwary. This is what they do. Since few of us have the resources of Jeff Bezos to deal with extortion attempts, we will have to find a more practical way to deal with them than counter-attack.

Following is my response to this customer:

Other than throwing the email away, what did you want to do about it? If you have no reason to believe that the email is anything other than spam, there is nothing more to be done about it.

What the criminals do is send thousands of emails like the one you quoted. Perhaps 1 in 5,000 believe it and send them money. That keeps them sending more.

Email was not designed to run on the Internet. It was designed to be used in a single hospital. There was no thought given to potential abuse because everyone knew everyone else. Bad behavior would result in a person being regarded as a misfit and might even get the person fired. From there, email spread to universities. It was found to be a very useful collaboration tool. It was a collegial environment where misuse would result in ostracism, just as in hospitals.

Universities would sync with each other over phone lines nightly using a protocol called UUCP. With some modification, that evolved into the US defense department network called ARPA net. Eventually, that evolved into the Internet. In 1994 – 1995 the Internet became widely available and was flooded by people who had no concept of how things work and no concept of acceptable use. What it had been until then was lost.

Email is a holdover from those early days.

Many attempts have been made to replace it with systems which include control over who can use what email address, who can contact who, validity verification and more. All have failed. The way email works is too entrenched to be altered in any meaningful way. People dislike change and disruption.

We throw away or refuse more than 95% of the email coming to our mail servers. We can’t get that number to 100% without accidentally discarding wanted emails.

The only remedy is to understand that email is flawed. Don’t waste time hoping for it to be something else. Throw the spam away and forget it. It doesn’t deserve more than one second of your time.

I was asked today if some email is not getting through the spam filters on our mail server.  In general, only spam is filtered out, but there are some uncommon cases where wanted email is rejected.   The more email is filtered, the more opportunity exists for mistakes.  Once in a while email does get classified as spam when it is not.  This is called a false positive. There will always be some false positives because mail servers are configured by humans and email is created and sent by humans.  Humans make mistakes.  It’s only a question of when and at what level false positives are acceptable.

An example would be if 2 people want to have an email conversation about certain kinds of pharmaceuticals.  Some of their emails might seem to disappear.  The thing to realize is that putting up with losing that kind of email also means not having to delete hundreds of spam emails daily, many thousands of emails per year.  Most people would say that the inconvenience is worth it.

We are now doing grey listing.  When a sender has not been seen before, a deferral response is sent (a 400 series response, not an outright refusal).  It’s accompanied by a message, “Please try again in 1 minute.”  Properly configured mail servers will try again because they understand that a deferral response is not  a refusal.   Spammer mail servers seldom try again because they need to send as much email as possible before they get black listed.

False positives can happen when the sending server is misconfigured.  Some servers fail to differentiate a deferral from a refusal.  They don’t try again.  Not to mince words, the person or people in charge of the server do not understand what they are doing.

Grey listing also helps us spot email being sent to web harvested addresses.  When we see the same email going to an architectural firm in Dubai, to a home tutoring service in Des Moines,  a car repair shop in San Diego and a rare antiques shop in London it’s a good bet that it’s spam.

We are blocking thousands of spam emails per day using these 2 techniques.

We have also started doing sender verification.  This is done by testing whether a bounce message to the sending address would be accepted.  If the sender address is fake, it’s reasonable not to take mail from it.  This is controversial because it puts load on other servers which are innocent of spamming and can be employed by spammers as an attack on those servers.  However, Gmail, Yahoo, AOL, Hotmail and others do this to our servers.  Fair is fair. But again, false positives can occur due to badly configured servers. The email standards documents say that bounces must be accepted when sent to live addresses.  Some servers refuse bounces because of user complaints that they are getting email returned they didn’t send.  There are solutions to that problem, but this isn’t it.  Some mail server administrators simply do not know that it is possible to tell the difference between verification and a real bounce.

Many people expect that email should be perfectly reliable and run their business partly based on the assumption that it is.  Unfortunately, that assumption is unrealistic.  Without filtering, email is essentially unusable because of the volume of spam.  The entire system is imperfect because it’s run by an unpredictable collection of imperfect humans.  It is flawed. To say it more plainly, it’s a mess.  Nobody in his right mind would design email to work the way it does on the Internet as it is today.

On balance, false positives are relatively rare.  For the vast majority of users they will never be a problem.  All the same, it’s a good idea to remember that they are possible.

At Deerfield Hosting we work hard to reduce the spam (unsolicited email) our users receive.  About 95% of email arriving at our servers is discarded because we can identify it as spam.  None the less, the remaining spam can still be a significant annoyance.  The problem with further filtering is false positives.  We can’t be throwing away important emails.  It’s a hard problem.

We have been noticing for some time the same from address sending to many of our users in different and unrelated domains.  Most often this is due to web site scraping, email addresses harvested from web sites.  To identify this kind of spam, we have started tracking inbound from and to addresses and generating statistics in real time.  When a particular from address exceeds more than a few unrelated domains, subsequent email from that address is blocked.

We welcome feedback on this.  If you are noticing that you are getting less spam or if you can see no measurable difference, we want to hear from you.

 

Your email account with us may be affected by our improved anti-spam systems.  If it is, you need to know what to do about it to minimize the inconvenience.

The number of spammers making use of snowshoe spamming has recently increased dramatically.  The practice is called that because the load of getting out for example a million spam emails is spread out over thousands of compromised computers.  It is easier to evade detection because each individual machine is sending a relatively small number of emails.  They stay under the radar so that the machine owner remains unaware.

Our first line of defense is blocking IP addresses of compromised machines.  If you get a message when you try to send an email which begins “Blacklist Reject”, it means the IP address your computer is using is on a list.  Your machine may have been compromised or your local service provider may have recently assigned to you an address of another machine which was.  What you need to do is check for a compromise or change your IP address.  A phone call to your service provider may  be necessary.  Other than providing advice, we can’t help with this.

The second line of defense is based on tracking the bounce rate of every email account.  We track bounces as a percentage of all sent emails and if it is too high, there is about a 99% probability (not an exaggeration) that there is some sort of compromise. The most reliable indicator of spamming is higher than normal bounce rates. Normal bounce rates are under 1%, if people are paying attention. The block threshold we use is 3%.

You need to receive and read your bounce messages to avoid getting innocently blocked.  If your email client (Outlook, Windows Mail, Thunderbird) sets the “Return-to” address to an address you do not check or you discard emails from “Mailer-Daemon”, you won’t have a clue when there is a problem.  If your account gets blocked, the bounce message will include a link to the mail server where you can remove the block.

A recent security study found that 37% of computers with Internet connections are being operated by remote users. That is, they are under the control of criminals. Often, user logins with passwords are sold by these people. With the new system, we have so far identified more than 5,000 computers which were logging in to our servers with valid passwords and sending spam.

Please understand that it is essential to protect the integrity and reputation of our mail servers, even if it means occasionally causing inconvenience. The purpose of the unblock webpage is to mitigate the inconvenience.   In a sense, you should be glad if you get inconvenienced in this way. It is an early warning that you have a serious problem.  A compromise could turn into problems such as identity theft and worse.

The good news is that we actively pay attention to keeping your email safe and secure.

 

Last updated August 14, 2017 to reflect security improvements.

We often get questions similar to this one:

I have been struggling to send and receive mail consistently since acquiring Apple/Mac products. Would you please see attached reference sheet and make sure I have filled everything in properly.

In order to provide a clear answer, I want to ignore the form you got from the Apple support people.  All you need to do is select protocols and then provide the necessary information for them.  The form lists all the protocols, then separately lists port numbers and settings.  This is incoherent.  Port numbers go with protocols.  Using a particular protocol implies a port number and vice versa.

Getting set up is easier when you understand what the protocols are and why you might select each of them.  The configuration follows on from there.

Recommended settings (for the impatient)
Server, both incoming and outgoing:  mail.YourDomain.com (use your domain name)
Incoming protocol: IMAP Port number: 993 – SSL/TLS – accept all certificates
Outgoing protocol: SMTP Port number: 587 – start TLS – accept all certificates
Login: full email address – Required

Some email clients repeatedly pop up warnings or errors related to server names or certificates.  This is because they are seeing a security certificate with the name of the server, not the server name you provided.   Old, obsolete email client software can’t request the certificate for your domain name.  If at all possible, switch to modern software or update what you are using.  Old software tends to contain security vulnerabilities and is best avoided.  If this is not possible, you will have to use the server name instead of your domain name to silence the warnings.

Our servers are 100% standards compliant and will immediately work with any device which uses standard values as defaults.  This means all you should need to do is select a protocol and then accept the defaults.  Unfortunately, sometimes the software fails to offer a value or offers something bizarre.

Email has 2 entirely separate parts. Interacting with a server which contains delivered email is one part.  The other part is sending email.  There is nothing in common between them other than the term and file object, “email”.  As you can see just above, I am suggesting IMAP for handling emails delivered to you and SMTP for sending.  First we will discuss delivered email (email waiting for you), then we will discus sending email.

Common Email Protocols

A protocol is a set of rules which defines how computers communicate with each other.  It’s how each machine knows what to expect from the other.  It’s easier to follow descriptions like this when you have some idea what the acronyms stand for.

IMAP – “Internet Message Access Protocol”
IMAPS – “Internet Message Access Protocol” – Secured (encrypted)
POP3 – “Post Office Protocol” version 3
POP3S – “Post Office Protocol” version 3 – Secured
SMTP – “Simple Mail Transport Protocol”
SMTPS – “Simple Mail Transport Protocol” – Secured
SSL – “Secure Sockets Layer”
TLS – “Transport Layer Security”
DNS – “Domain Name Server (or System)” – name to address translation
DNS MX – An entry in the DNS system which provides email routing information.

The Internet is all about connectivity, so there is a seemingly endless list of acronyms and protocols.  If you are interested in understanding more about how things work, read the page titled, “The Internet – Technical Basics” above.  Understanding goes a long way to making things easier.

The potential exists for the setup process to be simpler than it is in practice.  There are clearly defined standards for a client machine to connect to a server, request a particular protocol and even switch over to an encrypted connection.  Our servers support this.  Often,  the programmers who wrote the device software were too ignorant to take advantage.  Even if the programmers got it right, the documentation and support people remain clueless.  We are stuck doing it the hard way.  Apple devices, known for excellent user interfaces, remain strangely ignorant.

Picking up email

To interact with a server which has your email, there are many options.  The 2 common choices are IMAP and POP3.  Both can be used encrypted or not encrypted.  That makes 4 sets of settings.  That may sound complicated, but it’s actually simple.

Once your computer (the client machine) makes a connection to a server, it requests a particular port number.  The port number selection is what tells the server which protocol you want to use.  If you tell your client software to use IMAP with a POP3 port number, it’s not going to work.

1) IMAP – not encrypted
Protocol: IMAP
Port number: 143

2) IMAPS – encrypted “secure IMAP” – I suggest you use this.
Protocols it uses: IMAP, IMAPS, SSL, TLS
Port number: 993

The IMAP protocol is more advanced and has many more options than the POP3 protocol.  In a default configuration, your device  (even a web browser)  will first see only the header values.  Headers in email parlance are the metadata, From, To, Subject, delivery and other information describing what an email contains.  The actual content is not displayed or downloaded until you explicitly ask for it, for example by clicking on it.  By default, email will stay on the server until you remove it.

When you use IMAP, email can be organized into folders.  In a default configuration you generally get an Inbox folder, a Sent folder and a Trash folder.  You can add new folders and move email from folder to folder to keep it organized.

Within each folder, but not visible to you are 2 sub folders, “new” and “cur” (for current).  Email is delivered to “new” and as soon as you first see the headers, it is moved to “cur”.   Some devices show you only what was in “new” in the Inbox.  Generally, you can change what they show you.  When you connect later and don’t see what you saw earlier, nothing has disappeared from the server.  Your email software is simply not showing it to you.

To delete mail from the server when using IMAP, you need to purge or “empty the trash”.  We commonly see accounts go over quota because the owner never emptied the trash.   Permanently deleting email is a 2 step process: delete then purge.

3) POP3 – Post Office Protocol – not encrypted
Protocol: POP3
Port number: 110

4) POP3S – Post Office Protocol – encrypted
Protocol: POP3S
Port number: 995

The POP3 protocol is much older and much simpler.  In the default configuration, all email is downloaded to you when you connect and then deleted from the server.  On a phone or other small device, you almost certainly want to use IMAP.  You don’t want to get stuck waiting for a big attachment to download over a slow link.  Using IMAP you can defer looking at those emails until you are using a larger and probably faster device.

That’s receiving. Sending email can be done using IMAP, but that’s not widely used. Our servers support it, but I suggest you use SMTP.  It’s simpler.

Sending Email

Protocol: SMTP – I suggest you use SMTP for sending.
Port number: 587 or 26

The standard port number for SMTP is 25, but many service providers block port 25. They do this because it is easy to set up a mail server and start sending spam everywhere.  There is apparently an endless supply of people who believe they can make money this way.  As a work-around, our servers listen on additional ports.   Port 587 is the “mail message submission” port, which is the standard work-around.   Some poorly written and non standards compliant software (notably Microsoft Outlook) cannot be configured to use port 587.  This is why we listen on port 26 as well.

Long ago, sending email using an encrypted connection required connecting on a separate port number.  There is outdated documentation floating around which describes how to do this using port 465.  If you encounter references to port 465 in email client documentation or elsewhere, you are reading about antique software. Don’t use it.

For BOTH Receiving and Sending

Your login name is your email address.  You MUST log in.  Inexplicably, some client software assumes that you do not need to log in.  If you have problems sending, this is the most common cause.  Find and check a box labeled something like, “My server requires authentication”.

The server to connect to is mail.YourDomain.com in all cases. Replace “YourDomain.com” with your actual domain name.

Incorrect Default Server Names

It is very common for email client software (running on your device) to make bad assumptions about server names.  It may offer defaults like IMAP.yourDomain.com or SMTP.yourDomain.com.  Unless explicitly created (by you) via DNS zone file setup or editing, they do not exist and will not work.   Sometimes DNS MX values are looked up and offered.  A DNS MX record specifies where to route mail to get to your account, but is not necessarily where email is finally delivered to you.   Do not use MX values.  They will not work.

Certificates

Encryption certificates serve 2 purposes:  to verify the identity of what you connected to; providing encryption keys.   The name on the certificate is that of the server, not mail.YourDomain.com.  This means you must override the warning (or check a box) to accept the certificate.  You could purchase a certificate and have us set it up for you, but there is no necessity to spend the extra money for this.

Occasionally we move accounts from one server to another.  Your email client will continue to connect to the correct server as long as you use mail.YourDomain.com.  You can use the server name to avoid the certificate warning, but this would cause a problem the next time your account is moved to a new server.  By then the problem is likely to be a mystery and require detective work to solve it.

Using an encrypted connection provides a slight security enhancement. I dislike this because people tend to think it is doing more for them than it is. It’s a false sense of security.  If you have a reason other then not providing your password over a clear connection, please read this:  https://blog.deerfieldhosting.net/encrypted-email-or-is-it/

We use a separate mail server to provide better performance and redundancy.  Email is delivered there and held for 3 days so that it is available to you even if your hosting server account is unavailable.  You can use web based email on it if you need to.

Backup Mail Server

This is a highly summarized overview.  If you have questions or comments, please ask them!