Deerfield Hosting, Inc.

High Performance Web Hosting

Heartbleed Bug Details

by

We are responsible for more than 30,000 user logins on computers which had been susceptible to the Heartbleed bug.  As you may imagine, that means we have been watching very carefully for signs of trouble.  So far, we have found no evidence of any compromise.  What this does not mean is that complacency is acceptable.

Media reports have in some cases flatly stated that this bug allows an attacker to compromise any server with the bug.  That is, to gain access to the server at the administrator level.  This is almost total nonsense.  As you will see from what follows, the problem is entirely limited to data flows.  It’s true that if an administrator happened to log in while an attacker was running an exploit, there is a slight chance that his login name and password were captured.  On our systems I can say unequivocally that this never happened.  We use a two factor encryption system.   The second part was exposed to the bug, but the first part was never in jeopardy.

As an aside, this bug has created a much needed uproar in the security community.   I checked the online banking system I use.  I found that while it was never susceptible to the heartbleed bug, it was susceptible to man in the middle attacks.  That is nearly as serious!  I suggest you check and complain at any secure sites you need to use.  Here is a way to check:

Website Security Test

Just put the subject website name in the box.

Very few of our users have followed our advice and changed their passwords.  If you are one of them, please do so.  It’s true that media reports have been overblown and the probability that your account has been affected is low.  None the less, this is NOT something to ignore.

We have more information now about this bug, how it works, what it would take to compromise a server and what it would take to compromise any individual account.  It’s not nearly as bad as first reported. 

It’s a buffer overflow vulnerability.  When a program starts up, it allocates memory to hold what it needs to do its work.  In the case of openssl (the program with the bug) as it services requests it allocates and frees memory to temporarily hold the information flowing in and out.  When a request has completed, the pointer to this memory is discarded.  The problem is that whatever was in that area is still there.  The bug allows an attacker to request and get what was left over from the last request which used that memory area.

Memory is allocated on what’s called the heap, from the bottom up.  When openssl starts up, the first thing it does is load public and private keys into the heap.  This means the private keys are in the lowest part of memory.  Following is a diagram of subsequent memory allocations:

The client machine is allowed to check with the server periodically to see if the encrypted connection is still intact.  This is called a heartbeat.  The client sends the server a test packet and the server sends it back as verification.  The bug is that the server fails to sanity check the packet size which is provided in the first few bytes of the request, “length” in the diagram.  It returns a memory area as large as whatever the client put there.  If it’s bigger than what was sent, material left over from the last request is sent.  If that happens to contain something sensitive, we have a serious problem.

Initially it was reported that the attacker could gain access to the keys to the kingdom, the server private encryption keys.  Security experts, with an abundance of caution, are unwilling to flatly state that it is impossible for an attacker to get these keys, but that appears to be the case.  The keys are located in the lowest portion of memory which is never freed and reused.  Using an assortment of scenarios, literally hundreds of millions of attempts have been made to get the keys.  None has yet succeeded.

UPDATE: Although the process is difficult, it has now been proven that it was possible to obtain the server keys.

Again, the likelihood that your passwords have been compromised by this particular bug is very low.  Our servers were updated and new security certificates issued within 3 hours of the announcement. 

However, recent security studies have been done on the source of the bot net (robot network) problem.   A bot net is a group of compromised computers being controlled remotely by someone other than the owner.  They are used in concert or individually to attack other computers.  The studies have provided good evidence that about 37% of computers connected to the Internet with broadband connections are participating in bot net activity.  This means there is roughly a 1 in 3 chance that your computer is one of them.  If it is, you can bet your passwords are out there.  If you have been complacent about security, you really need to check for this.

The Need for Speed

by

When a Facebook post gets several million likes and is backed by a WordPress post on one of our servers, we handle some serious traffic.  Maintaining the fast response we strive for can be a challenge.

We continually search for ways to increase server performance for our busiest sites.  For the sake of reliability and stability, most of our servers run Centos 6, but Centos has become a source of frustration.  It’s obsolete.  Centos 7 (based on Redhat Enterprise Linux 7) is in the works and it will be a big relief when it finally appears.  When I read that RHEL 7 will be based on Fedora 19, I decided it was time to try Fedora as a server.  Fedora lists “First” as one of its core values and tries to implement the most recent innovations with releases twice per year.

Our latest server is running Fedora 20.   I can hardly believe the performance gain.  To test, we moved 5 very busy sites onto this server.  A server load average is a rough estimate of how many processes had to wait over the last second.  It correlates directly with web site performance, speed as perceived by a site visitor.  At quiet times on the previous server I was seeing loads at 1 and 2.  At the same times on this server, the load is 0.0, sometimes spiking up to 0.2.  At busy times I was seeing loads of 2 and 3, sometimes spiking to 5 and 7.  On this server it’s 0.2 and 0.3 with spikes up to 0.5.  On top of that, spikes disappear much more rapidly.  Where I had been seeing spikes fade away in 5 or 10 seconds, now I am seeing them fade in 2 or 3 seconds.  That’s huge.  It means not only that site visitors are getting page loads 10 times (or more) faster, but the fastest performance is being seen by 5 times as many visitors.

The gains came from a long list of improvements.  Before getting into what they are, a disclaimer.  To make this understandable to people who aren’t techno nerds, I’m oversimplifying a bit. 

Where loads are coming from can get really complicated with so many things going on at the same time in a server.  It matters what those things are because they are dynamically interacting with each other.  For example, under certain conditions we see loads more than double when the visitor hit rate doubles.  Sometimes the increase is logarithmic and sometimes quadratic.   But it can also increase by less than double.  It depends on the server software, the site application software and on the interaction between them.  Having said that, here’s the list:

  • The Linux kernel V 3.13 – Recent optimizations have a substantial effect.
  • The network stack and drivers have been improved.
  • Web server software: Apache 2.4.9 or Nginx 1.5.13 – Nginx is a bit faster, but when some features are essential, Apache is a better choice.  Apache 2.4 outperforms 2.2.
  • XFS file system.  This is disk reads and writes.  Under load it’s twice as fast as EXT4.  Disk operations are the most common performance bottleneck.
  • MariaDB (from MySQL 5.5) – this version is said to be 2 – 3 times faster in the most common use cases.
  • PHP 5.5.10 – by being less tolerant of errors and changing semantics, performance gains of 20 to 40% are often possible.
  • Zend Opcache – saves the step of compiling scripts into executable code by holding the code in memory.  It also saves a disk read.
  • PHP-FPM – this keeps the php binary up and running so that the web server process can pass it script names to execute, passing back the results.

We are now offering virtual private servers set up this way.  We are also offering the opportunity to run sites in a shared VPS set up this way.   You can find it in our ordering system.  This is a strong value for sites too busy to run on a cPanel shared hosting server.

I was asked, “Where’s Varnish?”  Varnish is a page cache which resides in memory.  It saves a disk access when a static file (html page, image, etc.) has been read recently.   This is useful on proxy servers with a server farm behind them.  In other situations it’s counter productive.  Modern operating systems try to take full advantage of available memory.  All memory not in immediate use is allocated to disk cache buffers.   When more memory is needed by a program it is taken from the least recently used (LRU) memory pages.  In other words, it’s already doing what Varnish does and much more efficiently in terms of the whole system.   Adding Varnish is much more likely to increase disk reads than reduce them.

You will find many pages on the Internet recommending Varnish incorrectly.  Something like 80% of the information and recommendations about getting the most out of your web site or server is dead wrong, too incomplete to be useful or doesn’t apply to the most common environments.   People like to write about the wonderful new things they’ve learned, but often fail to realize they don’t know enough about the bigger picture.  Be careful what you believe.

The Heartbleed Bug

by

Today, a serious and pervasive threat to security on the Internet was revealed: the so-called Heartbleed Bug. In my opinion, everyone who regularly uses a password on a “secure” Internet connection should have at least a rudimentary grasp of the problem. A web site has been set up to describe it in detail: Heartbleed.com.

The short version of the problem is that an encryption vulnerability was found.  Under certain circumstances, a third party can decrypt your session with a secured web site or impersonate a secured web site.  First the attacker must obtain the encryption keys from a secure site and this is what the bug allows him to do.  Once the attacker has the keys, if he can get access to what is flowing back and forth between you and the site, he can read it.  That includes passwords, credit card information, all of it.

News reports have given the impression that with stolen keys, an attacker can walk right into a server and get whatever they want.  This is wrong.  If a server administrator logged in remotely while his session was being read, the attacker could then log in with the same credentials.  This is quite different and not at all likely.  Most servers have constraints on where administrators can be when they log in.  It would set off alarms.

The likelihood that information has already been stolen from you is low.  Normally we don’t see security bugs exploited until they are well known.  This problem was first discovered last week and was announced publicly today.   The delay was to allow time to get fixes in place.  Our servers have been updated and certificates replaced.  We are no longer vulnerable to this threat.

What is IMPORTANT is that any secure sites you interact with have been updated.  If they do not post a notice, you should ask before logging in.  I just attempted to find out if my bank was aware of the problem.  I was unable to get an answer.  Hopefully the people who manage the web site have taken care of it, but the only safe assumption is that they have not.  I’m not going to use web banking until I can get an answer and you shouldn’t either.

NSO Transparency Report

by

On January 27th, the United States Department of Justice announced new rules regarding the disclosure of National Security Orders.  This included National Security Letters (NSLs) received by a company.   The DOJ and Director of National Intelligence (DNI) now allow a company to disclose the number of letters and orders it has received as a single number in bands of 250.  The first band would be 0-249.  It continues to be illegal to make this disclosure as an exact number.

At Deerfield Hosting, we believe that disclosing the exact number of orders a company has received poses no threat to national security.

Many people assume that tech companies receive and comply with large numbers of such orders daily.  However, the real numbers are probably quite small.  For example, Apple has just reported a band of 0-249.

In keeping with the new rules, our report is this:

  • National Security Orders Received: 0 – 249
  • Total Accounts Affected: 0 – 249

We believe that the new rules are a step in the right direction, but do not go far enough.  We also believe that orders of these types are a violation of due process and are unconstitutional.  It is our policy to refuse to comply with such orders unless received by a court and not merely by the DOJ.

Moreover, if you wish to know of any orders we have received which pertain to any accounts you may have with us, simply ask us.  We are quite prepared to break the law and give you an answer.

Email On Phones and Other Devices

by

Last updated August 14, 2017 to reflect security improvements.

We often get questions similar to this one:

I have been struggling to send and receive mail consistently since acquiring Apple/Mac products. Would you please see attached reference sheet and make sure I have filled everything in properly.

In order to provide a clear answer, I want to ignore the form you got from the Apple support people.  All you need to do is select protocols and then provide the necessary information for them.  The form lists all the protocols, then separately lists port numbers and settings.  This is incoherent.  Port numbers go with protocols.  Using a particular protocol implies a port number and vice versa.

Getting set up is easier when you understand what the protocols are and why you might select each of them.  The configuration follows on from there.

Recommended settings (for the impatient)
Server, both incoming and outgoing:  mail.YourDomain.com (use your domain name)
Incoming protocol: IMAP Port number: 993 – SSL/TLS – accept all certificates
Outgoing protocol: SMTP Port number: 587 – start TLS – accept all certificates
Login: full email address – Required

Some email clients repeatedly pop up warnings or errors related to server names or certificates.  This is because they are seeing a security certificate with the name of the server, not the server name you provided.   Old, obsolete email client software can’t request the certificate for your domain name.  If at all possible, switch to modern software or update what you are using.  Old software tends to contain security vulnerabilities and is best avoided.  If this is not possible, you will have to use the server name instead of your domain name to silence the warnings.

Our servers are 100% standards compliant and will immediately work with any device which uses standard values as defaults.  This means all you should need to do is select a protocol and then accept the defaults.  Unfortunately, sometimes the software fails to offer a value or offers something bizarre.

Email has 2 entirely separate parts. Interacting with a server which contains delivered email is one part.  The other part is sending email.  There is nothing in common between them other than the term and file object, “email”.  As you can see just above, I am suggesting IMAP for handling emails delivered to you and SMTP for sending.  First we will discuss delivered email (email waiting for you), then we will discus sending email.

Common Email Protocols

A protocol is a set of rules which defines how computers communicate with each other.  It’s how each machine knows what to expect from the other.  It’s easier to follow descriptions like this when you have some idea what the acronyms stand for.

IMAP – “Internet Message Access Protocol”
IMAPS – “Internet Message Access Protocol” – Secured (encrypted)
POP3 – “Post Office Protocol” version 3
POP3S – “Post Office Protocol” version 3 – Secured
SMTP – “Simple Mail Transport Protocol”
SMTPS – “Simple Mail Transport Protocol” – Secured
SSL – “Secure Sockets Layer”
TLS – “Transport Layer Security”
DNS – “Domain Name Server (or System)” – name to address translation
DNS MX – An entry in the DNS system which provides email routing information.

The Internet is all about connectivity, so there is a seemingly endless list of acronyms and protocols.  If you are interested in understanding more about how things work, read the page titled, “The Internet – Technical Basics” above.  Understanding goes a long way to making things easier.

The potential exists for the setup process to be simpler than it is in practice.  There are clearly defined standards for a client machine to connect to a server, request a particular protocol and even switch over to an encrypted connection.  Our servers support this.  Often,  the programmers who wrote the device software were too ignorant to take advantage.  Even if the programmers got it right, the documentation and support people remain clueless.  We are stuck doing it the hard way.  Apple devices, known for excellent user interfaces, remain strangely ignorant.

Picking up email

To interact with a server which has your email, there are many options.  The 2 common choices are IMAP and POP3.  Both can be used encrypted or not encrypted.  That makes 4 sets of settings.  That may sound complicated, but it’s actually simple.

Once your computer (the client machine) makes a connection to a server, it requests a particular port number.  The port number selection is what tells the server which protocol you want to use.  If you tell your client software to use IMAP with a POP3 port number, it’s not going to work.

1) IMAP – not encrypted
Protocol: IMAP
Port number: 143

2) IMAPS – encrypted “secure IMAP” – I suggest you use this.
Protocols it uses: IMAP, IMAPS, SSL, TLS
Port number: 993

The IMAP protocol is more advanced and has many more options than the POP3 protocol.  In a default configuration, your device  (even a web browser)  will first see only the header values.  Headers in email parlance are the metadata, From, To, Subject, delivery and other information describing what an email contains.  The actual content is not displayed or downloaded until you explicitly ask for it, for example by clicking on it.  By default, email will stay on the server until you remove it.

When you use IMAP, email can be organized into folders.  In a default configuration you generally get an Inbox folder, a Sent folder and a Trash folder.  You can add new folders and move email from folder to folder to keep it organized.

Within each folder, but not visible to you are 2 sub folders, “new” and “cur” (for current).  Email is delivered to “new” and as soon as you first see the headers, it is moved to “cur”.   Some devices show you only what was in “new” in the Inbox.  Generally, you can change what they show you.  When you connect later and don’t see what you saw earlier, nothing has disappeared from the server.  Your email software is simply not showing it to you.

To delete mail from the server when using IMAP, you need to purge or “empty the trash”.  We commonly see accounts go over quota because the owner never emptied the trash.   Permanently deleting email is a 2 step process: delete then purge.

3) POP3 – Post Office Protocol – not encrypted
Protocol: POP3
Port number: 110

4) POP3S – Post Office Protocol – encrypted
Protocol: POP3S
Port number: 995

The POP3 protocol is much older and much simpler.  In the default configuration, all email is downloaded to you when you connect and then deleted from the server.  On a phone or other small device, you almost certainly want to use IMAP.  You don’t want to get stuck waiting for a big attachment to download over a slow link.  Using IMAP you can defer looking at those emails until you are using a larger and probably faster device.

That’s receiving. Sending email can be done using IMAP, but that’s not widely used. Our servers support it, but I suggest you use SMTP.  It’s simpler.

Sending Email

Protocol: SMTP – I suggest you use SMTP for sending.
Port number: 587 or 26

The standard port number for SMTP is 25, but many service providers block port 25. They do this because it is easy to set up a mail server and start sending spam everywhere.  There is apparently an endless supply of people who believe they can make money this way.  As a work-around, our servers listen on additional ports.   Port 587 is the “mail message submission” port, which is the standard work-around.   Some poorly written and non standards compliant software (notably Microsoft Outlook) cannot be configured to use port 587.  This is why we listen on port 26 as well.

Long ago, sending email using an encrypted connection required connecting on a separate port number.  There is outdated documentation floating around which describes how to do this using port 465.  If you encounter references to port 465 in email client documentation or elsewhere, you are reading about antique software. Don’t use it.

For BOTH Receiving and Sending

Your login name is your email address.  You MUST log in.  Inexplicably, some client software assumes that you do not need to log in.  If you have problems sending, this is the most common cause.  Find and check a box labeled something like, “My server requires authentication”.

The server to connect to is mail.YourDomain.com in all cases. Replace “YourDomain.com” with your actual domain name.

Incorrect Default Server Names

It is very common for email client software (running on your device) to make bad assumptions about server names.  It may offer defaults like IMAP.yourDomain.com or SMTP.yourDomain.com.  Unless explicitly created (by you) via DNS zone file setup or editing, they do not exist and will not work.   Sometimes DNS MX values are looked up and offered.  A DNS MX record specifies where to route mail to get to your account, but is not necessarily where email is finally delivered to you.   Do not use MX values.  They will not work.

Certificates

Encryption certificates serve 2 purposes:  to verify the identity of what you connected to; providing encryption keys.   The name on the certificate is that of the server, not mail.YourDomain.com.  This means you must override the warning (or check a box) to accept the certificate.  You could purchase a certificate and have us set it up for you, but there is no necessity to spend the extra money for this.

Occasionally we move accounts from one server to another.  Your email client will continue to connect to the correct server as long as you use mail.YourDomain.com.  You can use the server name to avoid the certificate warning, but this would cause a problem the next time your account is moved to a new server.  By then the problem is likely to be a mystery and require detective work to solve it.

Using an encrypted connection provides a slight security enhancement. I dislike this because people tend to think it is doing more for them than it is. It’s a false sense of security.  If you have a reason other then not providing your password over a clear connection, please read this:  https://blog.deerfieldhosting.net/encrypted-email-or-is-it/

We use a separate mail server to provide better performance and redundancy.  Email is delivered there and held for 3 days so that it is available to you even if your hosting server account is unavailable.  You can use web based email on it if you need to.

Backup Mail Server

This is a highly summarized overview.  If you have questions or comments, please ask them!

Encrypted email – or is it?

by

We are often asked about certificate warnings which pop up in email clients.  When an account is moved to a new server, there is a new certificate and a new warning often appears.  In answer to a question about this:

It’s safe to accept the certificate.  We moved your accounts to a newer faster server a few days ago.  We had been using a wildcard certificate, but that was causing problems for people running software too dumb or too conservative to work with such a certificate.  It seemed like an appropriate time to switch.

Email client software often sets up TLS (encrypted) connections by default.  It makes people think sending stuff via email is secure, adding yet another misconception to the rampant ignorance.  It’s not secure.  Email is a store and forward system.  That means your message may cross the network encrypted, but it is then stored unencrypted on the target mail server.  It frequently passes through many servers before being delivered.  It’s trivial for an administrator of any of those servers to keep a copy – not encrypted.

The security of your message is in the hands of those administrators.  You will almost never even know who they are.

Nearly it’s only virtue is that your password is sent over an encrypted connection.    It also means that when someone at the NSA wants to read it, he will have to spend a few minutes on a powerful computer to decrypt it first.  If you want more secure email, you need to encrypt the content, not just the connection.  If you don’t want the NSA or anyone else reading your email at all, you’re basically out of luck.

Content encryption is good enough to deter most criminals and casual snoopers.  Unfortunately, a really sophisticated criminal can still decrypt it.  But you don’t need to worry about this too much unless you know that your content has a very high value to such a person.  If you make it even a little hard, they will move on.   There is no shortage of easy targets.

The bottom line is: encrypt the content if you need security.  That said, there are better ways to transfer sensitive information than via email.  There’s no reason to allow it to have such a high profile.  Virtually all cases of hacking are the result of gaping stupid security holes, someone incompetent in charge of security.