Deerfield Hosting, Inc.

High Performance Web Hosting

Hard Drive Failure Rates

by

In the last 6 months we have twice had nearly simultaneous drive failures leading to service outages.  It was hard at first to grasp how something so seemingly unlikely could have happened.  When it happened a second time, it was time for some serious scrutiny.  What seemed like common sense might not be correct.

We run servers in pairs.  A hard disk on one server has a corresponding hard disk on another server.  If one disk fails, the service can simply be powered up on the other server.  Both servers have to be down to cause an outage.

The solution became obvious once the problem was understood.

Google released a study of their experience with a very large population of hard disks and failures.  If you have a taste for a dry technical paper, you can find it here: Google media research  What they found was revealing.  The data set is based on consumer grade drives.  We use enterprise grade drives which have a much longer life, but the general observations will be about the same.  This summarizes failure times:

afr_age

As expected, hard disks show a high infant mortality followed by a period of (in our case) several years of reliable service.  Then suddenly failure rates increase.  That there is a decline in failure rates at 4 years is unexpected, but a gradually increasing rate after that is just what you might expect.  There is no data to support a rise and fall like that with enterprise drives.  It may or may not happen.

When we think about reliability, what we want to know is the likelihood of a failure event in a given time interval.  Then we can make statements (these are made up numbers) that the odds of a drive failure in a server over a months time are 1 in 300.  Then if a drive in a second server which is being mirrored to is the same, the chance of both drives going down in the same month becomes 1 in 600.  Since replacing a failed drive and re-mirroring takes 2 days, that would make a 1 in 9,000 chance of a failure before we could recover with no down time.  That seems reasonable enough, but it turns out not to be correct.  The problem is the failure rate distribution.

Many people are familiar with “the bell curve”, what in statistics is called the normal distribution.  The graph looks like this:

Empirical_Rule

If you tossed a coin 5,000 times and kept track of how many times in a row you got heads and tails and graphed it, that’s what it would look like.  The left being heads counts and the right being the tails counts.

Hard disk manufacturers supply a statistic meant to show product life called the mean time between failure – MTBF.  If the number was 5 years, the expectation is that most drives would last about that amount of time.  What they report generally doesn’t relate to reality very well as the Google paper shows.  Still, it’s a useful statistic.  If the MTBF is 5 years and we charted a large population of disks, you would expect the chart to be a normal distribution with 5 years being the top of the curve.  Lacking data, my guess at the standard deviation of a set of 5 year MTBF drives would be something like 3 to 6 months.  Failure of a specific drive is random within a time frame so it’s reasonable to expect a failure curve to look something like a normal distribution.  We are (were) working with 2 sets of hard drives all manufactured at the same time, all in exactly the same kind of server and in service for exactly the same amount of time.  What that means is that the top of the curve is going to be much narrower and the sides much steeper.  In statistical terms, the standard deviation will be a much smaller number.

So, the obvious solution?  Add randomness.  Add new drives, but not all new drives.  The older drives have life in them yet.  Besides being a waste of money it would lead to the same situation if we simply replaced all of them.  What we have done is replace half of them.  Each replication pair consists of an older drive and a newer one.  When an older drive fails it will be replaced by another older drive until we run out of them.  Introducing new drives will therefore be at relatively random intervals.  This will move the top of that curve all over the place in terms of single drives.  We may not see the odds against double failures as high as 1 in 9,000 but clearly it will be a huge improvement.  It would be nice to have actual data for predictions.  We don’t, so I will have to make a guess.  Based on a lot of consideration, 1 in 1,000 seems reasonable.  It’s also a number we can live with.

Keeping Your Site Safe

by

After a site has been compromised, we often get criticized for, “Not keeping my site safe”.  It’s not a reasonable criticism.  There isn’t a magic bullet to use against attackers.

We do scan every web server request for something like 10,000 known attacks using the web server plugin mod_security.   The rule set for scanning is updated daily to stay on top of the most recent kind of attacks, both generic and specific.

Every request also has to pass through 2 firewalls.  The first of these examines every packet, checking whether it came from a known bad actor IP address or is malformed in any way, which is a clue that it may have a bad intent.  The second firewall watches activity, looking for patterns typical of compromise attempts.  A simple example is repeated login attempts with different user names and passwords.  This is called a brute force login attack.

Yes, it would be possible to expand the screening to improve safety more.  The trouble is, the list grows exponentially as you expand it.  If we attempted to include anything like all possible attacks, web pages would never appear.  The servers would be too busy with screening to get around to sending them.

So why isn’t that enough?  The usual problem is the web scripting language PHP.  Sadly, it is vulnerable to attack by default.  Unless the programmer using it is aware of possible ways to compromise it and takes steps to prevent those compromises, a script will be vulnerable.  When you install a set of scripts on your site (such as WordPress) you have placed your security in the hands of the authors of the script.  There is no getting around this.

It is critically important to keep your site running the most up to date software available.   Even a short delay in updating can be fatal.  The more popular the software you are using, the more important this becomes.  We deal with WordPress compromises every day which should not have happened.

Grey Listing and Sender Verification

by

I was asked today if some email is not getting through the spam filters on our mail server.  In general, only spam is filtered out, but there are some uncommon cases where wanted email is rejected.   The more email is filtered, the more opportunity exists for mistakes.  Once in a while email does get classified as spam when it is not.  This is called a false positive. There will always be some false positives because mail servers are configured by humans and email is created and sent by humans.  Humans make mistakes.  It’s only a question of when and at what level false positives are acceptable.

An example would be if 2 people want to have an email conversation about certain kinds of pharmaceuticals.  Some of their emails might seem to disappear.  The thing to realize is that putting up with losing that kind of email also means not having to delete hundreds of spam emails daily, many thousands of emails per year.  Most people would say that the inconvenience is worth it.

We are now doing grey listing.  When a sender has not been seen before, a deferral response is sent (a 400 series response, not an outright refusal).  It’s accompanied by a message, “Please try again in 1 minute.”  Properly configured mail servers will try again because they understand that a deferral response is not  a refusal.   Spammer mail servers seldom try again because they need to send as much email as possible before they get black listed.

False positives can happen when the sending server is misconfigured.  Some servers fail to differentiate a deferral from a refusal.  They don’t try again.  Not to mince words, the person or people in charge of the server do not understand what they are doing.

Grey listing also helps us spot email being sent to web harvested addresses.  When we see the same email going to an architectural firm in Dubai, to a home tutoring service in Des Moines,  a car repair shop in San Diego and a rare antiques shop in London it’s a good bet that it’s spam.

We are blocking thousands of spam emails per day using these 2 techniques.

We have also started doing sender verification.  This is done by testing whether a bounce message to the sending address would be accepted.  If the sender address is fake, it’s reasonable not to take mail from it.  This is controversial because it puts load on other servers which are innocent of spamming and can be employed by spammers as an attack on those servers.  However, Gmail, Yahoo, AOL, Hotmail and others do this to our servers.  Fair is fair. But again, false positives can occur due to badly configured servers. The email standards documents say that bounces must be accepted when sent to live addresses.  Some servers refuse bounces because of user complaints that they are getting email returned they didn’t send.  There are solutions to that problem, but this isn’t it.  Some mail server administrators simply do not know that it is possible to tell the difference between verification and a real bounce.

Many people expect that email should be perfectly reliable and run their business partly based on the assumption that it is.  Unfortunately, that assumption is unrealistic.  Without filtering, email is essentially unusable because of the volume of spam.  The entire system is imperfect because it’s run by an unpredictable collection of imperfect humans.  It is flawed. To say it more plainly, it’s a mess.  Nobody in his right mind would design email to work the way it does on the Internet as it is today.

On balance, false positives are relatively rare.  For the vast majority of users they will never be a problem.  All the same, it’s a good idea to remember that they are possible.

New Anti-spam Measures

by

At Deerfield Hosting we work hard to reduce the spam (unsolicited email) our users receive.  About 95% of email arriving at our servers is discarded because we can identify it as spam.  None the less, the remaining spam can still be a significant annoyance.  The problem with further filtering is false positives.  We can’t be throwing away important emails.  It’s a hard problem.

We have been noticing for some time the same from address sending to many of our users in different and unrelated domains.  Most often this is due to web site scraping, email addresses harvested from web sites.  To identify this kind of spam, we have started tracking inbound from and to addresses and generating statistics in real time.  When a particular from address exceeds more than a few unrelated domains, subsequent email from that address is blocked.

We welcome feedback on this.  If you are noticing that you are getting less spam or if you can see no measurable difference, we want to hear from you.

 

Using HTTPS

by

OR: How Do I Change My Site to Use HTTPS?

Now that you have an SSL certificate installed on your site, the natural question becomes how to make use of it.   Just because it’s available doesn’t mean somehow magically it will be used.  You need to take some steps to get it into use.

Search Engines

You don’t want to lose any of the page ranking gains you may have made.  Google regards different URLs as different sites.  The URL http://www.YourDomain.com is different than https://www.YourDomain.com, which is different than http://YourDomain.com.  Among the first steps whenever you change a basic URL like this is to make sure Google understands that these are actually the same site.  You can let them know by using the web master tools as described here:

https://support.google.com/webmasters/answer/83106?hl=en&ref_topic=6029673

Yahoo and Bing have similar tools which you most likely will want to use as well.

Apache .htaccess Files

RewriteEngine On
# Force www
RewriteCond %{HTTP_HOST} !^www.domain.com [NC]
RewriteRule ^(.*)$ https://www.domain.com/$1 [R=301,L]
# Force HTTPS
RewriteCond %{HTTPS} !=on
RewriteRule https://%{HTTP_HOST}%{REQUEST_URI} [NC,R=301,L]

Replace domain.com with your domain name. The first rewrite condition and rule make sure that the site has been accessed using the www prefix. The part in brackets, R=301,L is what fires when the rule is matched and tells the web server to issue a 301 redirect to the https version of your site with the www. The “L” means “last rule”. The second part will fire if the site has been accessed using www, but not HTTPS.

Note: A 301 redirect is a permanent redirect.  A 302 redirect is a temporary redirect.  You may want to use 302 here during testing and change to 301 later.  Browsers remember a 301 and will make the substitution before trying to access anything.  This can cause a lot of confusion when minor mistakes are present.

Together, these rules will cause all accesses of your site to occur as https://www.YourDomain.com. Google recognizes a 301 redirect as a site URL change. This causes pages they have previously indexed which fall under the same URL to use the new one instead.

WordPress

Virtually all WordPress sites will want to have a 3rd section the .htaccess file above to allow for pretty URLs, using post names as access instead of numbered posts.  If those rules are present, make them the last rule set.

If you are setting up a new WordPress site, get your certificate in place before installing WordPress!  Doing so will save a lot of time later on.

Switching an existing wordpress site from http to https can be a challenge.  The first step is to change the URL, which you can do in the settings section.  But WordPress also embeds the URL in many places in posts, which sometimes leads to redirect loops getting set up.  There are scripts available to go through the WordPress database, changing all occurrences of http to https.

Internal Page References

When you hand write or edit html code to include links to other pages on the same site or resources such as images, you don’t need to include a protocol specification such as
http:// or https://.  Browsers understand that the reference is to the same site.

The main reason the protocol is sometimes specified is to make clear that the reference starts at the root of the site.  Then if pages are moved around, no editing needs to be done.  But even in those cases, you still don’t need the protocol.  The reference can be written as for example:  href=”//page.html”.  By using 2 forward slashes, you are telling the browser to use the protocol it has been using to access the page, starting at the topmost level.  This also makes the development process easier.  Without a domain name and protocol, all the code is much more portable and reusable.

Take Credit Where Credit is Due!

At some appropriate place on your site, let your visitors know that you are enhancing their privacy by encrypting your site for them. Awareness of security on the Internet is rising, but is nowhere near where it should be. Any little boost you can give it is a good thing. You deserve credit for doing your part.

 

SSL Certificates in Web Host Manager

by

Generating and installing an SSL Certificate using the Web Host Manager interface is easy.  The steps are:

  1. Click on “Generate an SSL Certificate and Signing Request”
  2. Fill in the text boxes with the appropriate information and click on “Create”
  3. Use cut and paste to copy the CSR, (Certificate Signing Request)
  4. Purchase your certificate and then use the CSR to complete the order in our billing system.
  5. Once you have your certificate, click on, “Install an SSL Certificate on a Domain”

Important notes:

  • If you want your certificate to work on both www.YourDomain.com and YourDomain.com, use the www version when generating a signing request.
  • If you want a certificate for a specific sub-domain (such as forum.YourDomain.com or store.YourDomain.com) you need to generate a CSR for that sub-domain.  If you have a large number of sub-domains to secure, consider a wildcard certificate.
  • As of this writing, cPanel makes outdated assumptions about additional certificates which may be required.  These certificates are called a “CA bundle”.  They are included when your certificate is sent to you.   The last box on the certificate installation screen shows the CA bundle as optional.  Don’t leave the box empty.  Fill it in with your CA bundle certificates.

You can order an SSL certificate from us here.  Select the type certificate you want from the category drop-down.