Uncategorized

Many people like to use forwarders so that they only have 1 email account to check.  Sometimes problems occur with this.

Unfortunately, forwarding to gmail accounts is problematical. Forwarding to yahoo.com is even worse. The 2 things which commonly go wrong is gmail deciding forwarded mail is spam and deleting it and the other throttling sending rates.

The cause of the problem is people complaining in their gmail account about spam which has been forwarded to them. We do fairly well at stopping spam, easily 98% and that is as good as anyone, including gmail. But when several or many people mark the remaining spam as what it is, gmail regards the likelihood of mail coming from our servers as being spam much higher. That is, they think spam just came from our server, the probability that the next incoming emails are also spam is much higher. That’s incorrect in this situation, but you can see why it might seem to make sense. When their system reaches a probability high enough, they start throttling and even discarding emails. They would discard emails when other characteristics of a particular email also suggest that it’s spam.

I have sent emails to them several times asking what might be done about the problem. The only response I have gotten is being referred to their bulk mail guidelines. Not applicable and quite pointless. This is not an unusual problem. They ought to account for it, but they don’t.

There are a couple of things you can do. One is to stop using gmail directly by setting up a forwarder:

https://support.google.com/mail/answer/10957?hl=en

You would then check the account which you set up as the forwarding target instead of gmail.

The remaining things you can do are to use an email client instead of web mail and set it up so that it checks multiple accounts or simply check several places.

In the last 6 months we have twice had nearly simultaneous drive failures leading to service outages.  It was hard at first to grasp how something so seemingly unlikely could have happened.  When it happened a second time, it was time for some serious scrutiny.  What seemed like common sense might not be correct.

We run servers in pairs.  A hard disk on one server has a corresponding hard disk on another server.  If one disk fails, the service can simply be powered up on the other server.  Both servers have to be down to cause an outage.

The solution became obvious once the problem was understood.

Google released a study of their experience with a very large population of hard disks and failures.  If you have a taste for a dry technical paper, you can find it here: Google media research  What they found was revealing.  The data set is based on consumer grade drives.  We use enterprise grade drives which have a much longer life, but the general observations will be about the same.  This summarizes failure times:

As expected, hard disks show a high infant mortality followed by a period of (in our case) several years of reliable service.  Then suddenly failure rates increase.  That there is a decline in failure rates at 4 years is unexpected, but a gradually increasing rate after that is just what you might expect.  There is no data to support a rise and fall like that with enterprise drives.  It may or may not happen.

When we think about reliability, what we want to know is the likelihood of a failure event in a given time interval.  Then we can make statements (these are made up numbers) that the odds of a drive failure in a server over a months time are 1 in 300.  Then if a drive in a second server which is being mirrored to is the same, the chance of both drives going down in the same month becomes 1 in 600.  Since replacing a failed drive and re-mirroring takes 2 days, that would make a 1 in 9,000 chance of a failure before we could recover with no down time.  That seems reasonable enough, but it turns out not to be correct.  The problem is the failure rate distribution.

Many people are familiar with “the bell curve”, what in statistics is called the normal distribution.  The graph looks like this:

If you tossed a coin 5,000 times and kept track of how many times in a row you got heads and tails and graphed it, that’s what it would look like.  The left being heads counts and the right being the tails counts.

Hard disk manufacturers supply a statistic meant to show product life called the mean time between failure – MTBF.  If the number was 5 years, the expectation is that most drives would last about that amount of time.  What they report generally doesn’t relate to reality very well as the Google paper shows.  Still, it’s a useful statistic.  If the MTBF is 5 years and we charted a large population of disks, you would expect the chart to be a normal distribution with 5 years being the top of the curve.  Lacking data, my guess at the standard deviation of a set of 5 year MTBF drives would be something like 3 to 6 months.  Failure of a specific drive is random within a time frame so it’s reasonable to expect a failure curve to look something like a normal distribution.  We are (were) working with 2 sets of hard drives all manufactured at the same time, all in exactly the same kind of server and in service for exactly the same amount of time.  What that means is that the top of the curve is going to be much narrower and the sides much steeper.  In statistical terms, the standard deviation will be a much smaller number.

So, the obvious solution?  Add randomness.  Add new drives, but not all new drives.  The older drives have life in them yet.  Besides being a waste of money it would lead to the same situation if we simply replaced all of them.  What we have done is replace half of them.  Each replication pair consists of an older drive and a newer one.  When an older drive fails it will be replaced by another older drive until we run out of them.  Introducing new drives will therefore be at relatively random intervals.  This will move the top of that curve all over the place in terms of single drives.  We may not see the odds against double failures as high as 1 in 9,000 but clearly it will be a huge improvement.  It would be nice to have actual data for predictions.  We don’t, so I will have to make a guess.  Based on a lot of consideration, 1 in 1,000 seems reasonable.  It’s also a number we can live with.