Web Site Security

We are responsible for more than 30,000 user logins on computers which had been susceptible to the Heartbleed bug.  As you may imagine, that means we have been watching very carefully for signs of trouble.  So far, we have found no evidence of any compromise.  What this does not mean is that complacency is acceptable.

Media reports have in some cases flatly stated that this bug allows an attacker to compromise any server with the bug.  That is, to gain access to the server at the administrator level.  This is almost total nonsense.  As you will see from what follows, the problem is entirely limited to data flows.  It’s true that if an administrator happened to log in while an attacker was running an exploit, there is a slight chance that his login name and password were captured.  On our systems I can say unequivocally that this never happened.  We use a two factor encryption system.   The second part was exposed to the bug, but the first part was never in jeopardy.

As an aside, this bug has created a much needed uproar in the security community.   I checked the online banking system I use.  I found that while it was never susceptible to the heartbleed bug, it was susceptible to man in the middle attacks.  That is nearly as serious!  I suggest you check and complain at any secure sites you need to use.  Here is a way to check:

Website Security Test

Just put the subject website name in the box.

Very few of our users have followed our advice and changed their passwords.  If you are one of them, please do so.  It’s true that media reports have been overblown and the probability that your account has been affected is low.  None the less, this is NOT something to ignore.

We have more information now about this bug, how it works, what it would take to compromise a server and what it would take to compromise any individual account.  It’s not nearly as bad as first reported. 

It’s a buffer overflow vulnerability.  When a program starts up, it allocates memory to hold what it needs to do its work.  In the case of openssl (the program with the bug) as it services requests it allocates and frees memory to temporarily hold the information flowing in and out.  When a request has completed, the pointer to this memory is discarded.  The problem is that whatever was in that area is still there.  The bug allows an attacker to request and get what was left over from the last request which used that memory area.

Memory is allocated on what’s called the heap, from the bottom up.  When openssl starts up, the first thing it does is load public and private keys into the heap.  This means the private keys are in the lowest part of memory.  Following is a diagram of subsequent memory allocations:

The client machine is allowed to check with the server periodically to see if the encrypted connection is still intact.  This is called a heartbeat.  The client sends the server a test packet and the server sends it back as verification.  The bug is that the server fails to sanity check the packet size which is provided in the first few bytes of the request, “length” in the diagram.  It returns a memory area as large as whatever the client put there.  If it’s bigger than what was sent, material left over from the last request is sent.  If that happens to contain something sensitive, we have a serious problem.

Initially it was reported that the attacker could gain access to the keys to the kingdom, the server private encryption keys.  Security experts, with an abundance of caution, are unwilling to flatly state that it is impossible for an attacker to get these keys, but that appears to be the case.  The keys are located in the lowest portion of memory which is never freed and reused.  Using an assortment of scenarios, literally hundreds of millions of attempts have been made to get the keys.  None has yet succeeded.

UPDATE: Although the process is difficult, it has now been proven that it was possible to obtain the server keys.

Again, the likelihood that your passwords have been compromised by this particular bug is very low.  Our servers were updated and new security certificates issued within 3 hours of the announcement. 

However, recent security studies have been done on the source of the bot net (robot network) problem.   A bot net is a group of compromised computers being controlled remotely by someone other than the owner.  They are used in concert or individually to attack other computers.  The studies have provided good evidence that about 37% of computers connected to the Internet with broadband connections are participating in bot net activity.  This means there is roughly a 1 in 3 chance that your computer is one of them.  If it is, you can bet your passwords are out there.  If you have been complacent about security, you really need to check for this.

Today, a serious and pervasive threat to security on the Internet was revealed: the so-called Heartbleed Bug. In my opinion, everyone who regularly uses a password on a “secure” Internet connection should have at least a rudimentary grasp of the problem. A web site has been set up to describe it in detail: Heartbleed.com.

The short version of the problem is that an encryption vulnerability was found.  Under certain circumstances, a third party can decrypt your session with a secured web site or impersonate a secured web site.  First the attacker must obtain the encryption keys from a secure site and this is what the bug allows him to do.  Once the attacker has the keys, if he can get access to what is flowing back and forth between you and the site, he can read it.  That includes passwords, credit card information, all of it.

News reports have given the impression that with stolen keys, an attacker can walk right into a server and get whatever they want.  This is wrong.  If a server administrator logged in remotely while his session was being read, the attacker could then log in with the same credentials.  This is quite different and not at all likely.  Most servers have constraints on where administrators can be when they log in.  It would set off alarms.

The likelihood that information has already been stolen from you is low.  Normally we don’t see security bugs exploited until they are well known.  This problem was first discovered last week and was announced publicly today.   The delay was to allow time to get fixes in place.  Our servers have been updated and certificates replaced.  We are no longer vulnerable to this threat.

What is IMPORTANT is that any secure sites you interact with have been updated.  If they do not post a notice, you should ask before logging in.  I just attempted to find out if my bank was aware of the problem.  I was unable to get an answer.  Hopefully the people who manage the web site have taken care of it, but the only safe assumption is that they have not.  I’m not going to use web banking until I can get an answer and you shouldn’t either.

We have many emails this morning with questions about privacy and security.   Given the news over the week-end, this is not surprising.  A person with more than top secret security clearance at the NSA (the [American] National Security Agency) revealing secret capabilities is a very big deal.

If you haven’t already heard about this, let me suggest that you get information from as close to the source as possible.  I have already observed news sources injecting bias.  Most of that is due to ignorance, but some appears to be willful.  Most reporters lack the background and will dish out what they have been spoon fed by some “expert”.  There is often a heavy bias.  Be careful what you believe.  The real story is NOT the whistle blower!  It is what he is talking about. The Guardian

The intent here is to provide some basic information about network security.  It relates to email and your personal information on the Internet.  Network security is a huge topic.  Any opinion about what the NSA has done or may do in the future will be hopelessly naive without a reasonable understanding of what is possible.

The recent revelations have been no surprise to people involved with computer security.  That the capabilities exist has been common knowledge for a long time.

Passwords

To get an idea how long it takes to crack a password, take a look at GeodSoft Password Cracking Time Calculator. The problem with this site is that it doesn’t mention what computing power is being brought to bear.  The time it takes to crack the typical password with a typical desktop computer is about 2 days using brute force methods (trying every combination).  Using dictionary words cuts that down to under an hour.

To consider what the NSA is capable of, you can divide that by at least 1 million.  An article about passwords with more detail.

A good password provides adequate protection against criminal activity, but this is only true for 3 reasons:  1) most criminals are stupid,  2) smart criminals have an abundance of easy targets,  3) what you have that they want isn’t worth the trouble.  If you make it hard, they will move on.  If someone smart with access to a super computer wants to know your password, he can get it.  You have no defense, unless you also have a super computer.

SSL and TLS – Secure web pages and email

SSL and TLS use public and private keys to provide encryption.  The source computer provides a public key which the destination computer uses to encrypt what it sends and decrypt what it receives.  It takes a lot of computing power to do this without the private key.   It is in essentially the same class as very good passwords.  For some (scary) detail please read this.

Many years ago in a college class on computer security, the instructor described a paper written in the late 1970s by a friend of his, a mathematician.   She had used a PDP-11 to generate mathematical key signatures which could then be used to crack any encryption in existence within a few minutes.  If you don’t know, a PDP-11 had considerably less computing power than your cell phone.  When she was about to present the paper, she was quietly taken aside by some unexpected guests.  The paper was never presented anywhere nor published and she moved on to other areas of research.  It’s safe to say that the NSA and FBI know all about her work.  It’s also safe to say they have expanded on it over the last 30 years.

SSL is excellent protection against common criminals and snooping individuals, but against the resources of a government or a consortium of smart criminals, it’s useless.

Implications

There are techniques which go beyond what is described above.  The simplest to understand employ rotation schemes.  They are based on the idea that if it takes 1 minute to crack a cipher, but the cipher is changed several times per second, in theory the system can’t be cracked.  In practice, it boils down to the attacker simply needing several thousand times the computing power of the target.  Too hard for criminals, relatively easy for governments.  The NSA can protect its secrets.  Individuals can’t.

Most likely you will see news stories about who has and has not given unfettered server access to the NSA.  Google, Yahoo, Facebook and Microsoft, just to name a few, are loudly proclaiming that they have not.  Given that the NSA has no need to be “granted” access, this is completely irrelevant.  If they want access, they have it.  It’s as simple as that.

Over the next few days you will hear various assertions being made about the safety of your personal information.  You need to listen carefully because there are no absolutes.   It is impossible to fully deliver on guarantees.  Every case is relative.

Data Mining

This is the process of detecting patterns in data which have implications and then searching for other occurrences of the same patterns.   It goes beyond seeing who a terrorist was in phone contact with.  When an organization follows standardized procedures, their activities generate patterns.  For example, a terrorist sleeper cell might be detectable from phone and Internet records without any advance knowledge of the individuals placing or receiving the calls, just from their frequency, duration and places of origination and termination.

The problem is that the target organization can be anything.  That includes a group of individuals who might be seeking political change.  Having identified such a group, counteracting it by co-opting its goals is a common political strategy.   So is discrediting the individuals involved.  Information is power.

Those are the facts.  You can choose to believe or not believe how far the NSA has gone.  You can choose to trust or not to trust the government of the United States.

It is a historical fact that no significant weapon ever developed has gone unused.  Even nuclear weapons have been used without being fired in the same way a gun pointed at someones head is a weapon being used.  I personally think it would be naive to believe that it’s all a mirage or that these capabilities will never be abused.   It’s instructive to remember G. Gordon Liddy and why the American government has a division of power.

The question is what to do about it.  Would you be interested in enhancements to protect your email privacy?  To protect your on-line privacy?  The integrity of your information on our servers?

You may be interested in learning more about The Tor Project

Please comment.  If you are uncomfortable doing so in public, do so in private.

This morning we had an email asking about setting up a secure web site.

[note color=”#a5f0fc”]I want to set up my own private “cloud backup” because the one I bought into and set up was a big ripoff and I’m pissed at them. My problem is that my own websites are not secure, get hacked, etc.

I wondered if I bought a SSL certificate, would that make one of my domains hosted with you be totally like Fort Knox or just no difference at all, except more outgo (expense) for that particular domain.

Is there ANY solution to get super safe online storage whatsoever? [/note]
There is no such thing as a totally secure web site. As far as that goes, there is no such thing as a totally secure server either. This applies to everyone everywhere. Always. The only computer which is totally secure is one which is OFF.

Having said that, it is quite possible to have and maintain a web site which can be characterized as “safe”. You only need to do a reasonably good job of security and it is extremely unlikely that you will ever get hacked. The miscreants who do these things don’t need to go to great lengths to find exploitable web sites. If you just make it hard for them, 99% of the time they will simply move on.

Your web sites do seem to get hacked at a greater rate than our other customers. I suspect that this is because you buy and install so many php scripts.

It is the basic nature of PHP that it is insecure.  If you simply write code, it will be vulnerable.  Having written it you need to go back and with a very sophisticated understanding of how compromises are engineered, bullet proof it.  99.9% of amateur programmers lack a sufficient understanding of security to do this.

Probably more than 50% of professional programmers lack the skills as well.  It’s hard.  Take Word Press as an example.  It is written by the best.  Yet every few months new vulnerabilities are found in it.

Since the advent of broadband and computers typically always on, the number of computers connected to the Internet which are (in varying degrees) compromised is presently estimated to be about 35%. In other words, more than a third of those machines is compromised. The people who do these things have gotten very good at it. The basic problem is that the design of Windows operating systems is flawed regarding security. Attempts to make it and keep it secure are band-aids after the fact.

You need to use a very high quality virus scanner and keep it running. Because scanners use signatures to identify viruses and new ones appear constantly, it’s not enough merely to have it running. You can get infected with a new one not yet in the database. This is why you need to periodically run scans, to pick up what may have slipped through.

Did you have in mind to use your account with us as an online backup solution? Is that what you meant by “cloud backup”?  This is against our terms of service.    TERMS

With 5 copies of everything and the use of very expensive servers to provide fast web site service, it’s ridiculously too expensive to be used that way. We can provide such space if you really want it, but have to charge for it separately.

Consider buying yourself a hard drive with a USB interface for backups. Unplug it and it meets the OFF condition I mentioned above! It’s also faster and easier than an online solution.

Super long and complex passwords only provide slightly better security than one which simply has: upper and lower case; a number; a special character (like ‘#’). Don’t waste time on this. An 8 or 9 character password which meets those conditions is fine.

An SSL certificate merely encrypts traffic to and from a web site. It is a significant improvement in security to log in and administer back-ends using SSL. But this is not the basic problem. If there is a vulnerability in a program or script, it is as easily exploitable over an SSL connection as over one not encrypted.

My guess is that more than 99% of site compromises I see are done using kiddie scripts.  A kiddie script is an attack script to exploit a particular vulnerability in a particular set of scripts.  They are downloaded and used by people who have no idea how they work.

If you pay basic attention to security and keep your scripts up to date, your chances of ever getting compromised are very low.

This morning, February 19, 2013 at about 3 AM an email arrived which set off alarms. Monitoring software on one of our servers had discovered a suspicious file: /lib64/libkeyutils.so.1.9.

Investigation revealed that this file is part of a server compromise. How the attacker is able to put this file in place is as yet unknown. What we do know at this point:

• RedHat Enterprise servers including CentOS and Scientific Linux are affected.
• Logins via SSH are recorded, including login name and passwords.
• Other logins, such as to email and cPanel are not affected
• Only 1 IP address has yet been recorded as the recipient of information
• More than 10,000 servers have been affected so far
• The goal of the attacker has so far been limited to sending spam email

To mitigate the threat we have set up scanning to find and remove suspicious files at 5 minute intervals and send alert emails when any suspicious file is found. This will trigger further investigation.

Since the source of the infection is unknown, the only prudent course is to assume the worst. We have set up firewall rules to prevent communication with the single IP address known to be receiving information. However, it would be naive to assume that this walls off the problem.

If you notice that your service is running more slowly than usual, the likely cause is actions we are taking to deal with this threat. It is possible that your service will be interrupted. Some counter measures are disruptive. For example, when server load becomes very high it can appear that a server is down because response is so slow. The fastest way to regain control in this case is a reboot.

If you call or email and do not get an immediate response, the reason is apt to be that we are working on a problem. We sometimes need to choose between solving a problem and explaining to 20 or 30 people that we are working on a problem. Frankly, it makes more sense to fix now and explain later.

If you observe problems with your service while this threat remains active, please be patient. We are all over it.