More Thougths on Maintenance, Updates and Fixing

The issue of maintenance, updates and software fixing actually deserves a few more considerations.

Since a long time, security experts have been saying that software updates are essential for the security of operating systems and applications. A long time ago instead, software updates were considered only when new features or features’ updates were needed. It took a long time but by now, at least for mainstream applications, periodic (eg. daily, weekly, monthly etc.) software updates, in particular for security issues, are standard.

Still there are many unresolved issues. First of all there are still systems which are not updated regularly, among them there are some “mission critical” ICT systems considered so critical that updates are not applied for fear of breaking the system or interrupting the service, and, at the opposite end, some consumer embedded systems considered to be of so little value that a security update feature has been dropped altogether. (In my previous post I briefly mentioned these two classes of systems.)

But there are two other aspects that must be considered. The first one is the time that it takes between the release of the security patch or update by the software vendor or maintainer, and its installation on the system. As in the case of both Bash Shellshock and OpenSSL Heartbleed, patches and updates have been released out of the normal schedule, and they should have been applied “immediately”. To apply “immediately” a patch released out of the normal schedule, one should at least have a way:

  1. to know “immediately” that the patch has been released
  2. to obtain and apply “immediately” the patch to the systems.

Obviously to do this one needs to have some established emergency security procedure in place. One cannot rely on someone reading/hearing the news somewhere and deciding by herself to act as a consequence.

The second aspect, again also relevant in the two incidents mentioned before, is what to do in the time between the announcement of the existence of the bug and the availability of the patch, what is called a 0-day vulnerability. Again there must be some well established emergency security procedure which indicates what to do. The minimum is to assess what is the security risk of running the systems un-patched, to investigate if there are other countermeasures (like signatures for IPS or other network filters, detailed monitoring etc.), and in the extreme case just to turn off the service until the patch is available.

It seems easy, but it is not yet easy to put in practice.

Why the Bash ShellShock bug is so threatening ?

This year we had already at least two bugs which someone claims “have threatened to take down Internet”: OpenSSL Heartbleed and Bash ShellShock.

I will probably be unconventional, but by now I believe that, in particular for the Bash ShellShock bug, a much bigger security problem lies elsewhere.

Indeed my personal experience with the Bash ShellShock has been that most affected systems have been fixed overnight by the next automatic scheduled system update. Some critical systems have been manually updated in a matter of a couple of minutes without any process interruption. So all systems have been fixed as a matter of usual maintenance of ICT systems.

So where is the problem? Why so much hysteria about it? How is it possible that even very large and well-know systems have been attacked exploiting this bug?

Since it seems that as of today there are still many (too many) systems vulnerable to both the OpenSSl Heartbleed and Bash ShellShock, we have to conclude that the problem does not lie with the availability of the fix but with the maintenance procedure of the systems themself.

It is then not so much a problem of the bug itself, but of those managing or designing the systems which adopt this software, and any software at this point.

I can see two major areas where there can be maintenance problems:

  • large systems which are built with the approach “if it is not broken, do not fix it”: often in this case maintenance, even security patches, is not applied, update and patching procedures are complex and executed rarely; of course, if the system has not been designed or configured with the possibility of being easily updated or patched, any vulnerability in any software component can become a real big problem
  • similarly but even worse, are embedded systems which have been designed with the “build it once and forget about it” approach: in this case just the idea of maintenance has not been considered, or it exists only in the form of re-installing the full software (but only if you know how to do it and if a fixed new version of the full software has been made available).

It seems to me that more than blaming those writing and maintaining these software components, we should first think about how to maintain the software on our systems, which are the procedures for normal updates and security updates, etc.