For at least the last decade, I’ve been telling readers that open-source development matters and helps make better software. If everybody can read the code of an application or an operating system, there can’t be any hidden backdoors; if anybody can rewrite that code to fix vulnerabilities and add features, the software’s progress can’t be thwarted by any one company’s distraction, fraud or bankruptcy.

My glowing endorsement of Mozilla Firefox 1.0 in November 2004 set the tone:

…the beauty of an open-source product like this is that you can participate in its evolution. Firefox’s code is open for anybody to inspect and improve...

Since then, I’ve recommended open-source operating systems, office suites, anti-virus utilities, secure-deletion tools, file-encryption software, two-factor authentication apps, PDF exporters, DVD rippers and video-playback toolkits. And I’ve had one phrase in mind each time: Given enough eyeballs, all bugs are shallow.

My experience using open-source software tells me this is true–even if that doesn’t guarantee a constant rate of improvement or an elegant interface.

And if any genre of software should benefit from this method of development, it ought to be code that Web sites use to secure their interactions with users from eavesdropping: Everybody sending or storing private information needs this feature, billions of dollars of transactions are at stake, and you don’t even have to worry about wrapping a home-user-friendly UI around it.

True, right? Except Heartbleed happened. Two years ago, an update to the widely-used OpenSSL encryption library added a “heartbeat” function that made it easier for sites to keep an encrypted session going. But it also harbored an disastrous vulnerability to buffer-overflow attacks that would cause a site to return 64 kilobytes of whatever happened to be adjacent in the server’s memory to an attacker: usernames, passwords, e-mail content, financial transactions, even the private key the site uses to encrypt the session. And the attacked site can’t check afterwards to see if it got hit. I defy the NSA to script a better hack.

And despite buffer overflows being a well-known risk with documented defenses, nobody caught this for two years. Two years! It took a Google researcher and engineers at the Finnish security firm Codenomicon to find the bug separately and report it to the OpenSSL team.

How bad is this? Ask security researcher Bruce Schneier:

“Catastrophic” is the right word. On the scale of 1 to 10, this is an 11.

It seems that everything that could go right in open source development went wrong in this case. As an excellent story from Craig Timberg in the Post outlines, the free nature of OpenSSL made it an obvious choice for hundreds of thousands of sites and something of a natural monopoly, that same enormous deployment of OpenSSL encouraged people to assume that they themselves didn’t need to inspect the code that carefully, and OpenSSL developers got so little financial support from the corporations relying on their work that they couldn’t even subject their code to a proper security audit.

The stupid thing is, we knew this could happen. See John Viega’s 2000 essay, “The myth of open source security,” in which he outlines how thousands of users failed to catch “a handful of glaring security problems” in code he’d contributed to the Mailman mailing-list manager:

Everyone using Mailman, apparently, assumed that someone else had done the proper security auditing, when, in fact, no one had.

That doesn’t mean that closed-source development suddenly looks better. (When all this is done, Microsoft’s proprietary and hideous Internet Explorer 6 may still have greased the skids for more successful attacks than OpenSSL.) But it does mean that selfishness/laziness/distraction and open source can become a toxic mix, one we should have seen coming.

Updated, 10:25 a.m., to add a link to Viega’s prescient article.