On cat herding and securing web dependencies

Why reinvent the wheel when someone else has already done the work? Shared building blocks speed up development. But how do we keep an overview of all the nuts and bolts we depend on? Software for the web is built from dependencies: modules, packages, plugins, bundles, components and whatnot, from many sources. New versions are released continuously. How do we keep this jigsaw puzzle secure? Must it be like herding cats?

Exit stage left-pad

NPM is a package manager for JavaScript, (in)famous for having a huge number of packages, many of which do small, trivial tasks. On March 23, 2016 the resolution of a naming dispute lead to a developer unpublishing all of his 273 packages, as he didn't accept the conclusion. One of his packages was left-pad, a utility for inserting characters on the left side of a string, consisting of 17 lines of code, including whitespace.

Thousands of applications worldwide used left-pad, some directly, and some via other packages that required it. When this chain of dependencies broke, the whole house of cards came crashing down, and applications could no longer be built or deployed. It was easy enough to fix, but it took 2.5 hours before things were working again.

When the other shoe, and thousands of tables, dropped

The publishing platform WordPress has a large plugin ecosystem. Plugins can interact with the core system in many ways. The plugin ThemeGrill Demo Importer lets you try out web page themes. It had a small flaw. It included code to reset a web site, including wiping the content (dropping all database tables) and then logging in the user as the administrator, if there was an admin account. The problem was that there was no authentication check for this. And it was exposed through an Ajax call anyone could access.

WordPress is estimated to be used by more than 60 million websites. With those numbers, it's not so surprising that some 100,000 websites had the vulnerable plugin installed. So, when the vulnerability was disclosed hackers started exploiting it. Whether they just wanted to wipe websites for fun, or had more sinister motives, or were really after the admin access they would get in some cases, the result was the same. The data was gone, and people were scrambling for their backups. (You have backups, don't you?)

Trust issues

Should we avoid dependencies, then? The thing is, no matter how you do your development, you have to trust someone. If you build everything in-house you need a large staff of developers that you have to trust to some degree. Even if you do everything yourself, you crazy person, you have to trust your abilities to get things right and not make some crucial mistake somewhere. And with cryptography, you have to trust the math, unless you have the brains and skills to verify it.

This is much like in the physical world. Running a company means you're trusting building engineers, locksmiths, security guards, and cleaning staff, whether they're in-house or outsourced. Dependencies are not the problem itself, it's how we manage it that counts. We need a process of trust management, which is related to risk management.

Know thyself and thine (vendors)

We need to consider which dependencies we use, and to what degree we can trust them. This is easier with 10 dependencies than with 10,000. Requirements vary; a blog can accept higher risk than a bank. If we choose to use a dependency tree bigger than what we're able to review, we should at least be aware of that fact, and plan for how we detect and react to breaches.

That a dependency has a lot of high-profile users is one indication of its trustworthiness, though an unscientific one. It also indicates it's a high value target for those looking for exploitable flaws. But generally, the more eyes are watching, the safer it will be. If the vendor of the dependency is themselves high-profile, so much the better.

eZ Platform v2.5 has 39 direct Composer dependencies, not including development packages. Eighteen of them are made in-house. Including the indirect dependencies, the total is about 110. Some 20 of these are made by Symfony/SensioLabs. JavaScript asset dependencies come in addition to this, as does the PHP language, Solr search engine, web servers, proxies, cloud vendors, and web browsers.

In tests we trust

Before we release a new version of eZ Platform, we test all aspects of it methodically. This means we can place a high degree of confidence in that version and all its dependencies. Note that our releases include a Composer lock file, which specifies the versions of all Composer dependencies. If you install using "composer install" you will get those exact same versions that we tested. If you do a "composer update", however, you may get newer versions of some dependencies, so beware of that. Verify before you deploy to production!

We have a library of packages on ezplatform.com that have been submitted to us. Some are made by eZ Systems employees, others by our partners. These packages can be very useful, but be aware that they will not have gone through thorough testing processes like our official releases. It's on you to verify that they are safe and fit for purpose.

OWASP stings again

Security is hard, and sadly, the entire field of software engineering is bad at what we do. Which is why I keep recommending the wisdom collected by the OWASP project. Using components with known vulnerabilities is on their Top Ten risks list, as a widespread problem with often disastrous consequences. They recommend removing unnecessary dependencies, inventory those you require, monitor for reported vulnerabilities, obtain dependencies from official sources, and monitor for unmaintained dependencies. Worth the read!

Please remember our guidelines for safely reporting security issues in eZ Systems products, and do the same for other vendors. Until next time, stay safe!