I Broke My Hamburger
Why Unit Tests Are A Vital Part Of A Continuous Integration/Continuous Delivery (CI/CD) Pipeline
by JamesQMurphy | May 13, 2020
Any change to your source code has the potential to break your application -- including version upgrades inside your
packages.json file. Most developers know this. In fact, I'd wager that most developers have dealt with a bug introduced from a third-party component, which is why we treat third-party upgrades just like any other software change.
I certainly know it. But every so often, I need a little reminder.
Your Friendly Neighborhood Dependabot
About a week ago, I noticed a failed build in my Azure DevOps build pipeline for a pull request that I didn't submit. Given that I am currently the only contributor in my repository, I was very curious to see who it was.
Turns out it was Dependabot, which automatically scans public GitHub projects (like mine), looking for vulnerable dependencies that it can identify. And it found one: JQuery had just released version 3.5.0, which addressed an issue that could make a site vulnerable to a cross-site scripting attack. Not only did Dependabot discover that I was still using jQuery 3.4.1, it submitted a pull request. The pull request build failed only because Dependabot didn't know about my version number checks, but that was a quick fix on my end. In fact, I was able to make the fix right on the GitHub website. After a successful pull request build, I closed the pull request, which triggered a main production build.
But I Tested It
Under my CI/CD process, production builds are first deployed to a staging environment. I use my staging environment as a sanity check, to make sure the version of code I'm about to promote doesn't cause the site to crash and burn. But it is a full-blown environment, complete with a Lambda function and API Gateway setup. The only resources that it shares with production are the DynamoDb tables. I can test quite a bit of the application if needed. In other words, no excuses!
Now, I did actually test the application, but barely. I use jQuery in my commenting system, so my testing consisted of making sure that comments were still visible. Once I saw that comments were still being displayed, I confidently approved the release for deployment to production.1 Because it's a public project, you can relive this momentous occasion yourself at this link.
So what was the problem? I forgot that Bootstrap also uses jQuery, and jQuery 3.5.0 broke the collapse functionality of Bootstrap (you can read about the issue on GitHub). This affected the "hamburger" icon on mobile views, but it also broke the context menu for logged in users:
The bug was totally reproducible in all environments -- not just staging. It was even reproducible on my local development system. So how did it slip through? Shouldn't I have tested my entire site?
Well, yes, I should have. But the only way I would have caught this bug was if I manually tested it, and since those kinds of tests are, well, manual, there's no guarantee that I would have tested it correctly. But here's the thing; I shouldn't have to test it manually. Therefore, I'll be looking at how to create browser-based unit tests in the near future. I've never done it before, but Selenium looks interesting enough to try.
A good CI/CD pipeline has solid builds and fast deployments, and should allow you to deploy changes quickly and easily. I was able to deploy both the initial security fix and the follow-up patch release with almost no effort at all. But without proper automated testing, a fast pipeline simply means that you can deploy bugs to production just as quickly as new features -- or security patches.
I think I was little too eager to accept a security fix proposed by Dependabot. Not that I'm blaming Dependabot -- the blame solely rests with me.↩