Software Testing - Now With Even More Meh!

During our last installment, I finished by mumbling incoherently that after writing unit tests for a given project and recording the code coverage, you should instantly delete all of them.  Maybe I was drunk when I wrote it.  Oh, wait!  No, I wasn't!  I meant it!

To be fair, this wild and crazy influencer life has gotten to me (I am much more apt to use exclamation points these days), and I admit to being a touch hyperbolic, but I do think you should delete most, if not all, of your tests after using them for initial design and testing work.

People will point out - and not completely irrationally, I might add - that the suite of tests you've created are a great way for tracking your quality when something breaks.  I also read on LinkedIn recently that writing tests gives you instant documentation for your code usage.

In theory, both of these axioms are true.  In practice, neither of these holds up well.

Let's take the second statement first.  Unit tests aren't really great documentation for your code.  Code can be difficult to understand for a myriad of reasons - predominantly because it's either poorly written or because the concept it's implementing is extraordinarily complex, and it can be difficult to follow without additional explanation elsewhere (whether via comments - not my preference because they tend to be redundant - or via a separate document - wiki, README.md, etc.) 

Unit tests don't solve this problem because they're still code.  If the problem is that the code is poorly written, and the tests were written by the same author, there's a good chance that the unit tests will also be poorly written.  

Even if the code is well-written, unit tests aren't treated with the same sanctity as the code on which they're based, so they're always more likely to degrade than the code itself.  In addition, the nature of unit tests is such that they tend to be more voluminous than the code itself, because they're intended to test numerous configurations of the code under test.  In the jargon of computer science, the code itself serves a declaritive purpose, laying out what the code is intending to do.  The tests, by contrast, are imperative in laying out how the code is expected to cover all reasonable scenarios.  

As such, they're much more verbose.

If the code is complex, then unit tests have a chance to unmask some of that complexity, but typically at the expense of even more verbosity.  And, this isn't generally a situation where a longer explanation using simpler words shines a light on the underlying concept.  It's more like a senior lawyer using "simple" law words to a junior lawyer to distill a particular concept in jurisprudence.

Unit tests may not abuse Latin in the same way the law does, but the effect is still the same - confusion for all without an extremely close reading.

This leads me next to the apparent gold standard of generating a testing history encoded in your suite of unit tests.

It is true that an existing suite of tests will expose a recently introduced regressive bug, if you run the tests after every change, but the ocean of information you need to swim through in order to find and fix that regression may not be worth it.

First, it's not uncommon for engineers to comment out or skip tests in a suite that tend to be fickle or complex (see the above section).  If the bug you expose is in one of those skipped tests, you'll either never encounter it - rendering that section of the test suite useless - or you'll need to run those skipped tests and potentially add hours or days to your workload, possibly to update tests that covered critical functionality or possibly to ensure you're just chasing the arbitrary green light with no practical gain.

In theory, those tests and the functionality that they cover should be encapsulated during the initial implementation phase.

But, you say (after taking a long dramatic pause and sitting back in your deep plush chair before tenting your fingers and responding)...software isn't that simple.  

I would respond by standing up, accidentally knocking the silver tea set between us to the floor, blush in response, and say "Exactly!" while silently hoping that neither of us will notice that a pot of Earl Grey is seeping into the carpet.

Anyway.  You're right.  Software isn't that simple.  So, why assume that a suite of very brittle, sometimes unused tests, serves as a proper safety net?  This is just testing theater.

You've already taken a calculated risk after the implementation phase that there won't be bugs in the code you wrote.  If, while you're making updates to the code, you expose a bug that the initial tests didn't cover, then write tests that cover that use case and check them in, so there's a record that can be analyzed and adjusted later.

The code quality (with its previously thoroughly tested functionality now deleted) isn't actually lower, it's just not something you can immediately spot.  Then again, slogging through 30 minutes or more of running a test suite waiting for errors isn't something you can immediately spot, either

What's more important is to develop a culture of writing good code that doesn't tend to have spooky side effects at a distance in your codebase.  We've all been there.  Somehow a change to a payments module now breaks something in the sign-in functionality.  WTF!?!?

Is it entirely realistic to avoid these entanglements?  No.  As we agreed before I dumped the tea, granulated sugar, and cream on the priceless Persian rug, software is complex.  You can't get away from strange couplings.  You can only make sure they're not a common occurrence in your team's (or, hopefully, your company's) software, so you're not in a constant state of firefighting in the name of better quality.

If you're successful in this venture, then you'll still be able to write quality code that's easy to test and debug without needing to search through virtual haystack after haystack when your testing needle breaks.

I've still got a number of thoughts to devote to other software tests (end-to-end, performance, integration, and even manual), so I'll stop here and go look for a large wad of paper towels and an insurance policy to cover the damage to the rug.  Don't worry.  It'll all be fine.

Until next time, my human and robot friends.