Software Testing - Advanced Meh

Now that I've thoroughly skewered unit testing dogma, it's time to turn our sites on the other types of testing within the software canon.

Let's start with an easy one - integration testing.  Integration testing is similar to unit testing, but tests groups of services (meaning as few as 2 and as many as ???).  Unit testing is expected to test functions or methods of tests in isolation.

Functionally, I think they serve the same purpose.  Their intention is to cover small parts of the application stack that are easy to examine and change in isolation.  But that hasn't stopped me from losing valuable years of my life sitting in meetings where it's absolutely imperative that we have bright lines between all of these terms.

Look, I get it.  There are cases where words matter - especially in diplomacy.  But, sometimes, sometimes, it's not really all that important if very similar functionality is grouped together and we're a bit cavalier about our word choice.  I know, as managers, that means we have to go find something else to badger our employees about, but it's important to note that pedantic and semantic aren't synonyms.  We should concentrate on the demarcation of terms where it matters.

[Side note: I learned the other day that Mark Cuban thinks people are stupid if they use larger words in place of smaller ones.  I think I understand his point - don't confuse people and impart a facade of pseudo-intellectualism in order to make yourself feel better.  But...English is a rich language and those words are there for a reason.  I could've said "don't confuse people and put on a disguise of fake smarts in order to make yourself feel better," in the previous sentence, but I didn't stress out about SAT vocabulary for a year simply to listen to some billionaire tell me I'm an idiot for using my education.  

There's a difference between using words to obfuscate and intimidate - which a lot of insecure software engineers do with our own jargon - and letting your vocabulary flow naturally.  There's a benefit to both approaches.  Clear communication is always the goal.  Rigid rules hinder that goal.]

Ok, so integration tests are effectively the same as unit tests for me.  If you read my last post, then you can guess the advice that's going to come next in regard to their usage - 

Use integration tests for designing and implementing your code (or testing out a specific use case).  Once the tests pass, you've recorded the code coverage, and they're checked in, delete most if not all of them.

You've used them for their intended purposes.  They're still available for posterity via git.  Keeping them around is like saving every newspaper as a personal window into current events, and we know how that winds up.  People don't find you to be methodical.  They, at best, label you 'eccentric' once you're found dead under a pile of extraneous The Onion copies from the 1990s (except for the one about running a new train line...up your ass!  That one's a keeper.  For those of you have no idea what I'm talking about, just nod politely and move along).

That was a lot of copy to say "treat integration tests like unit tests."  Mark Cuban would be pissed with my verbosity.  Oh well, a pseudo-intellectual can't change his faux-leather elbow patches.

The next type of test to examine is the end-to-end (E2E) test.  Compared to unit and integration tests, E2E tests hold significant value in being retained and run just prior to a software release.  Unfortunately, unlike unit and integration tests, maintaining E2E test stability is far more difficult due to their scope and complexity.

If it's not obvious, E2E testing is responsible for verifying large, critical swaths of functionality in your application.  In online travel, E2E tests may attempt to sign in, search, and book hotels, flights, cars, or travel packages.  That involves a lot of systems with delicate interactions.

Online travel has a lot of corner cases, but the problems faced in its E2E tests are probably not unique to its industry.  Let's look at a few examples to see the benefits and drawbacks of using E2E tests.

E2E tests around a user signing in are probably robust, since sign-in functionality is either completely controlled by your company, or the sign-in provider (Google, Facebook, etc.) is assumed to be rock-solid in their own reliability.

Sign-in functionality doesn't change that much, but its usage is critical.  The low volatility and criticality of its functionality make it an excellent candidate to add to a long-running test suite.

Other E2E tests prove to be a bit more dicey.  In the travel space, the inventory returned by search and book functionality is often controlled by someone else, if only in part.

This means that your E2E tests are reliant on someone else's systems to provide you with consistent, quality responses, often in a test environment, where consistent, quality responses are rare.  

With this in mind, you face a difficult choice when running an E2E test and it fails.  Do you choose to spend potentially significant time chasing down the cause and fix for the error if the problem is more likely to be a result of an unreliable API than your code?

Another option is to run the tests against a small subset of your production environment, but this could incur real costs if you're securing actual inventory and cannot easily release it at the end of the test.  If these tests are intended to run frequently (which is the case for most automated testing), you may wind up allocating non-trivial budget to these scenarios.

Yet another option is to mock the scenarios you want to test.  Depending on what aspect of your functionality you want to test consistently, this is either an acceptable trade-off or a reliance on a false prophet.  

If you're mocking out search functionality, the tests run fine, but the underlying response from the production API is completely different from what you've mocked up, don't be surprised at the failures, since your tests are running against a simulation, not a system as it exists in the wild.

With all that to consider, this is how I'd use E2E tests in my own systems:

• First, ensure that you're doing upfront work around reliability, so that you're not reliant on dodgy systems in either test or production that produce a lot of false failures leaving your engineers chasing non-existent problems.
• Second, rely on robust monitoring and alerting for your production systems.  While this won't stave off problems preemptively, monitoring - in conjunction with good system design - will limit the number of problems you face and allow you to fix problems quickly when they do arise.
• I prefer this approach to chasing bulletproof testing, because eventually, a bullet's gonna' get through, and limiting your risk is better than relying on a foolhardy notion of invincibility.
• Write E2E tests for low volatility, high criticality systems like sign-in functionality.
• Mock out when necessary, but ensure your team knows exactly what they're testing, not what they think they're testing.  Also make sure that they don't mock things to the point that what's expected to be a stand-in for a helicopter is, in fact, a representation of a club sandwich.
• Understand where your risks and weak points are, so you don't instill a false sense of confidence in your team regarding E2E tests.

And that's it.  Next up - performance testing!

Until next time, my human and robot friends.