Information Overload

Recently, I've been part of a few conversations that end with "maybe we can use this [some very particular] metric to track the status of these outcomes."

A good metric is always worth its weight in gold.  It can take a floundering process and generate the glue that drives progress forward.  The problem is finding the right metric.  With the amount of information available to everyone today, it's easy enough to pick a metric that will fit your dataset in one fashion or another even if doesn't add meaningful information.  And, if it doesn't add meaningful information, it's just contributing more noise.

Don't get me wrong, having access to more information is always beneficial, but as the years progress, it seems to be less important (in most cases) to actually concentrate on obtaining access to information and more important to craft it into something that has meaning.

I also expect that there will be a premium on services that cultivate this information for us ahead of time - maybe not for everyone, but for people who don't want all of their choices selected by an algorithm that seems to drive us further toward stupidity.

This tyranny of choice has been present for a while (think of all the different brand choices you have for pasta alone in the grocery store) but it gets exponentially worse online.  Broadcast TV had its problems with fomenting a mono-culture and tending toward a lowest common denominator, but at least I didn't have 100 options, 90% of which are shit staring me in the face when I needed something to watch.  Often, I'd just turn on the TV and become intimately familiar with "I Dream of Jeannie" or "The Beverly Hillbillies" reruns, because that's all there was.  Bruce Springsteen lamented about having 57 channels with nothin' on.  Man.  Those were the days.

Information overload is why, as I grow older, I spend more time searching for curated lists from sources I trust (or at least human sources).  There's always the risk that some critic will label something a masterpiece that I find simply inscrutable, but I prefer that to mass-produced dreck that keeps me encapsulated in a safe lukewarm bubble while incrementally upping the temperature.

However, even the curated lists now seem to be at risk.  I recently read an article on Spotify's preference for placing AI or unlicensed artists in its feed recommendations.  And though I didn't read specifically that their editors' picks were facing this kind of pressure, their business practices and race to squeeze as much profit from their users makes me far from certain future lists won't contain my favorite bot making a mash-up of all bubble gum hits from the past 40 years without all of the memories of the individual songs (Yeah, I still love my some Hanson.  Screw You!)

If you thought the Music Industry couldn't get any more soulless, you were wrong!

I also swore off recommendations from Kindle years ago.  It's either trying to upsell me on the latest potboiler from its book mill or it's letting its algorithm inform me that I'll like 4 books - all from the same author - simply because I just finished reading that author.  I don't mind reading multiple books by an author (Zadie Smith and Jane Austen immediately come to mind), but it's rare that I want to read everything by an author in one fell swoop.  I'm a sucker for the eclectic.

In a work setting, the danger isn't so much that someone's actively trying to drown me with informational sludge.  The danger is overfitting the data that's present to fit a preconceived hypothesis.  Or it's a reliance on the data to provide a clear path without realizing that there are nearly infinite paths.

This second problem occurs when attempting to track too many metrics for process improvement.  The DORA metrics are great at measuring the maturity of a software development organization.  But people tend to find them too simplistic and bolt on additional information assuming it will give them a better outcome.  

Apparently, even the DORA metrics themselves are subject to self-doubt.  The original 4 have now been supplemented with a Reliability metric attached to a very vague definition.  I can see the original 4 being supplemented with a service error budget, which, in combination with Mean Time To Recover, is a strong indicator of reliability.  But, the Reliability metric seems to include the error budget, MTTR, and a handful of other indicators - a potpourri for whatever makes your org feel good about itself.

The reason people augment metrics is to instill a sense of control.  They anticipate that such broad measurements as deployment frequency, lead time for changes, change failure rate, MTTR, and time a service is unavailable (my opinionated measure of reliability) are unable to fully capture the complexity of their systems.

To a large extent, they're correct.  We live in a complex world.  But, the flaw lies in trying to preemptively define further statistics without proper observation and experimentation.  In the case of MTTR, just about everyone will define a secondary Mean Time to Detect (MTTD) immediately.  MTTD, though, is just a subset of MTTR.  

If your MTTR meets your org's acceptable limits, then MTTD is extraneous.  Note, first, that I said "your org's acceptable limits."  That's an important number to establish and something most people don't give careful thought to.  You don't want your MTTR set to 0, because this is planet Earth, and things will go awry.  It is impossible to build a perfect system and you will go broke trying.  If you're able to successfully establish an MTTD of 5 minutes and meet that target, then - huzzah! 

Remember, though, that the M stands for Mean, so you'll recover from some problems sooner and some later. Forgetting that leads you into the same pitfalls as chasing perfect availability.  

But, again, let's assume you consistently meet your 5 minute goal.  If so, don't bother tracking MTTD.  You've met your goal, why overthink it.  I mean if you were lucky enough to become a billionaire, you wouldn't arbitrarily compare yourself to the other billion...nevermind, forget that example.

However, if your MTTR target is 5 minutes and you're routinely clocking in at 10 minutes, then MTTD becomes a vital statistic, at least temporarily.  It may turn out that the vast majority of your time is spent in detection.  In that case, you now have a statistic to help you improve (one that you can later discard once you've succeeded in your process improvement).

If you start out with too much information via too many metrics, you're once again subject to the whims of the tyranny of choice.  Which one do you focus on, especially if they contradict each other?  How much time and effort are you going to spend maintaining their upkeep?  How much time and effort are you going to spend fixing them if they're out of compliance?  If you're not maintaining them, you're just adding noise to the system that makes operation even more difficult, so why have them in the 1st place?

Again, wanting to create this sense of control is understandable, especially in the work world.  Work, in theory, is the one place where the chaos of life can be managed.  Things can be portioned into projects, and, even if things are over budget or past due, there's a finite amount of money or time that can be added to rectify the situation.  So people create smaller and smaller discrete components to further instill that sense of control, assuming that a simple sum will result from the whole of the parts.

Except a large number of discretized components explodes complexity.  Think about it - what's easier to pick up and play, a guitar or all of the atoms that compose the guitar?  

The more you quantize things, the more subject they are to small changes in variability and feedback loops.  So, my advice - pick the exact amount of information you need to reason about and control your system - no more no less - and proceed.

Also if you determine how to pick the exact amount of information you need, please let me know.  If you don't then opt for less information until you need more.  Realize that in making this choice, you may be throwing away valuable information, but hindsight is always 20/20 and there's a cost to hoarding, even with petabytes of storage available to you.  So, Keep It Simple, Stupid, and Hang In There, Baby!

Until next time, my human and robot friends.