We human types like to evaluate, compare, analyze, and decide in order to have the very best. This, by the way, is a good thing. Except when the data doesn’t reflect reality. After all, even most exacting logic fails us when a base assumption is incorrect.

Comparison tests for whatever you’re interested in are fun to read and give you a starting point when determining what’s “best”, but are not in any way an absolute indicator of “bestness” (no matter what the magazine wants you to think). A skilled rider on the worst motorcycle in a comparison would slaughter a mediocre rider on the best motorcycle. Every time. Ditto for bicycles, cars, etc. Engineering and manufacturing have gotten to the point where there are truly few lemons and magazines are forced to pick winners based on relatively irrelevant data.

Case in point for results not reflecting reality. In a recent comparison in a popular mountain bike magazine, five bikes were evaluated and a definitive rank order given. Except that, based on the comments, #3 would have leapt past #2 with different tires. Additionally, #4 was hurt by grips, seat, and other minor items. These were $3,000 bicycles and no one willing to drop that much cash on a bike is going to leave it stock. So 4^th place (loser!) could be fixed for a relatively small amount by replacing or adding parts that are 1) relatively inexpensive; 2) often replaced anyway based on personal preferences; and 3) would still give the bike a total out-the-door price that’s less than 1^st-3^rd place. These results are meaningless! [Fourth place bike? Wouldn’t touch it. What? I can make it comparable to the top bikes and it still costs less – done!]

So what’s that mean in business? A few examples, though you can probably think of many more.

• Personality assessments used in hiring are generally highly overvalued. Whereas they are very useful for development, they rarely provide much useful information for hiring. Sure there’re pretty graphs a and comparison ratings that make us feel like we are really comparing hard data. Just like with magazine comparisons though, they are pretty good at identifying the outliers to avoid, but just don’t give any definitive answers to compare normal folks. [This person is a “7” on sociability versus this other candidate’s “6.5”. But which can do the job better? How much sociability is required? How much is too much? What other personality traits would balance a low or high score? How do you know? How do you really know?”]
• Turnover rates. We want those as low as possible right? Um, maybe. Some turnover is actually good. You obviously want to retain the high-performers, but do you really want the dead weight sticking around. Low turnover might be a sign of awe-inspiring leadership. Or maybe it’s a sign of a very weak leader not holding people accountable for performance.
• Expenses – cut those down to the minimum. Well, that doesn’t work as a singular measure. Salary and benefits are a huge expense, but if we lay all the employees off no work will get done and no money will be made. Don’t want excessive inventory, but get rid of all inventory and it’s a little difficult to satisfy customers. Even looking at equipment: is the cheapest the least expensive? Probably not if we factor in maintenance and downtime costs. Same thing but back to employees: the cheapest employee may not be the least expensive when we factor in productivity and ease of managing.

Are any of these bad measures? Nope. It’s just crucial to remember that they aren’t definitive or absolute measures. They provide some, but not all, data. The risk is to draw too firm of conclusion from them, especially when they don’t capture real world use. Although all pieces are necessary to make the puzzle, you can’t make the puzzle from just one piece. The all time classic example of this is the “11 is one louder” scene from “This is Spinal Tap”.

Most decisions are not cut and dried. What works really well for someone else may not work for us at all. Everyone’s situation is different; their needs unique. “More” is not always better and “most” can be counterproductive. Rather we must balance out a number of factors to decide which option best fits our specific individual needs.