The problem with experts


My friend Lee once noted a problem he had to me. He was fully capable of scaling how much better he was when compared to someone worse than him. However, when it came to comparing himself to someone much better than himself, he had no sense of scale. Now, Dunning-Kruger possibilities aside, this is an astute observation.

We struggle with this problem a lot. Sure, when we learn a little about a subject, we can tell when someone knows less than us (or think we can). But when we encounter someone that knows more than us, well it becomes very hard to gauge. When I’ve argued that better communication is not the silver bullet to sports analytics, it’s because of this! You see, if multiple people know more than us and they disagree, how do we decide who to trust? The answer? We often go with either the people we like, or the ideas we like.

In fact, this blog has fallen for that very trap! Take Jonah Lehrer, who wrote Imagine. When this book came out, several of us were very impressed with it. None of the regular writers at the Wages of Wins are neuroscientists. Now, Lehrer has a degree in neuroscience from Columbia university, so it’s fair to say he knows more than we do. And his books had very compelling tales that were written in a great style. Except, his science was wrong in many cases! He knew more of the vocabulary and more of the studies to cite, but that didn’t mean he was the right person to listen to. So why did we? Because neuroscience isn’t a trivial field. Lehrer clearly knew more than us, and his tale was engaging. It is worth noting that Dave did directly contact Lehrer before his review went up and asked him if all of the studies he cited were accurate. Lehrer said they were. Which, of course, shows another problem: people can lie!

It is easy for popular wrong ideas to catch on. And sometimes this is done by “better communicating” You ever hear the line: “We only use 10% of our brains…” Yeah, it’s completely wrong. We use over half our brain just to see. Of course, a common theme around here is that people don’t fully understand how their eyes work. What better method of communicating is there than a simple line that explains the thesis and a great network to share it? Movies, books, etc. have moved this expression like wildfire!

Pankaj Ghemawat had another brilliant example. Tom Friedman popularized the notion of “the world is flat” in regards to globalization. This idea is cited everywhere! Pankaj had the following insight in a recent TED talk.

Let me give you an example. When I first published some of these data a few years ago in a magazine called Foreign Policy, one of the people who wrote in, not entirely in agreement, was Tom Friedman. And since my article was titled “Why the World Isn’t Flat,” that wasn’t too surprising. (Laughter) What was very surprising to me was Tom’s critique, which was, “Ghemawat’s data are narrow.” And this caused me to scratch my head, because as I went back through his several-hundred-page book, I couldn’t find a single figure, chart, table, reference or footnote. So my point is, I haven’t presented a lot of data here to convince you that I’m right, but I would urge you to go away and look for your own data to try and actually assess whether some of these hand-me-down insights that we’ve been bombarded with actually are correct. 

Let me stress that again. A best selling book  written by an “expert” was apparently devoid of any data to back its claims! Ideas that sound good and people good at communicating are great if you want the idea to spread. But, this says nothing of the intrinsic value of the ideas themselves. Communicating better can just as easily spread a bad idea as a good one. Which is why, when we judge our experts, we should be careful not to just side with ideas that meet our prior beliefs, or with people we like. And, if it’s not possible to learn the subject well enough, we should at least know the results to check for. I feel often we simply look for the people that have credentials better than ours to say the things we like to hear. And sadly, that’s just one of the problems with experts.


Comments are closed.