Steven Pinker, Malcolm Gladwell, and Me

A few days ago I was reading Marginal Revolution and I came across the following: Pinker reviews Gladwell.

A few months ago I saw Steven Pinker – a Harvard psychologist — in a rather lengthy interview on C-SPAN.  After the interview I ordered Pinker’s latest book: The Stuff of Thought: Language as a Window into Human Nature.  So I was familiar with Pinker’s work and was curious to see what he had to see about Gladwell.

The above link takes one to the New York Times review Pinker wrote of Gladwell’s latest book: What the Dog Saw.  The review begins by praising Gladwell’s gifts as a writer.  But it soon takes a turn as Pinker begins to argue that Gladwell – a journalist – is often not an “expert” on the topics he writes about.  My reaction when I read this was that this seemed to be an unfair criticism.  Journalists – as anyone whose work has been discussed by members of the media would know (and Pinker falls into this group) – do not claim to be “the expert.”  In fact, this is why the call on people like Pinker.  In other words, if journalists were the experts, they could skip the practice of talking to people like Pinker entirely.   

Pinker, though, had apparently not considered this point and quickly moved on to an effort to illustrate Gladwell’s perceived shortcoming.  What the Dog Saw is a collection of essays Gladwell originally wrote for the New Yorker.  And part of this collection is the following article Gladwell published last year: Most Likely to Succeed: How do we hire when we can’t tell who’s right for the job?  Readers of this article will discover that part of the story Gladwell tells focuses on how hard it is to draft a quarterback in the NFL.  The “experts” Gladwell called upon to tell this story were two economists named David Berri and Rob Simmons. 

In an article Rob and I published in the Journal of Productivity Analysis, we discuss how the statistical relationship between a quarterback’s draft position and his future performance is quite weak.  Pinker, though, apparently disagrees.  Nestled in his New York Times review is the following: It is simply not true that a quarter­back’s rank in the draft is uncorrelated with his success in the pros….

Upon reading this I went from “Hey, I think Pinker is being unfair to Gladwell” to “Hey, I think Pinker just attacked my research.”  So I decided to e-mail Malcolm and ask if he knew of Pinker’s source for his statement.  The question was mostly a joke.  My thinking was that Pinker’s reaction was simply a manifestation of the problem sports economists often face.  Sports are a subject matter with an abundance of “experts”.  Essentially, many people who have only watched sports often believe that they are an “expert” on the subject.  Having conducted research on sports and economics since the mid-1990s, I am accustomed to people who have only watched sports disputing the findings of my co-authors and I.  So I was thinking along the lines… maybe Pinker is a fan of the Jets, and he really hoped Mark Sanchez was going to live up to his lofty draft status.

Pinker’s response to Malcolm, though, revealed that Pinker was not just stating his belief as a sports fan.  Pinker claimed he actually had sources.  Specifically — as Malcolm detailed at his blog– Pinker found some stuff on the Internet that contradicted what Rob and I said in our article.

My sense is that Pinker never read our article.  What he did find on the Internet is evidence that a quarterback’s aggregate performance (i.e. passing yards, seasons played, Pro Bowl appearances) is indeed related to draft position.  And as Rob and I detailed in our article, this is true.  Aggregate performance and draft position are statistically related.  But as Rob and I argue, this is because in the NFL (like we see in the NBA) draft position is linked to playing time.  And this link is independent of performance.  In fact, Rob and I find that draft position – again, independent of performance – impacts a quarterback’s pay many years into a quarterback’s career.

To correct for this bias, we focused on per-play statistics.  And here is a sample of what we found.  After a quarterback has played five seasons in the NFL (minimum 500 career plays), here are the correlation coefficients between draft position and various career statistics:

Completion Percentage: -0.01

Passing Yards per Pass Attempt: -0.02

Touchdowns per Pass Attempt: -0.12

Interceptions per Pass Attempt: 0.00

QB Score per Play: -0.01

Net Points per Play: -0.02

Wins per Play: -0.02

QB Rating: -0.06

Our data set runs from 1970 to 2007 (adjustments were made for how performance changed over time). We also looked at career performance after 2, 3, 4, 6, 7, and 8 years.  In addition, we also looked at what a player did in each year from 1 to 10.  And with each data set our story looks essentially the same.  The above stats are not really correlated with draft position.

But that is not all we did.  Rob and I also looked at what factors determine where a quarterback is selected in the draft.  We then looked at how the factors that determine draft position predict performance.  This study revealed that the factors that get a quarterback drafted are not related to what he does in the NFL. 

Again, I don’t think Pinker read our paper before commenting.  And in this sense, he did the same thing he wants Gladwell to stop doing.  Pinker commented on a subject where he was not well-informed. But the difference is this… Gladwell makes an effort to understand the subjects he discusses.  Pinker didn’t make this same effort.

Beyond this point, let me also comment on the tone of Pinker’s critique (which is similar to the tone of many comments at Gladwell’s blog).  Many people seem to want Gladwell to only write about “the truth.”  In other words, people want Gladwell to focus on research that everyone agrees upon.  But for research to be “interesting” it must say something new.  And more often than not, the “new” research will contradict “old” beliefs.  Consequently, people who liked the old stories will disagree.  So when Gladwell talks about Allen Iverson not being a great player, or Kevin Garnett being underrated, or the inability of draft position to tell us much about a future quarterback’s performance, people who hold the “old” beliefs will disagree (and argue the “new” research is clearly bad).  According to Gladwell’s critics, Gladwell should see this disagreement and find something else to write about. 

But that is not a particularly reasonable approach.  Again, “interesting” research will challenge people.  Such challenges often provoke negative reactions.  To only choose topics that will avoid such reactions reduces your range of topics to a number close to zero. 

One last note… Martin Schmidt and I have a book coming out next March called Stumbling on Wins.  This book will provide more details on quarterbacks and the NFL draft, and also discuss a host of other decisions made in sports.  If you enjoy this story about the NFL draft, we hope you will enjoy our next book.  Of course, if you didn’t like our draft story…. well, we are sure you will love all the other stories we tell :)

- DJ

The WoW Journal Comments Policy

Comments are closed.