It Ain't Necessarily So

Today’s guest blogger is Steve Walters. Apart from his day job as a professor of economics at Loyola College in Maryland, Steve has served as a consultant to two MLB teams and writes occasional statistical analysis features for The Sporting News. He grew up in Salem, Massachusetts, remains a citizen of Red Sox Nation, and counts as his most cherished piece of sports memorabilia an autographed copy of MBA: Management by Auerbach.

Let’s start with a pop quiz: At what age does the average big-league ballplayer reach his peak?

If you said 27, you are an unusually conscientious student of sports and, quite likely, a devotee of the great Bill James, one of the founding fathers of sabermetrics and a consultant to the Red Sox.

You’re also wrong. (Profs love to do this sort of thing, don’t they? Make people nervous with an obscure question, let the brown-nose down in front show off a minute, then slap him down. This is why everybody hates profs.)

James took on this research question back in his 1982 Baseball Abstract. Baseball traditionalists held that players are in their prime from age 28 to 32. But James concluded that this belief was “blatantly false.”

He examined the career stats of 502 hitters born in the 1930s and found that, on average, they had their best years (“peaked”) at age 27. Most players, he wrote, “attain their greatest value before the 28-32 period even begins, are declining throughout that age range and have lost nearly half of their peak value by the time it ends.” Among statheads, that eventually became accepted as gospel.

Then along came Bowling Green State University statistician Jim Albert, who looked at a broader sample of data in 2002 and found that James’s findings were… well, flukey (see: http://bayes.bgsu.edu/papers/career_trajectory.pdf).

Albert examined the productivity of hitters born over six decades, and found that James’s 1930s sample of players, for whatever reason, peaked at a younger average age than any before or since. Players born in the 1910s peaked at age 28.0, those born in the Roaring ‘20s at 28.6, and Depression babies peaked at 27.1. Those born in the ‘40s, however, peaked at 28.9, the ‘50s at 28.7, and the ‘60s (thank you, modern training methods) at an average age of 29.8.

It’s fair to say then, that most players through history have peaked closer to age 29 than 27. Albert also found that half of the players born in the ’60s peaked between age 27.9 and 32, while a quarter peaked before and a quarter after that age range. That could be another statistical fluke, of course. But it could also mean that the traditionalists are right on target and the smart-ass sabermetrician led his readers astray.

Why do I bring this up? Emphatically not to suggest that profs always know more than best-selling writers like Bill James. Believe me, I know a lot of professors, and quite a few can’t find their hindquarters with both hands, much less divine the truth from data. (Especially humanities profs, but don’t get me started about that.)

The point is that it’s actually damned hard to figure out what’s really, really true by sifting through numbers. Sometimes profs do it better than intelligent laymen, and sometimes the reverse is true.

All I’m saying is that we need to be careful before we conclude that some “study” by anyone actually “proves” something. As the James/Albert episode points out, sometimes a well-constructed study coughs up a result for one era that turns out not to be typical of others. Or a researcher’s methodology may unintentionally twist things in a particular way. Or a boatload of statistical subtleties may confound things.

Unfortunately, thanks to Al Gore’s invention, we are awash in data, making it wicked easy to crank out studies and proofs. Post-MoneyBall, just about every big-league ballclub has hired some kid with an Ivy League diploma to “crunch the numbers.” Or actually call the shots.

Not that there’s anything wrong with that—until and unless we get sloppy, credulous, and excessively eager to do new stuff without making sure the old stuff is, you know, actually true. Like the old saying goes, it isn’t what you don’t know that does you harm, it’s what you know that’s wrong.

In academic research, there are (at least) two devices that help protect us against knowing wrong stuff. Before we publish something, we have to submit to blind peer review, which means a few jealous, picky, anonymous rivals get to dissect our work. Usually for excruciating months. And if it’s ultimately published, other rivals are invited to try to replicate our conclusions using our data, or new data from other samples.

This is not a guarantor of truth. Sometimes malodorous stuff sneaks through these filters, which reduce but don’t eliminate the chance of error. But there’s value to you, the reader, in knowing whether “research” has passed through such a vetting process. Caveat emptor. When you’re consuming statanalysis, ask yourself whether the author is an expert or pseudo-expert—and even then whether other experts have had a crack at debunking the work. (E.g., it’s notable that the book which inspires this blog is from a renowned university press, and that much of the research on which it’s based was initially published in refereed journals.)

And if you’re a researcher yourself, I’d encourage you to spend some effort on replication of others’ work. Ask for their numbers and crunch ‘em yourself, or get fresh ones like Albert did. (Aside: Has anyone studied when basketball players peak? Paul Pierce turns 30 before next season, and I’m worried that he’ll be completely cooked before the Celtics get good again.)

In any case, whether we’re readers or researchers our inspiration should be the great philosopher Porgy, who once famously confessed that “I takes dat gospel whenever it’s pos’ble, but wid a grain o’ salt.”

–SW

Comments are closed.