That's not a football!
Dave Berri is the General Manager of the Wages of Wins Network. He is a Professor of Economics at Southern Utah University, lead author of both “The Wages of Wins” and “Stumbling on Wins”, and past president of the North American Association of Sports Economists.
Earlier this month, ESPN introduced the Total Quarterback Ranking (QBR). This metric was developed by a number of people at ESPN (led by Dean Oliver) in an effort to replace the NFL’s current quarterback rating system.
As Dean Oliver noted in explaining QBR, although this measure is new, it clearly builds upon work that has been done in the past. Specifically, it builds upon the work of Brian Burke at Advanced NFL Stats (work – that Brian often notes – builds upon the work of many others). In fact, Brian also helped the ESPN people building QBR by explaining the metrics presented at Advanced NFL Stats.
Soon after QBR was introduced, Brian offered an evaluation of this effort at Advanced NFL Stats. Here is what Brian liked about ESPN’s stat.
- It includes sacks, running, fumbles and all the other important things that the traditional NFL passer rating doesn’t.
- It doesn’t double count anything, as the NFL passer rating does with completions.
- It is based primarily on EPA (Expected Points Added), which accounts for down, distance, and field position.
- It is also based on WP (Wins Probability), which considers time and score.
- It’s a rate stat instead of a cumulative stat.
And here are some issues that Brian has with QBR.
- It is proprietary.
- It is unit-less.
- It is an amalgamation of other stats.
One can go to Advanced NFL Stats for Brian’s complete discussion of these issues. In this forum I wish to briefly address a different issue.
Brian and I recently co-authored a chapter in a forthcoming academic volume exploring the economics of professional football (a volume edited by Kevin Quinn). Our chapter focused on how statistics can be used to evaluate players. More specifically, we looked at the work presented in The Wages of Wins (QB Score and Wins Produced) and also all the wonderful work Brian does at Advanced NFL Stats.
Part of our discussion focused on an issue that any football metric should address. Does the measure tell us much about future performance? Decisions are statements about what a player will do in the future. But if a player’s performance – as measured by any specific metric – is inconsistent, then the metric probably doesn’t help a decision-maker actually make decisions.
To address this issue, we looked at the consistency of various measures utilized to capture a quarterback’s contribution. QBR was not available when we wrote the chapter (we finished our chapter last April and the book should come out… well, sometime in the future). But now that we have three years of QBR data, we can now look at the same issue with respect to ESPN’s new stat.
The methodology is follows. We look at quarterbacks who played consecutive seasons. We then consider how much of a quarterback’s performance in the current season is explained by what a quarterback did the previous year. Here are the results Brian and I report in the chapter, with the QBR results added.
The first three measures – Expected Points, Win Probability Added, and Success Rate – come from Advanced NFL Stats. Wins Produced comes from The Wages of Wins (it was updated for this chapter) and QB Rating comes from the NFL. Of these, the most consistent is Success Rate while WPA is the least consistent.
We also presented the four elements of the NFL’s metric. As one can see, interceptions per attempt are essentially random (a point made in The Wages of Wins).
Finally we see the ESPN’s measure. The observations for ESPN’s measure are currently limited. But we can see that in the limited sample we have, ESPN’s metric is more consistent than the NFL’s metric. It is also more consistent than Expected Points, Win Probability, Wins Produced, and the NFL’s metric. However, it is not quite as consistent as Success Rate.
Of course, the consistency of all these measures pales in comparison to what we see in the NBA (72% of a player’s ADJ P48 is explained by what he did last year in the NBA). But it is the case that QBR is an improvement – at least from this perspective – on the NFL’s metric. And since that was the stated objective, we can argue that QBR is a success.
- DJ
P.S. All that being said, I do think Brian brought up some valid points. And hopefully those issues can be addressed in the future.
