Synergy and Talent Evaluation in the NFL

The weekly football column is all about “synergy”.  But before we get to that, let’s look at the weekly QB Score and RB Score rankings:

Week Eleven QB Score Rankings

Overall QB Score Rankings

Overall RB Score Rankings

Okay, we see the numbers.  Here are my football thoughts for the week.

When we look at basketball and football players, we think of the word “synergy”.  Players in these sports must work together for a team to be successful.  This characteristic of basketball and football appears to make the evaluation of individuals difficult.  Are the numbers a player posts due to his efforts? Or is it just his teammates?

One way to think about this issue is to consider the consistency of performance over time. If a player’s numbers changes as his teammates and coaches change, we suspect a player’s numbers are not about him.  In contrast, if numbers stay fairly constant even as teammates and coaches change, we suspect the role of teammates (and coaches) is not that important.

In basketball, we tend to see consistency.  We can see this in systematic regression analysis.  We can also see this when we look at the career performance of specific players like Kobe Bryant, Jason Kidd, LeBron James, Yao Ming, Tim Duncan, Kevin Garnett, and Paul Pierce.  Despite coaches and teammates changing, year after year these players are extremely good (and we can see the same pattern with “bad” players).

When we turn to football, though, we see a very different story.  Regression analysis tells us that the numbers posted by quarterbacks are extremely inconsistent over time.  We can also see this pattern when you look at individual players. Brett Favre will be phenomenal one year (2004) and below average the next (2005).  And you can see the same pattern when you look at many other signal callers.

Changes in Manning and Brady

One player who has defied this tendency is Peyton Manning.  An average quarterback will post a QB Score per play of 1.85.  Across the past eight seasons, the lowest mark Manning has posted is 2.36 (2001).  Six times he has posted marks that are over 3.00.

After seven games in 2007 we saw the same story.  Manning had posted a QB Score per play of 3.78, a mark that ranked second in the NFL (Tom Brady was #1).  And then something happened.  Marvin Harrison finally stopped playing after Week Seven.  Since Harrison went to the sideline Manning has had a couple of good games (most recently this past Thursday against Atlanta). But overall he has been below average, posting a QB Score per play of 1.53 since Harrison went completely away.

Let’s review.  Manning plays six games where he looks like Manning has always looked. And then Harrison – who Manning has been able to throw to since he entered the league – exits the field. Since Harrison left, the great Manning is suddenly below average.  This leads us to ask, is the greatness of Manning just about Harrison?

And Manning is not the only quarterback where we ask such a question. Tom Brady is in his 7th season as a starting quarterback for the Patriots.  Across his first six seasons he posted a QB Score per play of 2.10, with a career high of 3.01 in 2005.  After ten games in 2007, though, Brady has a mark of 4.72.  To put that mark in perspective, Manning posted a QB Score per play in 2004 of 4.78, the highest mark observed since 1994 (when my data set begins). 

How did Brady go from being a “good” quarterback to a quarterback who rivals the numbers of Manning at his best? One suspects the key is Randy Moss.  Why Moss?  Well, back in 2004, Daunte Culpepper, with Moss on his team, posted a 3.50 QB Score per play in 2004.  After Moss left Minnesota, Culpepper was well below average (before he got hurt).  In sum, Moss — when he is motivated — tends to make quarterbacks look really good.

What We Don’t Know About Quarterbacks

And so we see one aspect of the problem facing those charged with the task of evaluating quarterbacks.  How do you separate the quarterback from his receivers?  A quarterback cannot complete any passes without his receivers.  So is the completed pass because of the receiver’s talent, or the quarterback’s abilities? 

The problem is not just with receivers.  What about the impact of the offensive line?  Or the running game?  Or the play calling of the coaches?  All of these factors impact the numbers we assign to the quarterback.  And all of these factors are independent of the quarterback’s talent.

Given this problem, are we sure Tony Romo is worth $67.5 million?  Or is his above average play simply because he gets to throw to Terrell Owens (who is probably going to retire before Romo’s contract is up)?

Or what about the much maligned Joey Harrington?  Harrington has been about average in Atlanta this year, despite having little talent around him.  If Harrington got to throw to a motivated Moss (or Harrison, or Owens), would Harrington be a Pro Bowl quarterback?

A few weeks ago I brought up this very issue in talking about talent evaluation with an NBA executive.  Let me repeat what I said then.  Every NBA executive should be thrilled they don’t work in the NFL.   The NBA has very good data that allows decision-makers to both explain the past and forecast the future.  In contrast, the NFL’s data doesn’t tell us who was responsible for what we observed, and it doesn’t allow for adequate forecasting.

And before anyone imagines the NFL has some mysterious data source that does allow decision-makers in football to accurately evaluate talent and predict future performance, remember the pattern we see in the performances of Favre, Manning, etc… If people “knew” why these players were good or bad, I do not think we would see such wild fluctuations in player performance.   And current performance would have some statistical relationship to current pay (it doesn’t, but that’s a story for another day).  Instead, we find past performance and current pay are related (again, a story for another day).  This suggests that decision-makers do consider what they see in the past in evaluating talent.  But they find, as we report in The Wages of Wins, past performance in the NFL is simply no guarantee of future returns. 

– DJ

For more on QB Score, RB Score and what these metrics mean see

The New QB Score

Consistent Inconsistency in Football

Football Outsiders and QB Score

The Value of Player Statistics in the NFL

Comments are closed.