In the March 8^th issue of the ESPN the Magazine is an article by John Hollinger on the subject of plus/minus.  In “Fuzzy Math: Plus/Minus Tell a Story, Though Not the Whole One”, Hollinger details the problems with the latest addition to the standard box score.  Unfortunately I was unable to find an on-line version of the article.  So let me try and summarize the issues Hollinger raises.

• The first critique comes from Dean Oliver (author of Basketball on Paper and currently the stats person with the Denver Nuggets).  Oliver is quoted saying, “It’s (the plus/minus measure) noisy, uncertain and kind of a black box – you have a hard time understanding why its coming out the way it is.”
• Hollinger also notes the plus/minus stat doesn’thelp us compare players across teams.
• On a related point, the stat also doesn’t take into account substitution patterns.

Much of what Hollinger says in this article was originally stated in an article he first posted at ESPN.com in 2005 (insider access required). That article also noted that a player’s teammates impacted his plus/minus.

Fans of this approach, though, might argue that all that’s needed is adjusted plus/minus.  This approach – originally developed by Wayne Winston and Jeff Sagarin – employs regression analysis to control for a player’s teammates.  Theoretically, adjusted plus/minus should answer Hollinger’s critiques (but doesn’t – as I will note in moment – Oliver’s criticisms).

When we look at the adjusted plus/minus numbers, though, it doesn’t look like Hollinger’s issues have gone away.  Consider the case of Darius Songaila.

Last season Songaila posted a -0.076 WP48 [Wins Produced per 48 minutes] with the Washington Wizards.  This result is not unusual.  Songaila posted WP48 marks in the negative range in 2005-06, 2006-07, and 2007-08 (he posted a 0.056 mark – his career best – as a rookie at the age of 25 in 2003-04). 

Adjusted plus/minus, though, told a very different story.  According to basketballvalue.com, Songaila was the third best player on the Washington Wizards in 2008-09.  Again, adjusted plus/minus is supposed to control for a player’s teammates.  So when the New Orleans Hornets acquired Songaila, they probably expected to see a positive adjusted plus/minus as well.  But currently Songaila is posting the lowest adjusted plus/minus in New Orleans.   So with completely different teammates, Songaila – according to adjusted plus/minus – is a very different player.

This should be thought of as odd, though, since Songaila’s WP48 with the Hornets is still in the negative range.  In other words, his box score numbers are not very different despite the fact his teammates have all changed.

This result is not just confined to Songaila.  JC Bradbury and I – in a forthcoming article in the Journal of Sports Economics — report that only 7% of a player’s adjusted plus/minus is explained by what a player did the previous season (oddly enough, unadjusted plus/minus has a stronger – albeit still relatively weak – correlation).  In other words, the correlation coefficient for adjusted plus/minus from season-to-season is below 0.30.   And when we look at players who switch teams – as Songaila did – we fail to find a statistically significant relationship. In contrast, any measure (PERs, Wages of Wins measures, NBA Efficiency, Win Shares, etc…) based on the box score will have a correlation coefficient of at least 0.65, and often these marks are above 0.80.   And that correlation remains strong even when a player changes teams.

What does this mean for decision-makers?  Decisions are about the future.  Unfortunately – because plus/minus is so inconsistent across time — it doesn’t appear this measure can be relied upon to make decisions about the future.

It’s important to note that inconsistency is not the only problem with this measure.  The standard errors associated with this measure – even when multiple years are added – tend to be so large that for many players the results are statistically insignificant (Bradbury and I make this point in our article as well). 

Even if the problems of inconsistency and the standard errors could be solved, the critique from Dean remains.  As Dean notes, this measure is essentially a “black box.”  A decision-maker has no idea why a specific result is obtained.  So it’s hard to know what the results mean.

One can state this last critique as follows:  What plus/minus can show is a correlation.  When a specific player is on the court, a team tends to do good or bad.  But it doesn’t show causation.  And therefore, it’s hard for a decision-maker to know really what this means.

Of course, all of this doesn’t stop decision-makers from using this information. And as Avery Johnson details, the Golden State Warriors upset of the Dallas Mavericks in 2008 can be partially attributed to Johnson following the dictates of plus/minus analysis.  

Let me close with three more observations. 

• One senses that people might be able to tell a story about why Songaila’s plus/minus numbers have changed.  Such stories, though, are also a problem.  Analysis should begin with a story, and then this story should be tested.  We should try and avoid looking at a test and then making up a story.
• We should note that adjusted plus/minus analysts have fully acknowledged Dean’s observation that this measure has “noise.”  Unfortunately, when specific players are analyzed this observation seems to vanish.  In other words, we never seem to see someone argue that a player’s current adjusted plus/minus is just “noise.”  But if there is “noise” in the model, some of these results have to also be “noise.”
• The fact that some teams have turned to such measures confirms what has been argued about a traditional approaches to player evaluation.  Teams are turning to these measures because the traditional approaches do not appear to work. 

So although adjusted plus/minus has problems, it is understandable that teams are turning to this measure.  One suspect, though, that the problems – detailed by Hollinger, Oliver, Bradbury, and I — are simply not well understood by everyone.

- DJ

The WoW Journal Comments Policy

Our research on the NBA was summarized HERE.

The Technical Notes at wagesofwins.com provides substantially more information on the published research behind Wins Produced and Win Score

Wins Produced, Win Score, and PAWSmin are also discussed in the following posts:

Simple Models of Player Performance

Wins Produced vs. Win Score

What Wins Produced Says and What It Does Not Say

Introducing PAWSmin — and a Defense of Box Score Statistics

Finally, A Guide to Evaluating Models contains useful hints on how to interpret and evaluate statistical models.

David Thorpe – of ESPN.com – recently wrote a column comparing Kevin Durant to Carmelo Anthony (insider access required).  Thorpe’s analysis considered a host of factors including shooting, scoring, making teammates better, on-the-ball defense, secondary defender, rebounding, and intangibles.  For each category the players were graded on a 10 point scale, and the player with the most points was…

Wait, before I get to Thorpe’s answer, let me comment on the word “intangible.”  This word means “not tangible” or something that we cannot discern or measure.  And yet, Thorpe is able to tell us that Durant offers more “intangibles” than Melo (by a score of 7 to 5).  So we can’t measure “intangibles” but we know Durant offers more? 

Thorpe argues that Durant’s value – according to Thorpe’s scoring system – is 48 while Melo scores a 44.  So ½ of the difference between Durant and Anthony can be linked to something that – by definition – cannot be measured.  Thorpe is not the only person to abuse the word “intangible”.  But it’s odd to see someone assign a number to something that by definition, isn’t tangible.

Okay, let’s take a more tangible approach.  We begin with Durant.  Table One reports what Durant – and his teammates with the Oklahoma City Thunder – have produced after 60 games in the 2009-10 season.

Table One: The Oklahoma City Thunder after 60 games in 2009-10

As one can see, Durant leads the Thunder in Wins Produced.  Of the team’s 36 wins, 13.3 can be linked back to Durant. 

Moving away from the subject of Durant for a moment… one can see that the Thunder are led by a collection of young players.  Durant, Russell Westbrook, James Harden, and Serge Ibaka are all younger than 22 years of age.  And this quartet are on pace to produce 36 wins this season.  As has been noted in the past, young players (younger than 24 or 25) tend to get better (so although Thabo Sefolosha is already quite good, he is not likely to get much better).  This means that prospects for Durant and the Thunder are extremely bright.

Now let’s turn to Carmelo Anthony and the Denver Nuggets.  This team already is quite good.  But as Table Two indicates, Denver’s success is not really about Melo.

Table Two: The Denver Nuggets after 61 games in 2009-10

Of Denver’s 40 wins, only 4.7 can be tied to the production of Anthony.  And four players – Chauncey Billups, Nene, Chris Andersen, and Kenyon Martin – have done more for the Nuggets this season. 

Now we “know” what Thorpe said (remember he favored Durant over Melo) and what we learn from this analysis must be wrong.  On Wednesday night, Anthony and the Nuggets crushed Durant and the Thunder.  And when we turn to the box score, we see that Melo posted a 16.5 Win Score.  Meanwhile, Durant only posted a 1.5 mark.  So there you have it.  Melo is clearly better than Durant.

Okay, obviously one game is not much of a sample.  Let’s look at Table Three, where what Durant and Anthony have done with respect to all the box score statistics across the entire season is noted.

Table Three: Comparing Kevin Durant and Carmelo Anthony in 2009-10

When we look at the entire season we see – as we saw when we looked at Wins Produced – that Durant has done more.  Durant is currently offering more with respect to shooting efficiency, rebounds, blocked shots, and personal fouls.  consequently, we shouldn’t be surprised that Durant is producing so many more wins than Melo.

When we focus strictly on Anthony we see that he definitely scores points in large quantities. But his shooting efficiency is only average.  So yes, he is above average (because he rebounds and gets to the free throw line).  But Anthony is not quite as valuable as his scoring average suggests.

Let me close by once again noting how far Durant has come.  Despite being named Rookie of the Year, Durant had a disastrous rookie season.  Last year, though, his production was above average.  And now – at the age of 21 – he is a star.  If he continues to improve – and the same happens with his young teammates – Oklahoma City is going to be a dominant team in the NBA for many years to come.  So although the Thunder were crushed on Wednesday, the Thunder will be more like to be the crushers – as opposed to the crushees (crushees???) in the future.  And that outcome should be quite tangible. 

- DJ
The WoW Journal Comments Policy

Our research on the NBA was summarized HERE.

The Technical Notes at wagesofwins.com provides substantially more information on the published research behind Wins Produced and Win Score

Wins Produced, Win Score, and PAWSmin are also discussed in the following posts:

Simple Models of Player Performance

Wins Produced vs. Win Score

What Wins Produced Says and What It Does Not Say

Introducing PAWSmin — and a Defense of Box Score Statistics

Finally, A Guide to Evaluating Models contains useful hints on how to interpret and evaluate statistical models.

Not only do I write books, I also like to read once in awhile.  What follows is a brief discussion of several books that I recommend (and yes, I am pretending that other people should be interested in my thoughts on books).

Beyond Batting Average

Lee Panas – of Tiger Tales (one of my favorite baseball blogs)– has just released his first book. Panas describes Beyond Batting Average (available for immediate download and in paperback) as follows:

… this book is designed to help knowledgeable baseball fans gain a better understanding of the multitude of new statistics that have been introduced on the Internet and elsewhere in recent years. It puts everything in one place and ties all the metrics together into an organized 15 chapter story.

This comprehensive sabermetrics primer will introduce fans to these new measures with easy to understand explanations and examples. It will also illustrate the evolution of baseball statistics from simple traditional measures to the more complex metrics used today. You will learn how all the statistics are connected to winning and losing games, how to interpret them and how to apply them to performance on the field. By the end of this book, you should be able to evaluate players and teams through statistics more thoroughly and accurately than you could before

My copy of this book arrived yesterday.  Looking through the book it is clear that Panas – despite writing a relatively brief book – manages to cover much that’s known about the measurement of the performances of the hitters, pitchers, and fielders.  So if you are interested in learning more about the wonderful world of baseball statistics, I highly recommend this book.

Mathletics

Beyond Batting Average focuses entirely on baseball.  Mathletics – by Wayne Winston –covers baseball, basketball, and football.  Much of what Winston discusses on his blog – waynewinston.com – is adjusted plus-minus (APM).  Only about 30 pages of Mathletics, though, are about APM.  The vast majority of the book presents a host of interesting examples of how the study of statistics can help us understand sports. 

There are few observations to make about this plethora of stories.

• Mathletics is very much in the Moneyball tradition. In other words, one theme in Mathletics is that the traditional – non-statistical – approach to the study of sports will often lead people astray.
• Not only does Winston tell his stories, he often shows the readers how Excel can be used to study sports.  So this book is a marvelous tool for students.
• Of the box score methods used to analyze basketball, Winston has problems with the Player Efficiency Rating and says nice things about Wins Produced.  So I especially liked that section.

That being said, I am not a fan of APM (and we very briefly discuss some of the problems with APM in our next book).  Again, much of the Mathletics is not about APM, so even if you share my concerns with this method you will still really like this Winston’s book. 

How We Decide

Jonah Lehrer’s How We Decide has just gone to paperback.   This book offers a wonderful discussion of how human beings process information and make decisions. 

One issue Lehrer emphasizes is the limitations of the human mind.  People often state that one should try and look at “everything” before making a decision.  With respect to basketball, that would imply a decision-maker should look at the box score statistics (via PER, Wins Produced, and other measures), APM, and scouting reports in evaluating a player.  However, the human mind – as Lehrer notes – is limited in how much information it can actually process.  So people who try and look at “everything” are not actually processing information as well as they would like.  A better approach is to systematically uncover which information is actually important and which information should be ignored. 

Again, this is just one story in Lehrer’s book.  There are of course many others.  So if you have not read this book I also recommend adding this paperback to your library.

By the way, for those interested in a discussion of how data analysis trumps the non-systematic approach taken by many decision-makers, I would recommend Super Crunchers by Ian Ayres.

More than a Game

Okay, this is a book I have not read (but hope to in the future).

Dennis Coates – the first president of the North American Association of Sports Economists – has read a recent book by Brian Billick and offered the following comment at the Sports Economist:

I am reading More than a Game, written by Brian Billick, former head coach of the Baltimore Ravens and current analyst for Fox and the NFL Network. It is an interesting book on several levels.

Two points I want to bring up here I found interesting. The first is that Billick is quite forceful in arguing that finding a quarterback is very difficult, saying that nobody knows anything. People who have been very successful at picking/finding quarterbacks have all indicated that they were high on some of the bigger quarterback busts in draft history (think Ryan Leaf). Billick mentions in passing that scouting and player evaluation uses regression models. I would love to see those equations. The implication is that evaluation of other players is more successful. I wonder if pro football evaluators feel they are doing a good job in picking wide receivers.

The second point I wanted to bring to people’s attention is that Billick mentions the research by David Romer on fourth down. He points out that after the appearance of that paper, the share of fourth downs on which the teams go for it rose each year until 2008. Economists may not get politicians to understand that subsidies are not the best use of public funds for job creation but at least one economist may have successfully convinced head football coaches to go for it a bit more often. Billick also points out that Romer’s model does not account for things like media criticism. That is an interesting perspective. Better to do the conventional, if wrong thing, to avoid media criticism, than to give your team a better chance to win the game.
Billick’s perspective is interesting and worth a read.

It appears that Billick’s book supports the argument that Rob Simmons and I have offered that drafting quarterbacks in the NFL is largely a guessing game.  This is an important point to remember as NFL decision-makers and observers evaluate the latest group of quarterbacks in the NFL draft.

A Few Non-Book Stories

Dennis Coates doesn’t just comment on books.  Here is a comment he offered on an article by Bill Simmons:

I found this interesting article about the state of the NBA from Bill Simmons at ESPN. The article is worth a read.
I especially like this bit:

They arrived at this specific point after salaries ballooned over the past 15 years — not for superstars, but for complementary players who don’t sell tickets, can’t carry a franchise, and, in a worst-case scenario, operate as a sunk cost. These players get overpaid for one reason: Most teams throw money around like drunken sailors at a strip joint. When David Stern says, “We’re losing $400 million this season,” he really means, “We stupidly kept overpaying guys who weren’t worth it, and then the economy turned, and now we’re screwed.”
This isn’t about improving the revenue split between players and owners. It’s about Andre Iguodala, Emeka Okafor, Elton Brand, Andrei Kirilenko, Tyson Chandler, Larry Hughes, Michael Redd, Corey Maggette and Luol Deng making eight figures a year but being unable to sell tickets, create local buzz or lead a team to anything better than 35 wins.

I wonder if it might be because the NBA over values scoring, as Dave, Marty Schmidt, and Stacey Brook contend in Wages of Wins and other places. And maybe some NBA executives are beginning to see that.

By the way, Dennis Coates titled this post: Dave Berri Must Love This 

I should add, though, that Okafor, Kirilenko, and Iguodala are probably worth what they are being paid.  Inefficient scorers like Allen Iverson (a Bill Simmons favorite) are generally overpaid.  That being said, I do agree that if NBA teams are losing money – and I think the word is “if” since I do not know that anyone outside the NBA has actually seen the books (and sports owners do have a history of being less than honest on this subject) – part of the reason is that teams overpay for skills that do not generate wins.  Perhaps this current labor dispute will highlight that specific point.

Let me close with two more short stories.  First, I wanted to note that Brian Burke – of Advanced NFL Statistics – is now posting his statistical analysis of every quarterback, running back, and receiver in the league.  Plus his data goes back to 2000.  So that should be enough numbers to keep any NFL fan happy during the off-season.

And finally, I wanted to point everyone’s attention to the recent work fo Darren Rovell.  As I am sure most people know, Rovell is perhaps the leading journalist on the subject of sports and business.  One can see his work on CNBC and also at his blog (Sports Biz with Darren Rovell).  When we were looking for people to review advanced copies of our next book, Darren was one of the first names to come to mind.

One of his most recent stories was on the Sports Illustrated Swimsuit issue.   Rovell tells this story in an hour-long special on CNBC.  Here is how this special is described:

CNBC’s Sports Business Reporter Darren Rovell takes an unprecedented look inside the most profitable single-issue magazine franchise in the world. Find out how business, beauty, fashion and sports come together to create this much-anticipated, multi-dimensional franchise that alone generated 7 percent of Sports Illustrated’s advertising revenue in 2009.

The Sports Illustrated Swimsuit Issue means big business not only for parent company Time Inc., but also for the models, advertisers, fashion designers and locations that grace its pages. Rovell gives viewers a behind-the-scenes look at the scouting, set-up and inner-workings of the photo shoots as he travels to one of the exquisite undisclosed locations and interviews the models vying for the ultimate prize—being featured on the cover of this year’s issue and becoming a household name

Rovell’s report is still airing, or you can just watch it on-line.

And if you are interested , here is what Rovell says about our next book:

“‘Moneyball’ should have been called ‘MoneyBaseball.‘ Stumbling on Wins covers everything else. Every general manager needs to buy this book to save his owner money. Every fan needs to buy this book to know when it makes sense to yell at the general manager.”

Stumbling on Wins is scheduled to be released in three weeks.  So it won’t be long until everyone has a chance to offer comments on our latest.

- DJ

The WoW Journal Comments Policy

Here is a surprising headline from the Detroit News:

Ex-Pistons castoff Darko Milicic has Timberwolves howling (HT: MLive Full-Court Press)

The article goes on to state the following:

The crowd chanted his name and gave him a standing ovation.

A teammate said: “You can see all of his qualities and everything he brings to the table.”

His coach said: “He obviously has tremendous potential and capabilities.”

Which NBA superstar in the making were they talking about?

Darko Milicic?

Milicic, a bust in Detroit, Memphis and New York who said in frustration he would quit the NBA after this season, has found new life with Minnesota.

He was dealt to the Timberwolves by the Knicks at the trade deadline as a throwaway. The Timberwolves, however, said they saw potential in the 7-foot-1 center.

Milicic, who spent the entire season on the Knicks bench, responded with spirited practices with the Timberwolves. He played his first game last Sunday, and won over everyone.

“It felt good to be out there,” he told the St. Paul Pioneer Press after scoring eight and grabbing eight rebounds in 19 minutes. “I didn’t expect to play that long. I got tired.”

The Timberwolves lost to the Thunder that night.

Darko played 24 minutes Tuesday, getting four points and three blocks in Minnesota’s victory at Miami.

He seems destined to play a role in the team’s final 24 games.

And don’t forget, he’s only 24.

When I saw this article I knew I had to go look at what Milicic was doing for the T-Wolves.  The results – reported in Table One – might be disappointing for those howling Minnesota fans.

Table One: The Minnesota Timberwolves after 61 games in 2009-10

Milicic has only played 102 minutes for the T-wolves.  So this is a small sample.  And across this small sample we see a WP48 of -0.104. 

Average is 0.100, so Milicic should not be generating howls.  At least, not in a positive sense.

It is true that what we are seeing is somewhat surprising.  Here is what Milicic has done across his career:

2008-09: 0.052 WP48, 1,034 minutes

2007-08: -0.045 WP48, 1,664 minutes

2006-07: 0.061 WP48, 1,913 minutes

2005-06: 0.049 WP48, 767 minutes

2004-05: -0.211 WP48, 254 minutes

2003-04: -0.171 WP48, 159 minutes

After Milicic departed Detroit, he was generally a below average player who tended to produce positive quantities of wins.  In his brief time in Minnesota, though, he is posting numbers similar to what we saw when he was with the Pistons.

So why is Milicic being cheered?  Early in his career Milicic was penalized by the high expectations that go with a player selected second in the NBA draft.  Today, though, one suspects Milicic is the beneficiary of very low expectations.  So although Milicic is not playing well, relative to what is expected, he looks like a star.

His lack of production does fit in with many of his teammates in Minnesota.  Six of his teammates are producing in the negative range.  And all but three are below average.  If we look at what these veterans did the previous season we see that most veterans were below average in 2008-09.  So what we see in Minnesota is not surprising. 

Essentially this team is Kevin Love, Al Jefferson, Damien Wilkins, and Ramon Sessions.  These players have combined to produce 20.2 wins.  Had the rest of this team produced nothing this season, the T-Wolves would be on pace to win 27 games.  Because the teams employ so many players in the negative range, though, Minnesota is only on pace to win 17 contests.

So obviously, Minnesota needs to replace the many negative players with some positive players.  And although Milicic could be one of these positive players, at 24 years of age he probably won’t be that much help.  No, the T-Wolves need quite a bit more.  And until that “quite a bit more” arrives, Minnesota is going to continue to struggle.

- DJ
The WoW Journal Comments Policy

Our research on the NBA was summarized HERE.

The Technical Notes at wagesofwins.com provides substantially more information on the published research behind Wins Produced and Win Score

Wins Produced, Win Score, and PAWSmin are also discussed in the following posts:

Simple Models of Player Performance

Wins Produced vs. Win Score

What Wins Produced Says and What It Does Not Say

Introducing PAWSmin — and a Defense of Box Score Statistics

Finally, A Guide to Evaluating Models contains useful hints on how to interpret and evaluate statistical models.