Arturo Galletti has written an excellent column detailing the calculations behind adjusted plus-minus (APM). This column explains how this measure is calculated, and in the process, casts further doubt on the validity of this approach.  Below is a comment on Arturo’s post.  Before you read this, please read what Arturo said.  

Furthermore, you probably should also listen to our weekly podcast (with Andres Alvarez, Mosi Platt, Arturo, and I).  This podcast was devoted almost entirely to this subject. 

Okay, now that you have read Arturo’s post and listened to our podcast, here are some additional thoughts.  These additional thoughts begin with a review of what I think we already knew about APM.

Problems with Box Score Models

The APM method appears to be a response to the methods used to analyze the box score.  So our review begins with the models used to evaluate a player’s box score measures.

Perhaps the oldest model created to evaluate NBA players is NBA Efficiency.  This simple model – which adds together a player’s positive stats and subtracts the negative (without any effort to weight these) – has it roots in the TENDEX model created by Dave Heeran.  And Dave Heeran said this model goes back about 50 years.

The NBA Efficiency model – as has often been noted – is not highly correlated with team wins.  And this is because it rewards inefficient shooting.  If a player exceeds a minimum threshold for shooting efficiency (33% from two-point range and 25% from three-point range), the more a player shoots the better will be his NBA Efficiency score (a similar observation is made about John Hollinger’s Player Efficiency Rating). Since inefficient shooting doesn’t actually win games, models with this problem will have a hard time explaining outcomes.

The inability of the NBA Efficiency family of measurements to explain wins has not gone unnoticed by some people.  People have seen players with high efficiency marks (like Allen Iverson) leave a team and the team hasn’t actually gotten worse. Or join a team and not make it much better.  This has led some to question whether these statistical formulas really capture player performance.

These questions, though, didn’t just cause people to question the formulas.  What people have actually questioned is the box score numbers used to calculate these metrics.  Because basketball is a team sport, it is reasonable to think that a player’s numbers depend on his teammates.  Furthermore, there are events that happen on the court that the numbers don’t capture (like on-the-ball defense).  Consequently, those box score number –some argued — can’t be relied upon to measure player performance (of course, such an argument — as discussed before in this forum –ignores the consistency we see with respect to NBA box score numbers).

Moving to Plus-Minus

Such a story has led people to look past the box score numbers at a player’s plus-minus.  Plus-minus captures how a team does when a player is on and off the court.  The problem with plus-minus, though, is that basketball is a game of five-on-five.  So a player’s plus-minus is a function of the following factors: the player’s ability, the ability of his teammates, the ability of the teammates who take the floor when the player is on the bench, and the quality of the opponents the player is facing.  Of this list, we really just want to know about the player’s ability.  So how do we capture this one factor?

The solution people have offered is adjusted plus-minus (APM).  This measure – which several NBA teams have now apparently employed for a few years – is supposed to control for a player’s teammates and opponent.  And therefore, it is supposed to be the “best” representation of a player’s ability.  But upon further review….

Here is what we know about APM. 

As detailed in a published journal article, Stumbling on Wins, a soon to be published article in an academic collection, and the FAQ page for this forum…

The APM coefficients are often insignificant.  

For example, consider Corey Brewer.  With the Timberwolves this year, Brewer had an APM of 0.57.   So according to this number, Brewer was an above average player with the T-Wolves.   When we look at Kobe Bryant with the Lakers this year, his APM is -10.87.  And that means that Kobe is a below average player in 2010-11.  Actually that is an understatement.  A mark of -10.87 means that Kobe is just awful this year.   

Or does it?  For both players the standard error of the coefficient is so large that the correct interpretation of the result is that neither Brewer nor Bryant had a statistically significant impact on the outcomes observed for their team.  In other words, because the standard error is relatively large (a general rule of thumb is that the coefficient should be twice the value of the standard error) we cannot differentiate the coefficient from zero.  And therefore, we cannot conclude a relationship between the player and outcomes actually exists (i.e. neither Brewer nor Kobe matters for their respective teams).

People have argued that when you add more data the problems of large standard errors will be reduced.  This is true, but even when we have more years it is still the case that many of the estimated coefficients appear to be statistically insignificant (Brewer and Bryant both have insignificant coefficients when we look at two years).  Furthermore, one reason we see “improved” results with more years is that when you add more data to any model the standard errors will fall (because number of observations is part of the standard error calculation).  So that may not mean the model is any better. 

The APM coefficients are inconsistent across time

Beyond insignificance we also have a problem with inconsistent measurements across time.  Decisions are made about the future.  So we don’t want to know if a measure can just explain the past.  We need to know whether future measures are correlated with measures taken in the past.  For simple plus-minus, year-to-year correlations are quite low. 

One might think this is because plus-minus doesn’t control for teammates and opponents.  In other words, APM – which supposedly controls for teammates and opponents – would solve the problem observed with plus-minus.  But as reported in various places before, only about 7% of a player’s APM this year is explained by the player’s APM last year.  And when a player switches teams, the player’s APM this year is not statistically related to his performance the previous season.  And that means APM can’t tell you anything about what a player will do when he changes teams.  So if you change teammates –something APM is supposed to be controlling for – you don’t get the same APM.

Arturo Deconstructs the Model

The issue of insignificance and inconsistency suggest the APM model can’t be used by decision-makers.  But there is yet another issue.  Arturo Galletti has offered an extensive discussion of this model that details how it is calculated.  And this discussion reveals a few items of interest.

Quoting from Arturo’s article:

….two things jumped out (when Arturo looked at the APM model). One the correlation to wins was very low (~10% R^2) and the +/- numbers don’t quite add at the team level. Somehow they do add up in the final +/-  APM numbers.

Let’s talk about the lack of correlation.  Arturo notes he looked at this model from a variety of different angles.  And as Arturo notes…

Every single regression gave me less that 5%  R-Sq. So I feel confident in the statement that the correlation of the model in step 1 (as described) is <5%.

So the model designed to control for the quality of a player’s teammates and the opposition the player faces only explains less than 5% of outcomes.  The lack of explanatory power, though, is not something proponents of APM have gone out of their way to highlight. 

So how does one take a model that can’t explain outcomes and transform it into something that can?  Well, there are two more steps.  Again we turn to Arturo’s post:

The model now takes the True +/- values (outcome from step one) for each player from the first equation and regresses those against those player’s stats to determine weights for each stat.

This second step has a reported r-squared of 44%.  Again, that isn’t explaining outcomes very well either.

To get to a model that explains outcomes, we have a final step.  Again from Arturo…

The final step is to take the Pure regression (step one) and the Stats model (step two) and adds them up by player like so:
APM = x* Pure +/- + (1-x)*Statistical +/-

And proceed to adjust x between 10% and 90% for each player to minimize the error.

So what does that mean?

Here is how Arturo summarizes the explanatory power of the model:

…the r-squared for the APM model is very much a fabrication. The correlation to point margin & wins of the model shown in BasketballValue is artificially inflated by adding the error back in.

Summarizing the Story

So the APM model has the following three characteristics:

1. The coefficients are often not statistically significant.  So for most players, the correct interpretation of the results is that the player in question does not have a statistically significant impact on outcomes.

2. The results are very inconsistent over time.  So a decision-maker cannot look at past values and use these for decisions about the future (of course, all decisions are about the future).

3. And the model itself doesn’t really explain outcomes.  At least, it doesn’t appear to explain outcomes without that very interesting third step.

As Arturo summarizes…the APM model examined does not hold up under scrutiny. It is built to account for all the variability in the process but hold very little actual correlation to the actual process.

One should remember – as Arturo notes – that there is more than one version of the APM model.  So it is possible that other versions address these issues.  But at this point, we can’t be sure about these other approaches.  Or as Arturo put it in the comments section on his post…

The APM model as currently constructed on BasketballValue is not something I can put any credence in at this point, given what I now know about it’s construction. However, models like Wayne Winston’s are interesting as points of references. I do tend to take closed models with a huge grain of salt now. Call me Doubting Thomas.

- DJ

Editor’s Note:  Nerd Numbers – from Andres (Dre) Alvarez – is temporarily unavailable.  And yes, that means the Automated Wins Produced site is down as well.  But that hasn’t stopped Dre from writing.  The following is in response to people who think the sky is falling because Melo has forced a trade to New York.  Turns out – as this great column indicates — this has happened before. 

By the way…before you get to reading Dre’s latest, let me also give a shout out to Mosi Platt.  On Wednesday I wrote a quick post on Mike Bibby’s move to Miami.  Mosi has written something that is much better: Did Losing to the Knicks Push Arroyo Out and Bring Bibby In?  So click on over to Mosi’s work and then come back here (or you can read Dre’s work first and then check out Mosi’s post; either order will work!)

The Story

The labor talks are coming and Melo’s decision to force himself onto New York has added alarm. It appears that the new thing to do in the NBA is for the top players to decide to play together and form a few super teams. This means only large market teams will have the good players and the league will fall into disrepair.

The NBA is a very different league than the other major sports in the US. In the NFL, where I believe there are 47.3 players on the field with 24.5 different positions (sarcasm, but isn’t football complicated?) it’s rough for one player to have complete control of a team. In the NHL even the league’s iron men only play about 1/3 of the game. In MLB players are forced to be only 1/9 of the offense (even less with the DH aka the dumbest rule ever) and are only responsible for a small part of the field on defense. In the NBA though a player can play almost the entire game and be the key player on both offense and defense. As I’ve noted only 46 players in the last 33 years were responsible for 80% of the regular season success of their playoff teams.

The scarcity of these top players does allow for super teams ruling the league a very scary possibility. Did I say possibility? I meant reality. In the last 30 years only 9 franchises have won a title. With Miami, Los Angeles (the good one), San Antonio, Boston and Chicago all in the hunt it doesn’t look like this year will change that.  The idea that 2011 is the year where suddenly the competitive balance in the NBA vanishes is a myth. It was never there to begin with.

Perhaps the real issue though is that recently the players have been calling the shots. If LeBron decides to hop ship to Miami and Melo decides to go to New York then what’s to stop your team’s best player from just deciding they want to play wherever they like? (For the record when I phrase it like that it makes me feel very awkward as a fan.)

The truth is that in the last 30 years almost all of the championship teams in the NBA managed to win by grabbing a star player from another team. Another scary fact is that much of the time this was driven by the player’s desire to play in a better market — or for a contender — and not by savvy front office movers. Let’s run down the list shall we?

The 1981,1984,1985,1987 and 1988 Los Angeles Lakers

Calling the shots: The Players (1)

The Player: Kareem Abdul Jabbar (1976)

Kareem Abdul Jabbar decided he would rather play in a big market than for Milwaukee. Can you blame him? A few helpful draft-picks in Magic Johnson and later James Worthy helped the Lakers to multiple titles. The first step though was grabbing one of the top players in the league from a small market team. This was of course driven by Kareem’s decision to move. Even back in the 1970s players were calling the shots!

The 1982, 1984 and 1986 Boston Celtics

Calling the shots: The Management (1)

Robert Parish was perishing on a terrible Golden State team. It turned out Boston already had a big name on their roster and just needed another big name or two. Golden State gave up the Chief (and a draft pick that turned into Kevin McHale but who’s counting?) and let Boston turn into a power house. I’m sure it will make Golden State much happier to know it wasn’t a whiny player forcing his way onto a better team but terrible management that let this happen.

The 1983 Philladelphia Seventy-Sixers

Calling the shots: The Players (2)

The Player: Moses Malone (1983)

Your team has recently made the finals. You have the league MVP. You get to feel the pain as they leave to a juggernaut team and all you get is some chump change in a fake trade. I’m not talking about LeBron James. I’m talking about Moses Malone almost 30 years ago. Heat fans will hope this plays out the same as it did for the Sixers.

 The 1989 and 1990 Detroit Pistons

Calling the shots: The Management (2)

The Player: Bill Laimbeer (1982)

Laimbeer had to wait until the Pistons drafted Dennis Rodman to have another solid player to propel his team to greatness. Cleveland let another big name go for not much in return. Luckily (or unfortunately depending on your perspective) Laimbeer was traded before he ever had a chance to be a truly great player in Cleveland.

The 1994 Houston Rockets

Calling the shots: The Management (3)

The Player: Otis Thorpe (1989)

Otis Thorpe was immediately paired with Hakeem. They had to lay low until the Pistons got old and the Bulls weakened with Jordan deciding to play baseball. It’s hard to think of this move as fleecing another team as Rodney McCray was part of the trade and he played well for Sacramento. Moving on.

The 1995 Houston Rockets

Calling the shots: The Players (3)

The Player: Clyde Drexler (1995)

What do you do if you were on a great team that missed its shot? Well, just ask your management to trade you to the defending champs. Clyde managed to finally get a ring on his finger and was a great pickup for the Rockets. Oddly this was at the cost of Otis Thorpe.

The 1996, 1997 and 1998 Chicago Bulls

Calling the shots: The Management (4)

The Player: Dennis Rodman (1996)

The Bulls lost some time and talent when Jordan took a break. They shipped Horace Grant and needed some “size”. Management took a chance on Dennis Rodman and it paid off in huge ways. Rodman, Pippen and Jordan were three of the league’s top players and they dominated the league for three years. I don’t remember this being a world ending event at the time.

The 2000, 2001 and 2002 Los Angeles Lakers

Calling the shots: The Players (4)

The Players: Shaquille O’Neal (1996)

Orlando did everything it could to keep Shaq including giving him a contract offer that let him leverage a sweet deal from Los Angeles. Like Kareem before him Shaq wanted to go to a big market where he could work on his side projects like his great rapping and acting careers. Orlando couldn’t help but feel betrayed by the fact Shaq decided to play for another team, even though the NBA rules made Shaq play in Orlando at far below market value for four years.

The 2004 Detroit Pistons

Calling the shots: Management (5)

The Players: Ben Wallace (2001)

Ben Wallace helped bring the Pistons back to their glory days. People say this team didn’t have a star. Yes it did, it was Ben Wallace and Orlando foolishly gave him away thinking he was just a consolation prize for Grant Hill (funny how that worked out). People can praise Joe Dumars for this masterful stroke or just thank his dumb luck, take your pick.

The 2006 Miami Heat

Calling the shots: The Players (5)

The Players: Shaquille O’Neal (2005)

Shaq wanted a contract and a title. He was able to force his way into both when Los Angeles didn’t think he was worth the money. Miami benefited from this trade by getting a title. They then managed to ship Shaq’s large contract and as a byproduct get cap space to let LeBron make a similar call. If the players are calling the shots in Miami, well all I can say is that I am jealous of Miami.

The 2008 Boston Celtics

Calling the shots: The Players (6)

The Players: Kevin Garnett (2008)

I will join your team if you give me a lucrative long term deal, star teammates and a title shot! Yup KG pulled the same trick as LeBron and got immediate results. If he’d decided to do this a few years sooner it’s possible Boston could have had even greater success than two finals appearances and a title in three years.

The 2009 and 2010 Los Angeles Lakers

Calling the shots: The Management (6)

The Players: Pau Gasol (2008)

Here’s an interesting note. The Celtics in 2008 were turned into a contender because the players called the shots. In an almost odd move the front offices pulled off a trade that made a super team to counter them by sending Pau to Los Angeles. Long term this deal wasn’t a bad deal for Memphis, but if owners are worried about creating super teams it is curious the defending champs were constructed thanks to the owners making a super team.

Summing Up

The 1991-1993 Bulls and the 1999,2003,2005,2007 San Antonio Spurs were the only title teams I could find where they won by drafting top players. Twenty three out of the last thirty titles have been won thanks to either players or management moving a star player to their team. The key is that the players have done this just as often as the management. There isn’t some great conspiracy ruining the league. There is a shortage of good players (read Wages of Wins for more info) and the good teams are those that manage to get them. I don’t think Miami will tear the league up because LeBron decided to play there. I don’t think New York has a real shot at the title. What I do think is that good players will play where they want, savvy managers will find a way to get them, and fans will watch them play. That is unless there’s a lockout.

-Dre

Mike Bibby’s career in Washington is only going to last two games.  And now he is headed to the Miami Heat. Judging by the comments from the Heat, the addition of Bibby is being met with a fair amount of optimism.

“…the Heat’s players were already in a welcoming mood for Bibby, who officially clears waivers at 6 p.m. ET Wednesday. Heat forward LeBron James said he was on the verge of becoming teammates with Bibby in Cleveland several years ago before the veteran guard was traded from Sacramento to Atlanta.

This time James won’t miss out on sharing a perimeter role with Bibby.

“A few years ago, we tried — we had an opportunity to get him in Cleveland,” James said after Tuesday’s 2½-hour team film study and practice session. “It didn’t work out, when he was getting traded away from Sacramento and went to Atlanta. So I’ve had some conversations with him and said, ‘It would be good to have you as our point guard.’ It’s good that it’s come full circle.”

“I think it’s a big thing for us,” said Heat guard Eddie House, Bibby’s brother-in-law. “He’s going to do a lot of things that can help LeBron, Dwyane and everyone else around here.”

Is Bibby, though, really going to help that much?

To answer this question, let’s look at what Bibby did for the Atlanta Hawks this season.

After 60 games the Atlanta Hawks have won 36 games.  The team, though, has only out-scored their opponents by 54 points this season.  So the team’s efficiency differential is only 0.97; and that means the team’s Wins Produced is only 31.6.

When we look at the Hawks roster – reported in the table below – we can see that Al Horford, Josh Smith, Joe Johnson, and Marvin Williams have produced 28.8 of these wins.  As for Bibby, his WP48 [Wins Produced per 48 minutes] in Atlanta was only 0.078 (average WP48 is 0.100).  And his Wins Produced was only 2.7, which tells us that Bibby didn’t play a big role in Atlanta’s success this season.

Bibby not helping the Hawks much, though, isn’t the question.  Can Bibby help the Heat?  To answer this question let’s first compare Bibby to the two point guards the Heat have employed this year.  The following table compares the career numbers– prior to this season — of Bibby, Mario Chalmers and Carlos Arroyo (Arroyo was cut to make this move happen).

Prior to the 2010-11 season, Bibby had played for 11 years and produced 66.5 wins.  When we consider WP48 we see a career mark of 0.109, which is slightly above average.  But again, he has been slightly below average this year.  And the same is true in four of the past five seasons.  Of course Bibby is now 31 years old.  So returning to the above average player seen in the past seems unlikely (although possible).

Turning to the players Bibby is replacing… both Chalmers and Arroyo has also posted career WP48 marks that are somewhat below average.  And when we turn to this season – detailed below – we once again see somewhat below average marks.

This season Chalmers has posted a 0.078 mark.  With this move, one would assume Chalmers goes to the bench and the Heat will add Bibby – and his 0.078 WP48 – to the starting line-up. So Bibby doesn’t appear to be a big upgrade over Chalmers.

Meanwhile, Chalmers takes on the role of Arroyo, who only posted a 0.021 WP48 mark this year.  Yes, Chalmers has done more than Arroyo.  But across the 995 minutes Arroyo has played this year, moving from a 0.021 WP48 to a 0.078 mark would only add about 1.2 wins for the Heat.

To make this move happen, Bibby accepted a buyout that will cost him $6.2 million.  He did this so that he can win a title.  And that may actually happen.  But I don’t think – given the numbers posted by Bibby, Chalmers, and Arroyo – that Bibby is really going to improve the Heat.  So if the Heat do win a title, this move is not going to be the reason why that happens.

Let me close by asking a related question:  How much money have the Heat players given up to win a title?  The Super Friends have all come with a discount.  And I think Mike Miller took less to joint the Heat.  And now Bibby is doing the same. One would think that people who want athletes to focus on wins — and not money – would love this team.  But I sense that is not the case.  LeBron and company have actually just generated a great deal of hate.  Which leads me to ask… what do these sports fans want?

- DJ

This is the question my latest for The Huffington Post addresses.  Short answer… the attitude and abilities of the Pistons are not in alignment.  And this disconnect has led to anger and frustration (which is being directed at the coach).

In addition to a link to my work for The Huffington Post, let me direct everyone’s attention to the following stories:

A Few More Links

• Ty Willihnganz:  Best and Worst Scorers in the NBA East (apparently scoring isn’t as hard as it appears)
• Ty Willihnganz:  Contested vs. Free Rebounds in the Bucks-Bulls game (apparently many rebounds aren’t free)
• Arturo Galletti: The Rebuild & the Rankings (among many other stories, Arturo apologizes to Darryl Morey and the Rockets).
• Henry Abbott: The Knicks could use a good geek (short comment below)

Quick Comment on the Value of Corey Brewer and Kobe Bryant

Henry’s story argues that Corey Brewer may be capable of making a positive contribution to a team’s success.  The post sites statistics like adjusted plus-minus.  Brewer posted a 1 year adjusted plus-minus (APM) of 0.85 with the Minnesota Timberwolves this year.  And his defensive measure was in the negative range (which is “good”).  Then again, by the same measure Kobe Bryant has a measure of -11.15 (which has to be one of the lowest APM marks in the NBA this season).  And Kobe’s defensive measure is very far in the positive range (which is “bad”). 

So is Brewer “better” than Kobe this year?  Well, APM is very inconsistent from season—to-season.   So like the weather in the Midwest (and this is something often said in places like Nebraska), if you don’t like a person’s APM just give it some time and it will probably change. 

Of course, that is a problem for decision-makers.  How are they supposed to know if a result seen from APM reflects a player’s ability or is simply the “noise” in the model?  If you can’t answer this question (and I don’t think you can), I don’t think you can use this information in making decisions.

Just for the record… according to the automated Wins Produced website (from Dre Alvarez), Brewer has a WP48 of -0.022 and an Adjusted P48 of 0.120.  Kobe’s marks are 0.207 and 0.344.  Such marks are quite consistent with what we have seen from Kobe and Brewer across each player’s respective careers (ADJ P48 has a 0.85 correlation from season-to-season; so it is quite consistent).  In other words, the box score says Kobe is much better than Brewer.  And this has always been true.

Now maybe Brewer’s defense is just amazing.  And maybe his defensive ability puts him on par with Kobe this season.  But I suspect that isn’t true.  So maybe the Knicks were wise to let Brewer find employment elsewhere.

By the way…. a statistical measure doesn’t have to be consistent with conventional wisdom.  In fact, whether a measure is consistent with conventional wisdom or not is not relevant to whether the measure is “good” (at least, that has always been my argument).  The problem with APM is not that it is defies conventional wisdom (like Wins Produced, this is often the case). The problem is that it is quite inconsistent over time.   In addition, the results are simply hard to explain (i.e. can Brewer be such a great defender that he simply overcomes all of his remaining shortcomings?).  Such problems suggest that APM probably doesn’t help people make better decisions.

- DJ

P.S. One last note… Henry doesn’t think the Knicks should have used the APM results to decide whether to keep Brewer.  He thinks they should use it to go look at more film. But if APM tells you a player is “good”, and you then go look at film, what are the odds you will walk out of the film session thinking the player is “good”?