Last summer everyone “knew” that John Wall was the obvious player for the Wizards to select with the first pick in the NBA draft. Well, everyone not looking in this forum. In this forum (and at Arturo’s Amazing Stats), an issue was raised about John Wall. And here is how the issue was phrased:
Now it’s very important to emphasize what I am saying. I am not saying – and I repeat, I am not saying – that Wall will never be a great basketball player. What I am saying is that in college and summer league he was not a great basketball player (again, I am differentiating what Wall has done from what he might do in the future).
We have now moved further into the future (relative to where we were last summer). So what does Wall look like 64 games into his rookie season?
APM Tells Us John Wall (and Kobe Bryant) are Very Bad
To answer this question, let’s refer to a model that Slate.com recently told us that “many NBA stats experts use as part of their player evaluation system.” No, I am not talking about Wins Produced. Nor am I talking about Player Efficiency Rating (although Slate.com told us that teams use this method as well). What I am talking about is Adjusted Plus-Minus (APM).
According to APM numbers reported at Basketball-Value.com, John Wall hasn’t just had a difficult rookie season. John Wall – again, according to APM – is actually the least productive player employed by the Wizards. Yes, you read that correctly. Wall is the least productive player on a team that has only won 16 games this year. And let me add that when you look at the Cleveland Cavaliers and Sacramento Kings – the only two teams that are actually worse than the Wizards this season – there is not a single player on either team offering less than Wall.
When you look at the entire league, you see eight players in the NBA who have a lower APM than Wall. Apparently – according to APM – Wall has been an incredibly poor choice.
Now some people might object to this analysis. Certainly when I suggested that Wall may not be a great player, there was quite an uproar. And all I did was suggest a possibility. I never said Wall was definitely not going to be a great NBA player. The APM model, though, is clearly telling us that so far, Wall has definitely not been a very good NBA player. In fact – according to APM – Wall is one of the worst players in the NBA.
So where is the uproar? And where is the uproar when APM argues that Kobe Bryant is well below average? Or Ray Allen, Kevin Martin, or Al Jefferson? Yes, these players are also well below average this season according to APM. But as far as I can tell (and maybe I am just not looking in the right places), I don’t see too many people upset with the idea that a model NBA teams are paying money to employ arguing that John Wall and Kobe Bryant are two of the worst players in the league (and this seems odd since I have occasionally heard quite an uproar over what Wins Produced says about Allen Iverson, Carmelo Anthony, Derrick Rose, Ben Wallace, Dennis Rodman, etc… ).
The APM Evaluation Again
Of course, one can’t evaluate a model by looking at whether it conforms to our non-statistical beliefs. In fact, it is clear the NBA doesn’t take this approach. In other words, if the NBA thought their non-statistical approach was good enough they wouldn’t bother hiring statistical consultants. No, the way to evaluate APM is to consider the following (all of which was noted a few days ago):
- The estimated APM coefficients are often not statistically significant. So for most players, the correct interpretation of the results is that the player in question does not have a statistically significant impact on outcomes. In other words, it isn’t the case that Kobe Bryant is really a poor player this year. What we can actually say about Kobe from the APM results is that Kobe’s impact is not statistically significant. And this is the same story for most players.
- APM people will note that if you add more years, standard errors will fall. But we need to remember that the size of a data set and the standard errors are inversely related. In other words, for any regression, more data will lead to lower standard errors. One would also note that for APM estimates across five years, it is still the case that many players are not found to have a statistically significant impact on outcomes.
- APM results are very inconsistent over time. So a decision-maker cannot look at past values and use these for decisions about the future (of course, all decisions are about the future). This is especially true for players who switch teams. And that means, APM results for players on different teams isn’t telling you much about what that player will do for your team.
- The model itself doesn’t really appear to explain outcomes. As Arturo Galletti noted, the initial model (i.e. the model where – in the words of someone selling this model to an NBA team – “the effects of the other players on the floor are accounted for”) explains less than 5% of outcomes. This point was never noted by the APM people in the past (at least, it isn’t mentioned HERE).
Mosi Platt – of the Miami Heat Index – discussed the APM issue a few days ago and wondered by the statisticians the NBA has hired weren’t able to note the problems with this model. My sense is that when an APM person is hired by an NBA team, that person becomes the “stats department” for that team. In other words, there isn’t an independent group evaluating models for the NBA.
If this is true (and I can’t say for certain), one wonders if people in the NBA were told about the issues with APM before they purchased the product. Were they told…
- the model where we attempt to control for teammates explains less than 5% of outcomes.
- most of the results from the model are statistically insignificant, and that means APM cannot tell you if most players had any statistically significant impact on outcomes.
- the results from the model are very inconsistent across time. Specifically (see the FAQ page for this point): “…as Berri and Bradbury (2010) noted, an examination of 239 players revealed that only 7% of the variation in a player’s adjusted plus–minus value in 2008-09 was explained by what he did in 2007-08.”
If the NBA teams who purchased this model were told all this, and then still decided to pay money for the model, then that might be fine. But if this information was withheld from the buyers of the model, then there might be some problems.
We do know that APM people have been quite critical of box score methods in the past. And some (but not all), have been very critical of Wins Produced. But as is often noted, a team’s offensive efficiency and defensive efficiency (from these we derive ADJ P48 and Wins Produced) explain outcomes in the NBA (the same cannot be said for PER – a model, once again, that Slate.com says NBA teams also use). And unlike APM, ADJ P48 is quite consistent across time.
In other words, it is possible to use the box score numbers and create a model that both explains outcomes and is consistent over time. And that means the APM model appears to be a solution in search of a problem. Or to put it simply, not only is the APM model not very helpful, it isn’t clear the APM model is really necessary.
Back to John Wall
Okay, if APM isn’t necessary, how are we going to evaluate John Wall? Well, how about we just look at Wins Produced.
Here is what John Wall and the Wizards look like – in terms of Wins Produced — after 64 games:
As noted, the Wizards have only won 16 games. If we look at last year’s performances, we would have expected the Wizards to have only won about 10 games. So as bad as the Wizards have been, they are a little bit better than last year’s numbers suggest. And this is primarily because Nick Young has progressed from “awful” to “just bad” while Javale McGee has progressed from “below average” to “above average”.
McGee is progressed so far that he is now the most productive player on the Wizards. Unfortunately that isn’t saying much. McGee’s WP48 [Wins Produced per 48 minutes] is only 0.154. And if that is your best player (a point similar to something I made about five years ago), then your team is not likely to be very good.
A relatively poor “most productive” player is not the only problem afflicting the Wizards. The team also doesn’t employ very many above average players (average WP48 is 0.100). Of the players who have played more than 200 minutes, only John Wall, Rashard Lewis, and Trevor Booker have been above average. And that tell us…
- contrary to the APM story, it appears John Wall has been slightly above average (you could also say “about average”).
- in terms of WP48, Booker has done more than Wall. So Wall isn’t even the most productive rookie on the Wizards.
So let the outrage begin!! Wins Produced says Wall isn’t the most productive rookie on the Wizards.
Once again… that doesn’t mean that Wall will never be any good. It just means that so far he has been about average. If that makes you unhappy… well, at least I am not saying Wall is one of the worst players in the game.
P.S. So why don’t teams use Wins Produced? Well, I have spoken to NBA teams in the past. For the last team that called me, though, I told them to simply go buy the Wages of Wins. More specifically, I told them that an intern could calculate Wins Produced. So there was no need to hire me. Yes, apparently I need to work on my “pitch” to NBA teams J