Hey EA, eventhough it's like 2 months late, can you make that spreadsheet avail? It'd be fun to punch in current RAPTOR ratings below, and do season long projections, then do adjustments for playing time.
That was great to get a peek behind the stats curtain that I'm too lazy to hunt down. I would be interested in another compare/contrast at the half way mark of the season.
It's worth remembering Raptor is primarily a player rating and projecting tool. They do bundle player ratings together to produce team forecasts, but AIUI that isn't the primary purpose or focus, and that's why the team component is just a simplistic summing of the individual ratings, as others have noted I tend to evaluate it with that lens more than the team prediction one, personally, although I can understand not doing so.
For those who are curious, the current team raptor stats to compare to the ones in the video are:
RAPTOR is having a love affair with GPII and also thinks Steph's (admittedly improved) defense is significantly better than Draymond's but not in the same stratosphere as Andre.
So, like your average sportsfan, it has some good opinions and some really dumb ones
My own before-season predictions are looking pretty good right now (a coworker even commented on how spookily accurate they were). I only predicted the Western conference (like hell I know what's going on out east). I used 538's predictions as one source to inform my own ideas, so I can't discredit them too hard, but I was a lot closer than their model was at the time.
Even so, some things surprised me.
- The Warriors came out strong from the beginning. Bjelica and Porter surpassed my expectations, to say nothing of low-minutes-MVP candidate GP2 (I was all about him in Summer League, but I thought he'd be way more marginal in the regular season, lulz.) Plus the defense is really organized! I put us at 3rd overall assuming the new guys would take a while to fit in and we'd need Klay to put us over the top.
- The Lakers are even more dysfunctional than I thought. I had them at 6th.
- The Thunder have stolen more games than expected. At least 1/3 of them are the Lakers' fault.
A day or two I had basically every other team ranked in the exact order they were in the standings (I think the Mavs were up a spot or two compared to my guess?) but since then there's been more movement in the standings, like Denver dropping from 5th to 8th because Jokic is out. Just goes to show, it's not only what you predict but when you follow up that matters.
Wow this was super fun, and really shows how many moving parts go into the win prediction — Ie the errors have to really add up!
Also makes me wonder how the numbers are obtained for the Pythagorean win ratio. Technically it also accounts for pace (implicitly via total points scored/allowed), so it’s more informative than just the point differential.
Haha, how is this not a travel on PG? And what's the rule on the refs reviewing this? I imagine they went back to replay to check if he got the shot off on time, but are they not allowed to rule it a travel after the fact? It feels like this is exactly what replay would be good for...
Unless something like a travel is called, it can't be reviewed or overturned even if they are reviewing the play for something else. Which is BS - but it's the way it is.
Thanks Eric! This is a great breakdown. Super interesting. I really enjoyed how you explained the limitations of the model and how realistic adjustments can make significant impacts to final results. Very cool
Whoaaa... I realllly liked this video. This is a much more practical application of stats and modeling that I think a lot of people miss out on. I doubt anyone at 538 thinks their model is a crystal ball to predict the future... it's more like a gathering a measurements and a rough estimate.
A good analogy would be like hiring a contractor to replace your fence... you can get a rough ballpark figure of the cost using price-per-linear foot and type of materials for the fence, but until the guy shows up the property and can assess all the special cases, you can only take that original estimate as a ballpark figure.
Nov 24, 2021Liked by Eric Apricot, punk basketball, Daniel Hardee
When a data-driven, numbers-oriented, scientific website is so wrong so often, it's time to ask what's up. I think I looked up at one point how off 538 has been in previous years and it's something like ten games per team per year, roughly. Which is a large margin for error. And I think you mentioned possibly the most important problem with their model. It's the idea that if you add up some simple value for each player on a team, you get the total wins. But that's not how the game works. It's actually the 2021 Laker Theory of Value: take some players known to have racked up numbers and put them together, and you'll get the sum of those numbers. Well... so far that looks like a mess. Because if you add Westbrook's assists to Carmelo's shooting, you might get something good; or you might get Westbrook passing it to where he isn't -- and neither of them play defense. And similarly, the Warriors sum is so much greater than its parts. Passing, spacing, gravity, screens, communication on defense, etc.: these are all team behaviors that increase in value depending on the fit and familiarity. If you put Draymond on certain teams you may not get that much offensive advantage, which is what the 538 model in fact indicates. But on the Warriors he's a crucial point forward. He makes Steph alone a good deal better, as well as everyone else. The recent Raptor game is another piece of evidence against the 538 model. In just about every measurement, Steph had a mediocre game. But if you watched the game or even better the Explain One Play, Steph by his gravity, movement and screens got Wiggins and Poole great open looks. If every game were like that, the model would have a much higher number for Poole than for Steph, and we would all laugh haha.
If I were going to Vegas and had to bet on basketball games to save my life and was given a choice between basing bets on the 538 model or on the wisdom of the DNHQ chat, I'd go with the latter, easy. Human eye test has its flaws, but so do many metrics.
PS: A lot of us including management liked GPII over Avery Bradley for the 15th spot even though over the years his NBA box stats haven't been better, so that's another win for the eye test.
I don't disagree with your post but this is actually not accurate. When the 538 predictions came out this year, I went year by year since 2016 to see how bad they actually were... historically it's more like 5-7 games off on average.
Season / Avg. Win Diff / Stdev. Win Diff / Avg. Win Diff without Outliers*
'16-17 / 5.07 / 3.85 / 5.07
'17-18 / 6.00 / 4.33 / 6.00
'18-19 / 6.93 / 4.86 / 6.93
'19-20 / 7.30 / 7.31 / 6.38 <-- RAPTOR introduced
'20-21 / 6.60 / 6.06 / 5.93
When you account for outliers, the predictions got progressively worse until '19-20.
*I recalculated the average win diff by removing samples that were off by 3 standard deviations from mean (eg. '19-20 GSW)
I mean any idiot who was predicting us to win 37 or 39 games based on "stats" or "analytics" was being deliberately obtuse for multiple reasons:
(a) We have GOAT or co-GOAT offensive player Steph Curry.
(b) We have one of the GOAT defensive players Draymond Green.
(c) Even last year, we won 39 games in a 72 game season. This was with Steph missing 9 games. We were 37-26 with him in the lineup.
(d) When we played a more optimal style to finish the year, we ended the season 15-5.
(e) Most importantly, in non-Wiseman, non-Oubre Steph minutes last year, we had the metrics of a high 50s low 60s win team. How anyone who is taking an "analytical" look into the team chooses to ignore that is beyond me.
A lot of my work comprises of building models. If one of my models was giving a result this dumb, I'd take a long hard look into the assumptions the model is making.
You're absolutely right. 538 seems almost deliberately obtuse in ignoring available metrics. Like ... uh, win-loss record previous season? It's absurd.
They're excellent for political polling. But their basketball forecast model has been broken for years. I cant understand why they don't pull it back for a few years and get it readjusted. They just keep diving deeper and deeper into tweaks instead of fixing the fundamental errors
Their political polling is also based on a great deal more data and understanding. Hearing Silver explain some of the nuances shows he understands political polling about as well as anyone -- and that's how he rose to fame. Their basketball model? Not even close to the same dedicated thoughtfulness and expertise.
As it turns out you were probably pessimistic though we won't know for sure until Thompson and Wiseman show up. In the meantime, specifically in reference to Wiseman, that means that other players are getting those minutes at center and they are all better than what you would expect Wiseman to be. In addition, the end of the bench, GPII and Chiozza have been far better so far than what anybody could have reasonably expected.Between the 2 of them they fill the vacuum created by Livingston's retirement. Also there ought to be some sort of fudge factor for having the greatest shoot of all time healthy and on your roster. We know that makes a huge difference that doesn't show up if his team<cough>Oubre</cough>mates aren't capable of capitalizing on it.
I love how this video was so carefully reasonable, sensible, and realistic, and then the team just blew the doors off of it.
Hey EA, eventhough it's like 2 months late, can you make that spreadsheet avail? It'd be fun to punch in current RAPTOR ratings below, and do season long projections, then do adjustments for playing time.
That was great to get a peek behind the stats curtain that I'm too lazy to hunt down. I would be interested in another compare/contrast at the half way mark of the season.
It's worth remembering Raptor is primarily a player rating and projecting tool. They do bundle player ratings together to produce team forecasts, but AIUI that isn't the primary purpose or focus, and that's why the team component is just a simplistic summing of the individual ratings, as others have noted I tend to evaluate it with that lens more than the team prediction one, personally, although I can understand not doing so.
For those who are curious, the current team raptor stats to compare to the ones in the video are:
Team Raptor 11/24/2021
Player Min Off Def
~~~~~~~~~~~~~~~~~
Steph: 542 +9.2 +2.0
GP2: 225 +4.8 +9.3
Dre: 250 +0.6 +6.9
Bjeli: 257 +0.9 +3.6
Dray: 477 -0.8 +1.5
OPJ: 260 +1.6 +1.8
Loon: 311 0.0 +1.6
Wiggs: 500 -0.1 -0.5
Lee: 301 -0.6 +1.2
Poole: 500 +0.4 -2.4
JKu: 94 -4.3 +2.7
Chioz: 66 -3.8 +0.8
Dowtin: 6 -8.4 -5.4
JTA: 255 -1.7 -2.2
Moody: 85 -6.3 -8.4
RAPTOR is having a love affair with GPII and also thinks Steph's (admittedly improved) defense is significantly better than Draymond's but not in the same stratosphere as Andre.
So, like your average sportsfan, it has some good opinions and some really dumb ones
My own before-season predictions are looking pretty good right now (a coworker even commented on how spookily accurate they were). I only predicted the Western conference (like hell I know what's going on out east). I used 538's predictions as one source to inform my own ideas, so I can't discredit them too hard, but I was a lot closer than their model was at the time.
Even so, some things surprised me.
- The Warriors came out strong from the beginning. Bjelica and Porter surpassed my expectations, to say nothing of low-minutes-MVP candidate GP2 (I was all about him in Summer League, but I thought he'd be way more marginal in the regular season, lulz.) Plus the defense is really organized! I put us at 3rd overall assuming the new guys would take a while to fit in and we'd need Klay to put us over the top.
- The Lakers are even more dysfunctional than I thought. I had them at 6th.
- The Thunder have stolen more games than expected. At least 1/3 of them are the Lakers' fault.
A day or two I had basically every other team ranked in the exact order they were in the standings (I think the Mavs were up a spot or two compared to my guess?) but since then there's been more movement in the standings, like Denver dropping from 5th to 8th because Jokic is out. Just goes to show, it's not only what you predict but when you follow up that matters.
Wow this was super fun, and really shows how many moving parts go into the win prediction — Ie the errors have to really add up!
Also makes me wonder how the numbers are obtained for the Pythagorean win ratio. Technically it also accounts for pace (implicitly via total points scored/allowed), so it’s more informative than just the point differential.
OT: https://youtu.be/o9Lx1JqaDLg?t=115
Haha, how is this not a travel on PG? And what's the rule on the refs reviewing this? I imagine they went back to replay to check if he got the shot off on time, but are they not allowed to rule it a travel after the fact? It feels like this is exactly what replay would be good for...
Last 2 minute report will be interesting… thankfully Dallas win in the end, so it didn’t effect the outcome. But that’s like 5 steps
Unless something like a travel is called, it can't be reviewed or overturned even if they are reviewing the play for something else. Which is BS - but it's the way it is.
Thanks Eric! This is a great breakdown. Super interesting. I really enjoyed how you explained the limitations of the model and how realistic adjustments can make significant impacts to final results. Very cool
Whoaaa... I realllly liked this video. This is a much more practical application of stats and modeling that I think a lot of people miss out on. I doubt anyone at 538 thinks their model is a crystal ball to predict the future... it's more like a gathering a measurements and a rough estimate.
A good analogy would be like hiring a contractor to replace your fence... you can get a rough ballpark figure of the cost using price-per-linear foot and type of materials for the fence, but until the guy shows up the property and can assess all the special cases, you can only take that original estimate as a ballpark figure.
When a data-driven, numbers-oriented, scientific website is so wrong so often, it's time to ask what's up. I think I looked up at one point how off 538 has been in previous years and it's something like ten games per team per year, roughly. Which is a large margin for error. And I think you mentioned possibly the most important problem with their model. It's the idea that if you add up some simple value for each player on a team, you get the total wins. But that's not how the game works. It's actually the 2021 Laker Theory of Value: take some players known to have racked up numbers and put them together, and you'll get the sum of those numbers. Well... so far that looks like a mess. Because if you add Westbrook's assists to Carmelo's shooting, you might get something good; or you might get Westbrook passing it to where he isn't -- and neither of them play defense. And similarly, the Warriors sum is so much greater than its parts. Passing, spacing, gravity, screens, communication on defense, etc.: these are all team behaviors that increase in value depending on the fit and familiarity. If you put Draymond on certain teams you may not get that much offensive advantage, which is what the 538 model in fact indicates. But on the Warriors he's a crucial point forward. He makes Steph alone a good deal better, as well as everyone else. The recent Raptor game is another piece of evidence against the 538 model. In just about every measurement, Steph had a mediocre game. But if you watched the game or even better the Explain One Play, Steph by his gravity, movement and screens got Wiggins and Poole great open looks. If every game were like that, the model would have a much higher number for Poole than for Steph, and we would all laugh haha.
If I were going to Vegas and had to bet on basketball games to save my life and was given a choice between basing bets on the 538 model or on the wisdom of the DNHQ chat, I'd go with the latter, easy. Human eye test has its flaws, but so do many metrics.
PS: A lot of us including management liked GPII over Avery Bradley for the 15th spot even though over the years his NBA box stats haven't been better, so that's another win for the eye test.
> ten games per team per year, roughly.
I don't disagree with your post but this is actually not accurate. When the 538 predictions came out this year, I went year by year since 2016 to see how bad they actually were... historically it's more like 5-7 games off on average.
Season / Avg. Win Diff / Stdev. Win Diff / Avg. Win Diff without Outliers*
'16-17 / 5.07 / 3.85 / 5.07
'17-18 / 6.00 / 4.33 / 6.00
'18-19 / 6.93 / 4.86 / 6.93
'19-20 / 7.30 / 7.31 / 6.38 <-- RAPTOR introduced
'20-21 / 6.60 / 6.06 / 5.93
When you account for outliers, the predictions got progressively worse until '19-20.
*I recalculated the average win diff by removing samples that were off by 3 standard deviations from mean (eg. '19-20 GSW)
Absolutely accept your numbers, well researched. Mine was a more cursory run-through.
I mean any idiot who was predicting us to win 37 or 39 games based on "stats" or "analytics" was being deliberately obtuse for multiple reasons:
(a) We have GOAT or co-GOAT offensive player Steph Curry.
(b) We have one of the GOAT defensive players Draymond Green.
(c) Even last year, we won 39 games in a 72 game season. This was with Steph missing 9 games. We were 37-26 with him in the lineup.
(d) When we played a more optimal style to finish the year, we ended the season 15-5.
(e) Most importantly, in non-Wiseman, non-Oubre Steph minutes last year, we had the metrics of a high 50s low 60s win team. How anyone who is taking an "analytical" look into the team chooses to ignore that is beyond me.
A lot of my work comprises of building models. If one of my models was giving a result this dumb, I'd take a long hard look into the assumptions the model is making.
You're absolutely right. 538 seems almost deliberately obtuse in ignoring available metrics. Like ... uh, win-loss record previous season? It's absurd.
They're excellent for political polling. But their basketball forecast model has been broken for years. I cant understand why they don't pull it back for a few years and get it readjusted. They just keep diving deeper and deeper into tweaks instead of fixing the fundamental errors
Political polling is way more nonsensical than sports analytics imo
Their political polling is also based on a great deal more data and understanding. Hearing Silver explain some of the nuances shows he understands political polling about as well as anyone -- and that's how he rose to fame. Their basketball model? Not even close to the same dedicated thoughtfulness and expertise.
As it turns out you were probably pessimistic though we won't know for sure until Thompson and Wiseman show up. In the meantime, specifically in reference to Wiseman, that means that other players are getting those minutes at center and they are all better than what you would expect Wiseman to be. In addition, the end of the bench, GPII and Chiozza have been far better so far than what anybody could have reasonably expected.Between the 2 of them they fill the vacuum created by Livingston's retirement. Also there ought to be some sort of fudge factor for having the greatest shoot of all time healthy and on your roster. We know that makes a huge difference that doesn't show up if his team<cough>Oubre</cough>mates aren't capable of capitalizing on it.
I am definitely a pessimist. I just don’t tend to woe-is-me in public
I feel personally attacked by this
I know I'm not supposed to feed the trolls on this site, but what about triggering the snowflakes?