After I published my WHL goalie confidence intervals, a few people questioned the validity of a non-team specific method of evaluating goaltending. It’s a fair criticism, given that the performance of goaltenders varies more widely than in the NHL. The standard deviation of career sv% of WHL goalies active in 2013-14 was .01396; for the same in the NHL it was .009543.
Basically, there’s a theory that shot quality matters more in the WHL because the distribution of team defense is so much greater. The good defensive teams are really good, and the bad defensive teams are really bad. Because of this, the effects of shot quality (distance from the goal, breakaways, shots from the home plate area) that are so small as to be a non-issue at the NHL level come into play in the WHL. My theory is that there are just more bad goalies, and it’s harder for WHL teams than for NHL teams to replace one.
How do you test whether team defense has a role to play in boosting or lowering a goalie’s save percentage? A Ryan Miller in 2012-13 Buffalo situation is rare in the WHL. For the most part, bad goalies (however much of that can be attributed to team play) seem to congregate on bad teams.
I ended up comparing goalies who were traded during their career and comparing their save percentage splits in the context of how good their team was.
Process: I peer pressured Josh Weissbock into grabbing me data for all WHL goalies traded from 2000 to now (all the cool kids are doing it!). If a goalie had multiple seasons with one team prior to being traded, I combined them (adjusting for games played per season) to estimate an overall save percentage while he played for that team. I also found the percentage of games won for each season and combined those for an overall win%, to roughly estimate how good each team was pre- or post-trade.
I essentially looked at the change in save percentage each goalie experienced after being traded and compared it with the change in win percentage. If better teams help their goalies save a higher proportion of shots, we should see a positive relationship: as the change in win% (i.e. how good your new team is compared to your old team) increases, your change in save percentage should also increase.
The dotted line means nothing. Well, it does, but don’t let the positive slope fool you into thinking there’s a giant correlation. What matters is that R^2 value of .0313. This tells us that the data points we see are scattered so far away from the line that only about 3% of the change in save percentage is explained by the change in team points. That’s a really weak relationship!
The P-value here tells us what percentage of times the seemingly-positive relationship between win% change and save% change wouldn’t show up at all if we were to randomly sample this population 100 times. 15% of the time it wouldn’t show up at all. The threshold we use for ‘proof’ is .05 or less: that is, this would only fail 1 in 20 times. So while there’s a relationship, it’s not statistically significant to where we can say yes, a bad goalie moving to a better team is 95% likely to have a better save percentage.
There may be the barest hint of a relationship there, but it’s quite tenuous, and even if we had better stats available to evaluate team defense I doubt it would be strong enough to explain, say, a Taran Kozun. For now it doesn’t seem that goalies on terrible teams are disadvantaged to a significant extent, nor that those on good teams have an inflated save percentage.
Data via Google Drive and Dropbox