Confidence Intervals for WHL Goaltenders
by Megan Richardson
I first saw the use of confidence intervals for goalie save percentage last year after the Roberto Luongo trade that left Eddie Lack in the starting role in Vancouver. It may have been Eric Tulsky who tweeted out a cautionary 95% confidence interval that (I don’t remember the actual statistics, so humor me with my guesstimates) 20 games at a .920 save percentage left room for Lack to develop into somewhere between a .900 save percentage goalie or a .940, or something of that kind.
The thing is, most of the time, goalies are voodoo. For a while, I’ve kind of ignored goalies because they’re tough to predict in comparison with individual player possession statistics. Even with a full season of data, it’s hard to know what they’ll put up the next year. Look at this year’s Vezina winner in Semyon Varlamov. Here are his last five seasons’ save percentages in reverse order: .927, .903, .913, .924, .909. If there’s this much variation at the season level, what the heck does that say about how individual games go, or even playoff series?
Why does this happen? To put it in a blunt and unsavory manner: there’s a lot of luck in hockey. I know it screws up our well-crafted sports narrative, but truthfully there are lucky bounces, deflections, and all kinds of random chance involved.
Basically, the message is: you know less than you think you do about how good a goaltender is. A 95% confidence interval tells you that 19 of 20 times, a goaltender’s career save percentage will fall between the low and high value. It’s the statistics equivalent of “I am basically sure this goaltender is this good.” I may not stake my life on it, but I would bet $20 confidently.
(Side note: in this spreadsheet I throw GAA out the window. When you want to talk about how good goalies are, you just can’t use GAA. Think about how it’s calculated: it’s a goalie’s save percentage multiplied by the number of shots he faces per game played. A .920 goalie who sees 20 shots per game on a good team will have a far lower GAA than a .920 goalie on a poor team who sees 40 per game. Half of this statistic is controlled by the goalie, but half is set by his teammates–and when you’re trying to isolate goalie performance, that’s really not fair. Here’s a post that explains it better than I can.)
Here’s the 95% confidence interval for every goalie who saw a shot in a 2013-14 regular season WHL game. CI Low is the lowest career save percentage we could expect, and CI High is the goalie’s ceiling:
Look at the pattern below: when a goaltender hasn’t seen many shots, he has a huge confidence interval. Each shot contributes to a smaller confidence interval, but it contributes a little less. The difference between 60 and 61 shots against is bigger than the difference between 1000 and 1001. So we really need to see young goalies before we can tell anything about how good they will be. I added a new variable on the right that’s the spread of how
You probably want to know who are the (objectively) best and worst goalies in the WHL. The goalies who are ‘proven’ to be best would have the highest save percentage for the low end of their confidence interval, i.e. their floor is very high. Here they are:
I arbitrarily picked .895 as the lowest value that would still qualify a good goalie. Sue me. Of particular interest for me are goalies who still have a high confidence interval difference–that is, more data will narrow their floor and ceiling values, so they could still pull up their low value a bit more.
And here are the ‘worst’ goalies, the ones with a low ceiling (below .900):
There’s a bit of sample size fuzziness going on here with all the goalies who played ~20 games. I’d ignore those for the moment and focus on Rathjen, Lee, Cotton, Moodie, and Sacher. For perspective, the league-wide save percentage last year was .903.
Lastly, I thought it would be interesting to compare last year’s save percentages with career numbers. Take this with a grain of salt, because I’m not pulling out 2013-14 numbers to compare with career numbers prior to the 13-14 season. There’s a bit of feedback going on.
Here are goalies (at least 10 games last season) who might be in for a rebound, as their 2013-14 numbers were at least .005 below their career average.
And here are goalies (1o+ games)who had a career year and might experience a drop in save percentage this year:
*A last note on Taran Kozun: I noticed that he had a .897 sv% with Kamloops, and then .928 with Seattle after he was traded. In terms of the whole season, his numbers were in the good/elite category. However, if you’re a Seattle fan and expect a .928 sv% this year, I doubt it. Team defense doesn’t affect sv% to remotely that extent. More likely he had a bit of a slump in Kamloops and played well in Seattle–as all goalies do, going through little slumps and little streaks that even out over the course of a season or career.
Update 9/10/15: In the course of researching for some other work, I came across this 2013 post by Eric Tulsky that aggregated work by Brian MacDonald. I feel confident I have not read it before and did not incorporate it into my work. I view this piece as approaching the same idea from a different direction. From that standpoint, I highly encourage anyone who comes across this work to read the linked post, as it includes a visual representation of what I have tried to communicate here, as well as a handly table of heuristic approximations of the same intervals.