shinnystats

meandering through the world of hockey statistics.

Confidence Intervals for WHL Goaltenders

I first saw the use of confidence intervals for goalie save percentage last year after the Roberto Luongo trade that left Eddie Lack in the starting role in Vancouver. It may have been Eric Tulsky who tweeted out a cautionary 95% confidence interval that (I don’t remember the actual statistics, so humor me with my guesstimates) 20 games at a .920 save percentage left room for Lack to develop into somewher between a .880 save percentage goalie or a .940, or something of that kind.

The thing is, most of the time, goalies are voodoo. For a while, I’ve kind of ignored goalies because they’re tough to predict in comparison with individual player possession statistics. Even with a full season of data, it’s hard to know what they’ll put up the next year. Look at this year’s Vezina winner in Semyon Varlamov. Here are his last five seasons’ save percentages in reverse order: .927, .903, .913, .924, .909. If there’s this much variation at the season level, what the heck does that say about how individual games go, or even playoff series?

Why does this happen? To put it in a blunt and unsavory manner: there’s a lot of luck in hockey. I know it screws up our well-crafted sports narrative, but truthfully there are lucky bounces, deflections, and all kinds of random chance involved.

Basically, the message is: you know less than you think you do about how good a goaltender is. A 95% confidence interval tells you that 19 of 20 times, a goaltender’s career save percentage will fall between the low and high value. It’s the statistics equivalent of “I am basically sure this goaltender is this good.” I may not stake my life on it, but I would bet $20 confidently.

(Side note: in this spreadsheet I throw GAA out the window. When you want to talk about how good goalies are, you just can’t use GAA. Think about how it’s calculated: it’s a goalie’s save percentage multiplied by the number of shots he faces per game played. A .920 goalie who sees 20 shots per game on a good team will have a far lower GAA than a .920 goalie on a poor team who sees 40 per game. Half of this statistic is controlled by the goalie, but half is set by his teammates–and when you’re trying to isolate goalie performance, that’s really not fair. Here’s a post that explains it better than I can.)

Here’s the 95% confidence interval for every goalie who saw a shot in a 2013-14 regular season WHL game. CI Low is the lowest career save percentage we could expect, and CI High is the goalie’s ceiling:

ci1

Look at the pattern below: when a goaltender hasn’t seen many shots, he has a huge confidence interval. Each shot contributes to a smaller confidence interval, but it contributes a little less. The difference between 60 and 61 shots against is bigger than the difference between 1000 and 1001. So we really need to see young goalies before we can tell anything about how good they will be. I added a new variable on the right that’s the spread of how

ci2

You probably want to know who are the (objectively) best and worst goalies in the WHL. The goalies who are ‘proven’ to be best would have the highest save percentage for the low end of their confidence interval, i.e. their floor is very high. Here they are:

ci3

I arbitrarily picked .895 as the lowest value that would still qualify a good goalie. Sue me. Of particular interest for me are goalies who still have a high confidence interval difference–that is, more data will narrow their floor and ceiling values, so they could still pull up their low value a bit more.

And here are the ‘worst’ goalies, the ones with a low ceiling (below .900):

ci4

There’s a bit of sample size fuzziness going on here with all the goalies who played ~20 games. I’d ignore those for the moment and focus on Rathjen, Lee, Cotton, Moodie, and Sacher. For perspective, the league-wide save percentage last year was .903.

Lastly, I thought it would be interesting to compare last year’s save percentages with career numbers. Take this with a grain of salt, because I’m not pulling out 2013-14 numbers to compare with career numbers prior to the 13-14 season. There’s a bit of feedback going on.

Here are goalies (at least 10 games last season) who might be in for a rebound, as their 2013-14 numbers were at least .005 below their career average.

ci5

 

And here are goalies (1o+ games)who had a career year and might experience a drop in save percentage this year:

ci6

*A last note on Taran Kozun: I noticed that he had a .897 sv% with Kamloops, and then .928 with Seattle after he was traded. In terms of the whole season, his numbers were in the good/elite category. However, if you’re a Seattle fan and expect a .928 sv% this year, I doubt it. Team defense doesn’t affect sv% to remotely that extent. More likely he had a bit of a slump in Kamloops and played well in Seattle–as all goalies do, going through little slumps and little streaks that even out over the course of a season or career.

Decoding the Draft: The Effect of Height on Draft Position

I have something of a vested interest in researching how NHL teams view short players, and to what extent any height bias is warranted. In recent years, a number of the Winterhawks’ most prolific players have been on the shorter end of things when drafted. Sven Baertschi (5’10”), Brendan Leipsic (5’8″), Nic Petan (5’8″), Oliver Bjorkstrand (5’10”), and Derrick Pouliot (defenseman, 5’11”) all come to mind.

There are a number of intriguing perspectives on teams’ motivations for preferring tall players, and whether they are justified. This Arctic Ice Hockey question post aggregates a number of observations from members of the analytics community.

(When I was introducing my dad to this project (he’s an NBA fan), he pointed out, “Teams like to draft the player with the most potential. If you draft the small guy and he’s a bust, there’s not much you can say. If you pick the big player, at the end of the day you can still say you drafted the most athletic guy.” So at the very least, there’s the cover-your-ass perspective to be considered.)

Purpose of study and hypothesis

But for the purposes of this post, I put the question of motivation and true value aside. I’m not looking at how good short players are compared to tall players. I want to know how valuable teams think short players are compared to tall players. I begin with the null hypothesis that all teams value players equally, regardless of height. Given a 5’3″ prospect and a 6’3″ prospect who both scored at a point per game pace, a GM would flip a coin to decide who to draft. If short players are consistently picked far lower than their point production would suggest, while tall players are picked higher, there may be indications of a height bias in drafting.

Procedure

I began with Justin Fisher’s NHL Draftbook, a meticulous Excel spreadsheet including draft position, height, and weight for every drafted player from 1963 to 2013. I limited my study to 17 year old CHL players drafted from 2004-2013, which gives a nice sample size without the complications of European or NCAA equivalencies.

What is short? What is tall?

Because I later wanted to compare short and tall players, I needed to decide who fell into each category. Plus, I appreciate this kind of visualization when starting a project. I began by graphing height distributions for forwards and defensemen.

forwards height distribution

Forwards have a right-tailed distribution: the data is clustered on the left side but drawn out on the right side. Anecdotally, this suggests that while there’s a fairly definitive lower bound to the heights of drafted players, teams are more open to drafting exceedingly tall players. The bars in orange represent what naturally falls into a short and tall category.

defensemen height distribution

Here’s the same distribution for defensemen, who have a similar right-tailed distribution.

The Sham Model, Revisited

I adapted Rhys Jessop (Thats_Offside)’s Sham Sharron thought experiment to fit this project. In his original post, Rhys replaced every Canucks draft pick since 2000 with the highest point-getting 17 year old CHL player available at that time. Here, I wanted to be somewhat more precise about approximating point production, so I used points per game.

Let me take a moment to add the same disclaimer Rhys uses, although here it’s obviously for a different kind of thought experiment:

We did not design Sham Sharron to be a good or intelligent way to draft. Sham is a benchmark we have created to test each team’s scouting department against publicly available information. We do not for a second believe that any NHL team should draft in the same manner as Sham.

I converted the overall draft order of my selected sample to an ordinal variable (1, 2, 3, etc.). Then, I sorted each draft class from most to least points per game. I assigned the top PPG player “expected draft position” 1, and so on down the line. Here’s a snippet of how this looks for the 2013 draft:

ordinals

The difference between the expected and overall ordinal variables became a new variable Diff. A negative value indicates a player was drafted lower than anticipated, while a positive value suggests a player was drafted above where his point production would lead us to expect.

Results

What is the effect of height on draft position?

Remember, I said initially that given the choice between a 5’3″ and 6’3″ forward of identical point production, a GM without height bias would flip a coin to choose his draft pick. This graph pretty clearly demonstrates that shorter forwards systematically experience a decline in draft position relative to players of typical height, while tall players are given a boost above what their point production would suggest. On average a one inch increase in height, all else equal, leads to a 2.758 spot boost in ordinal drafting position. When you consider all the goalies, defensemen, non-CHL players, and overagers being drafted, this could well correspond to a far greater decline overall. With an r^2 of .1576 (.1795 if you adjust for CHL league/era using Rhys’s method), there are missing pieces for sure, but the relationship is definite.

 

sham difference by height overall

Here’s the damning graph I posted on Twitter a couple of weeks ago.

Let’s also take a brief look at defensemen. They’re mostly ignored in the Sham method, and for good reason–while we can be sure the primary responsibility of most forwards is point production, we can’t say that about defensemen. But I did the work anyway, just for fun. Here’s the graph:

sham difference by height d

A higher r^2 here suggests that although an increase in height causes draft position to rise by less than among forwards, the overall relationship is stronger. Height explains slightly more of why high-scoring short defensemen are passed over than it does for forwards.

sham difference by height d overall

Comparing the two condensed graphs, it looks like within a wide parameter (5’11” to 6’6″), defensemen experience non-trivial but relatively typical shifts in draft position. The extremes (5’9″ – 5’10”, 6’7″ – 6’8″) do interesting things, but I’m inclined to disregard that for lack of sample size.

Results by round

To get even more specific, we can take a look at results by round to see in which rounds players tend to be drafted in line with their Sham estimates and when they’re boosted/lowered.

sham difference by round f

It appears that the effect I noted earlier of tall players being boosted (whether deservedly or not) occurs only in the first three rounds–primarily the second round. Conversely, the perceived value of drafted short players seems to reach a steady low in the fourth round.

sham difference by round d

Once again, the perceived value of tall players is most exaggerated in the first three rounds, while short defensemen have a steadier decline around the fourth round. You’ll notice, too, that for forwards it bottoms out around -15 or -20, while at -10 the difference for defensemen isn’t quite as striking. I’m interested to hear that.

When are short/tall forwards drafted?

forwards height distribution by round

The data suggests that selection of tall forwards peaks in rounds 2 and 3, while there’s a noticeable spike in the 5th round for short forwards. This may be when teams take a flyer on a short guy with high point production. You’ll recall I pointed out that the market for short forwards bottoms out in the fourth round. In a world where height made no difference to NHL success, we would likely call short forwards drafted in the fourth round ‘steals.’

Results by team

I have results by team as well, which aren’t noticeably groundbreaking or enlightening ; I won’t clutter up this post with 30 charts, but I will include that data in the spreadsheet at the end of this post.

Discussion

There are numerous potential pitfalls related to my approach, so I’ll go ahead and address some of them here. First, I should reiterate that this method looks not at the value of prospects themselves, but how teams perceive the value of those players. I also fully concede that there are lots of areas in which old-school scouting matters, and for which there simply isn’t the data to create a more fleshed-out model. For example, Alex Galchenyuk isn’t even in this data set because he played only two games during his draft year due to injury. But lingering injuries that drag down success matter as well, and accounting for those is better done with a human touch. I also confess that while I initially intended to get rid of penalty minute outliers to reduce the influence of enforcers with zero point production, I just didn’t get around to it. Sorry, Marc-Andre Roy.

There’s also a problem with the fact that I converted draft position to an ordinal variable, removing the ability to discern value difference between picks. Picks 4 and 5 in my ordinal scale could be 7 and 10, while 30 and 33 could be 100 and 125. However, I think that effect is somewhat lessened. The numerical differences between picks in this model tend to be greater in later rounds, but given the value of a pick, the difference between 7th and 10th overall may in fact matter more than between 100 and 125. They come close to cancelling one another out.

I could do a few things in the future to increase the explanatory value of this model, if I wanted to take on the grueling, masochistic task of approximating a model for the draft. I would adjust for team success (does Nic Petan’s point total matter less in light of the Winterhawks’ massive scoring?), linemate success (avoiding the Mackinnon/Drouin, Kane/Gagner trouble), and isolating even strength points.

Most importantly, which I glossed over in the introduction, we can’t say for certain that all teams primarily value point production. As I already noted, enforcers are certainly desired by some teams, and they influence this model. Some teams pick for speed, for a ‘two-way player’ or ‘shutdown defenseman’, for speed, for pedigree, for the all-important and yet altogether unknowable “grit”. Whether or not those qualities contribute to NHL success, teams certainly think they do.

Conclusions

There is a statistically significant relationship between a player’s height and the difference between his ‘expected’ and actual draft position. This relationship exists for both forwards and defensemen, though a larger coefficient for forwards suggests that they are more advantaged/disadvantaged by teams’ perceived value based on their height. However, this relationship is not wholly predictive. There are other factors at work that it would be interesting to add: plus/minus, linemate success, team success, etc.

The logical next step is to look at the success of these drafted players, to confront the notion that short players struggle more to produce than tall players. My suspicion is that while height does play some role in NHL success, a) it is less important than hockey minds seem to think (market inefficiency?) and b) it is somewhat fed by an opportunity bias of tall players getting more chances to succeed at the NHL level/short players lacking those opportunities.

To step back, it sometimes seems odd how the mythology of height bias comes into play. An example I keep coming back to lately is 5’9″ Max Domi, 1.359 points per game, versus 5’8″ Nic Petan, 1.69 points per game. I struggle to grasp how that extra third of a point per game gets lost in the battle over an inch (and possibly a pedigree), and I welcome that potentially pedantic discussion.

Data

Here’s a Dropbox link to download most of the spreadsheet I used. Feel free to use any of it with credit (and hopefully credit to Justin Fisher, who created the Draftbook and deserves many accolades).

Extra Skater Junior Advanced Stats

Prior to its going dark, Extra Skater was the sole repository of advanced junior hockey stats. Fortuitously, Tyler Hunnex (@TylerHunnex, letsgobirds.com), a Seattle Thunderbirds blogger, had downloaded the entire WHL archive in Excel and was kind enough to send them along to me. Additionally, I was able to copy the entries for the top 1000 point getters in the CHL in 2013-14 from archive.org. Initially I only shared the links on Twitter, but I’m glad to post them permanently for downloading via Google Drive and Dropbox. Unfamiliar statistics are defined in the Glossary sheet of the CHL stats workbook.

2013-14 CHL stats // download via Google Drive // download via Dropbox

2013-14 WHL stats // download via Google Drive // download via Dropbox

2014 WHL Final Stats

Two evenly-matched teams met in the WHL Final this year. Each team scored 20 goals over the seven game series, four of them on the power play. Their save percentages were .918 and .916. If all you saw were these numbers, you wouldn’t know anything about the Winterhawks or the Oil Kings–their best lines and players, their style of offense, the efficiency of their defense. Looking at more telling numbers can supplement merely watching the series and give us an understanding of the above qualities.

Read the rest of this entry »

Follow

Get every new post delivered to your Inbox.

Join 121 other followers