Whenever someone in the hockey management community uses “analytics” in a sentence and their name isn’t Kyle Dubas or John Chayka, I ask myself the same question each time — are they talking about analytics because they used them in their decision-making process, or because they want to make their statement appear to hold more water than it actually does by name-dropping analytics and hoping no one fact-check’s their work?
Take these comments by Vegas Golden Knights general manager George McPhee for example.
“Goaltenders can play longer than most other positions. We made this decision based on some analytics, some experience and the type of athlete Marc-Andre is.” -McPhee
— SinBin.vegas (@SinBinVegas) July 13, 2018
The use of the word “analytics” implies a statistical based, non-emotional, entirely unbiased decision-making process that is verifiable by simply studying the data.
That implication has within it the power to change the perception of the signing.
So, we are forced to ask ourselves, which was this? An explanation of his approach toward signing veteran goalie Marc-Andre Fleury to a three-year extension at an average annual clip of $7 million, taking place between the ages of 35 and 37, by using, among other things, analytics… or was it McPhee trying to pull the wool over the eyes of fans and media members alike (of which I include bloggers) by using a buzzword with the expectation that many won’t look too deeply into the subject?
Are there statistics that support the decision to purchase Fleury’s 35-through-37 seasons at $7 million per year?
We start by first coming to understand a concept I only recently became aware of thanks to our own Dalton Mack. Survivorship (or survival) bias.
Via Wikipedia:
Survivorship bias or survival bias is the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. This can lead to false conclusions in several different ways. It is a form of selection bias.
Survivorship bias can lead to overly optimistic beliefs because failures are ignored, such as when companies that no longer exist are excluded from analyses of financial performance. It can also lead to the false belief that the successes in a group have some special property, rather than just coincidence (correlation proves causality).
To sum this up in terms relevant to that of what we’re looking at, we need to understand that the goalies we’re going to be talking about are not a full representation of the whole. They are, most often, an elite subsect of goalies which is why they have played well into their thirties, while the vast majority of goalies do not.
They themselves are the exception and not the rule, which means we need to understand what we’ll be looking at is not necessarily a definitive look into the topic we’re discussing.
Wikipedia gives this example:
For example, if three of the five students with the best college grades went to the same high school, that can lead one to believe that the high school must offer an excellent education. This could be true, but the question cannot be answered without looking at the grades of all the other students from that high school, not just the ones who “survived” the top-five selection process.
Feel free to re-read that a couple times. I know I did.
So, now that we’re all on the same page, let’s continue.
Setting up the data pool
Since 1999-2000, a time frame I’m not all to comfortable using but do so for the sake of an enlarged pool of players, there have been 55 goalies who have played games while between the ages of 35 and 37, the years we know the Fleury extension will be active through.
One of those is Scott Foster, the emergency backup who made seven saves on seven shots. While a great story, Foster is not particularly relevant to the point being discussed, so we’re going to ignore him.
To make life a bit easier for us (specifically me writing this), we’re going to make an assumption and use it to inform the data we keep versus that which we ignore. This in the hope of eliminating statistical noise.
The assumption is that Marc-Andre Fleury will remain (mostly) healthy through his three-year extension and, as the “face of the franchise”, he will not be bought out or placed on Long Term Injury Reserve (as a way of getting the contract off the books or opening a roster spot, in such a way as has been done to players like Joffrey Lupul). He might become a very expensive backup but that is neither here nor there.
As such, we’re setting a lower limit for games played at 98. This gives us our pool of names, which is now limited to just the 15 most-active goalies, ones who played at least that number of games between the desired ages.
We lose out on names like Curtis Joseph, Mike Richter and Jose Theodore, as well as some of Fleury’s contemporaries such as Henrik Lundqvist, Pekka Rinne and Mike Smith, but the sample size for some of them (most, Joseph aside, are below 70 games) is just too small for the purposes of this exercise. If analytics were used here, sample sizes had to be as well, after all.
Aging curves and what we know about them
Hockey Graphics has done a lot of research into the subject of aging curves, as is evident by their 2014 article here.
Here is what they say about the subject (after trying to fit the data into a curve they instead opt for a more linear approach).
The fit here is slightly worse than the curve (the R^2 drops .04), but the gradual drop in goalie performance is a bit more believable toward later goalie ages than the quadratic fit. Now, the drop in performance relative to average looks like this:
Change from 35 to 36 years old: -.004
Change from 36 to 37 years old: -.004
Change from 37 to 38 years old: -.004
Change from 38 to 39 years old: -.005
Change from 39 to 40 years old: -.005
So now our 35 year old above average .920 goalie is basically just barely below average in two years at .912, and at age 40 is still horribly bad – .898, but isn’t gong to put up one of the worst goalie seasons ever (these #s may look similar to that of say Martin Brodeur who at 40 is now around the .900 area).
The chart they are referring to is this one:
Goalies, at least for the most part, don’t tend to simply fall off a cliff into oblivion but instead see a more gradual decline in statistical output that takes an elite goalie (they use a .920 goalie as the model) down into a slower, more team-friendly decline.
Of course, in the wording there is where we might find a snag. This presumes the goalie starts the curve off being elite.
And, I grant you, Save Percentage is not the absolute best way to determine a goalie’s performance. Unfortunately, so far as I know, no one has done an aging curve on High Danger save percentage or expected save percentage. For this reason we’re just going ahead and using what we have available.
Crunching numbers
Important thing we need to make clear before continuing to look at the numbers as recency bias (yes, another bias, there are a lot of them) threatens to skew how we view the data.
The league average save percentage has gone from .904 in 1999-00 to as high as .915 (on multiple occasions), to the .912 we saw in 2017-18. This is why, in a nutshell, I wasn’t particularly happy going back this far for goaltender data.
This effects how we view the numbers we’re being presented with as it means that goalies like Patrick Roy, who played his 188 games from 2000-2003, or Dominik Hasek, who played 167 games at ages 35 through 37 in 1999 to 2002, need to be held to a different standard than, say, Roberto Luongo and his 113 games between 2014 and 2017.
I’ll be doing my best to provide context to each goalie based on the era they played in to hopefully clear up what the difference between a .908 goalie in 2003 is compared to one in 2017.
Breaking it down, five of the goalies were below the league average during this time of their career, one was even with the field, and the other nine were, at least somewhat, above average compared to league average.
This is good… right?
Well as you’ll remember when I explained survival bias, what we’re finding is exactly what the aging curve predicted would happen. Roy, Brodeur and Hasek, goalies who were elite-level goalies playing into their mid-to-late thirties, and even Luongo who has had a long career of far above average numbers with his .919 save percentage, all remained very good goalies with age.
These players represent four of the nine goalies who saw their play remain above league average in their 35-through-37 years.
What we’re left with is basically a tie. Five all, with one draw. If that is the group that Fleury more accurately finds himself in then it changes whether the numbers support his contract.
Perception, as it pertains to the veteran netminder, is the key factor here.
Separating fact from myth about Marc-Andre Fleury
Depending on who you talk to Fleury is either an elite goalie whose year in Vegas is more in-line with what Fleury is on a skill-based level than the raw numbers of his 14-year career, or he’s an average to slightly above goalie who fluctuates wildly from season to season and is just happening to come off a career year.
Which opinion is true to you likely depends on how much value you put in statistics (he is a career .913 goalie, though he has been a .916 goalie since 2007-08), or, possibly, how much you like Fleury or how importantly you view his 46 games (not including the 20 playoff games) in Vegas.
This perception of Fleury, and the distinction between them, matter greatly in this conversation because what we’ve been looking at is an aging curve that presumes, and a data pool that may consist of, elite-level goaltenders.
If Fleury is not an elite goalie, but instead the inconsistent one with both major upside and wild fluctuations in overall play year-to-year his career numbers suggest he is, the aging curve nor some of the players discussed will do him justice.
That wouldn’t bode well for the Golden Knights.
Conclusion and final thoughts
To conclude the original point of the article before moving on to what it all means…
Yes, so far as my abilities in math and research are able to determine, it is entirely possible that GMGM and his team used analytics and statistical models to project out Fleury’s age-based decline and decided, along with their wealth of experience and knowledge of Fleury’s physical attributes, that his decline would likely follow the path of the goalies who remained at or above league average into their mid-thirties.
Yes. This is fair. His use of the word “analytics” holds water.
Well, sort of.
Because for it to hold water it would also mean that he, unlike many who also look to use and understand analytics and the insights they offer, believes Fleury to be an elite goalie. That his one season with Vegas overrules his career with Pittsburgh and that his decline will follow the path of Roy, Hasek, Luongo and Brodeur.
The pragmatist and the pessimist in me has to ask… what if he isn’t?
If he is not, his aging curve that starts at the .913 career save percentage (or even then .916 since 07-08 number from earlier) looks a lot worse going forward than what the model predicts, due to those predictions being for a .920 goaltender. What we’d be seeing is a stiff decline from slightly above average goalie to below (possibly well below) as the extension kicks in.
All for the price of $7 million per season.
If he isn’t an elite goalie then it would appear as though Vegas has taken a great risk, a literal 50/50 chance that Fleury remains, at the very least, good throughout his extension. That’s worrisome to those of us who have their eyes on Vegas’ long-term prospect of cup contention.
It can’t be said McPhee didn’t use the information available to him in making his decision, and likely that information extends well beyond even what I’ve used here (but an article that hits on EVERY piece of available information would likely be twenty-thousand words long and I dare say my editors would mutiny) though the question as to whether he’s interpreted the data correctly or been led astray due to a faulty perception is still to be determined.
The answer to that will have great and lasting impact on the future of the Golden Knights.