Fact check: Does the keeper make more saves against higher rated opponents?

Share

In a recent Reddit post, Arlington69 presents statistical evidence which appear to confirm that goalkeepers tend to have a higher save ratio when your team is matched up against higher rated squads. If he is right, this could suggest that handicapping exists. But do the numbers really add up the way Arlington69 thinks? We did the crunching.

The claim

The basis of Arlington69’s latest claim is the statistic below. Using data collected from his own matches, Arlington69 calculated his keeper’s save percentage when matched up against opposing squads rated lower and higher than his own squad.

Screen dump of part of the data presented in the original post.

The important column is “Shots on target Saved” (in bold). You will notice that Arlington69’s keeper made more saves when the opposing team was rated +4 than when it was rated -3 relative to his own squad. This leads him to the following conclusion:

“When I played a team with a rating higher than mine their keeper saved 50% of my shots on target whilst when I played a team of lower rated than mine their keeper saved 60% of my shots on target. You would expect player with higher stats to find it easier to score goals than lower rated players. If handicapping exists the result is what you would expect to see.”
(– Post on Reddit)

Criticism

It obviously would require some fairly big samples to conclude that keepers in general make more saves against higher rated opponents. Although Arlington69’s samples in total cover more than 6000 shots on target, the actual number of independent observations in each of the five samples are much lower because many of the shots were made by the same players. In total, the five samples cover a maximum of 542 different players. Hence, the actual sample sizes vary between 39 and 268 independent observations.

To test whether those samples are sufficiently big to conclude that the measured differences in save percentage are statistically significant, we calculated 95 % confidence intervals for all five rating difference levels. A confidence interval is used to determine whether the observed differences are statistically significant or random.

Note that we calculated confidence intervals both when using the number of matches and the number of shots on target as sample size.

The results can be seen below:

Matches (M) Goals / Match Shots on Target per Match
(SoTpM)
Shots on Target
(M * SoTpM)
Observed save % Save % Confidence interval
(n = M)
Save % Confidence interval
(n = SoT)
<= -3 98 3.2 7.2 706 55,6% 45% – 65% 51% – 59%
-1 and -2 194 3.3 8.0 1552 58,8% 51% – 65% 56% – 61%
0 and 1 268 3.5 8.4 2251 58,3% 52% – 64% 56% – 60%
2 and 3 137 3.3 8.2 1123 59,4% 51% – 67% 56% – 62%
>= 4 39 3.6 9.5 371 62,4% 47% – 77% 57% – 67%

What you should notice is that the confidence intervals are overlapping between all five samples, no matter whether we consider the sample size to be the number of matches (n = M) or the number of shots on target (n = SoT). In statistical terms, this means that the measured differences aren’t statistically significant, meaning that we can’t conclude that keepers in general make more saves against higher rated opponents.

Simply put, Arlington69’s samples are far too small to tell us anything about handicapping.

Conclusion

As noted by Arlington69, you would expect players with higher stats to find it easier to score goals than lower rated players under normal circumstances. And for all we know, they do.

In earlier articles, we have demonstrated that higher rated versions of a player score significantly more goals than the lower rated editions. This inevitably leads us to conclude that higher rated teams in general score more goals. All other things equal, a team containing the 95 rated TOTM Lukaku minimum has a +1 higher OVR rating than a team using 86 rated Lukaku because TOTM Lukaku’s +9 rating difference increases the OVR by 1. In addition to that, it is likely that the average team containing 95 rated TOTM Lukaku will contain better team mates as well, not least because it was released later in the year where teams in general tend to have improved. Thus, the fact that 95 Lukaku scores more goals than 85 Lukaku implies that higher rated teams in general score more goals than lower rated teams, which in turn means that we can rule out the notion that keepers make more saves against higher rated opponents.

%d bloggers like this: