In a small series of articles, we scrutinize a number of home brewed experiments aimed at testing whether scripting exists. Although most of them fail to meet basic, scientific requirements, there is something to be learned from them.
During FUT 14, FIFA Addiction published the results of this experiment about handicapping. The purpose was to examine whether better squads are handicapped.
The author borrowed a full squad of TOTS players and played some 20 hours against a friend. At some point, the involved players swapped squads and tried it out the other way around. It is not clear exactly what went on during those 20 hours of play, as there are no systematic records of results. In the – quite brief – concluding remarks the author states that ‘the players seemed sluggish compared to Gold teams we normally use. Not reacting well, not sprinting as quick as some 70 rated silver players we had used’ – and – that they ‘had to try twice as hard with TOTS and they weren’t winning tackles or making good passes’.
These observations led FIFA Addiction to conclude that TOTS players not only are leveled with the opposing team, but in fact downgraded. Purportedly, lower stats means better performance.
Personal views are not evidence
FIFA Addiction’s conclusion is largely based on weak and questionable observations such as that ‘the players seemed sluggish’. Such observations are not facts by any lengths. Neither this article nor the clips it portray allows us to conclude that the apparent sluggishness was caused by the players on the pitch rather than the humans controlling them. Of course, it doesn’t make things any better that the details about what exactly went on during those 20 hours of play are unavailable. We don’t know who played whom, how many times, with which squad and to what result.
Aside from these academic concerns, there are other reasons to challenge FIFA Addiction’s conclusions.
We miss a motive
I hope that others than me are a bit puzzled here: Why would EA want to reverse the effect of stats? For my own part, I can come up with at least two compelling reasons why it would be downright stupid to make players with better stats perform worse for a company that operates in the pack selling business:
- Reversing the stats wouldn’t make matches more even and hence fair. It simply would turn the advantage upside down.
- EA’s pack selling business relies on that we as consumers find it worthwhile to spend money on packs in search of rare players. Making Ronaldo perform worse than cheaper alternatives would effectively diminish the incentive to buy packs.
So, what FIFA Addiction left us with here is a conspiracy theory with a flavor of logic inconsistency, which is supported by an extremely weak set of data. This is not the kind of stuff that is going to win you a Nobel prize.
Better stats equals better performance
Aside from the problems listed above, we already have strong, systematic evidence contradicting the theory presented by FIFA Addiction here.
Up until EA shut down the web app during FIFA 15, FUTWIZ collected performance data from player cards up for sale on the market, enabling them to present assist and scoring averages for every player in the game. In this article, I make a comparison between comparable players and found that there was a direct link between stats (not necessarily rating) and performance. I took it even further in this article, where I studied the exact relationship between individual stats and overall performance, and was able to determine which stats contribute the most
Of course you may lose against a lower rated squad, but having the better players does improve your chance of winning. Hence, stats are definitely not reversed and squads are definitely not leveled. If they were, there is no way, players with higher stats could have ended up scoring more goals per match than comparable, lower rated players.
Why one should approach such experiments with skepticism
One of the purposes of discussing experiments like the one presented by FIFA Addiction is to demonstrate that it is relatively easy to create experiments, which at a glance and with the right presentation skills will appear to prove just about anything.
You obviously can’t measure whether better squads are handicapping by feeling your way forward in any meaningful way. In reality, this experiment confuses claim with evidence. Observing that your players feel sluggish in a number of matches is nothing but a repetition of a claim. It has nothing do to with evidence, and no matter whether you base your claim on 1 hour of play or 20 hours of play, it doesn’t become evidence.