- It was not sensitive to results other than first place.
- It could reject the null hypothesis that the motion is "fair" simply when one team's results had a lower variance. Since it's at least folk wisdom that Opening Government teams typically have lower variance in their results, it could reject motions as unfair for reasons that are endemic to the format, and nothing to do with that particular motion.
After a bit of thought, I instead propose the following alternative:
A motion is fair iff the expected number of team points for a team in any position is 1.5.
This has the useful quality that it is sensitive to all the ranks that a team could attain, not just first place. It also avoids mis-identifying variance issues as unfairness issues. It seems, to me at least, that a particular position having a higher variance in its results does not ipso facto make a motion unfair. It also seems to me a feasible standard, in that CA teams could reach it (with lots of effort, critical thinking, and a little luck). (It is certainly more feasible than the extreme alternative, that teams in any given position must have an equal chance of coming first, second, third or fourth.)
What follows is a cookbook, so that any interested party could implement this test. I assume a very basic knowledge of matrix algebra. If you're not interested in the maths, feel free to skip to the end.