On Thursday, August 6, 2015 at 8:28:24 PM UTC-4, Andy Blackburn wrote:
On Thursday, August 6, 2015 at 7:40:21 AM UTC-7, Andy Blackburn wrote:

We now have at least one contest we can compare the results from to see if pilots scored significantly differently with stealth vs not - there is normally a lot of variance in the scoring even over a long contest so it may or may not show anything but at least we will have an indication if anything significant is going on.


I ran some numbers to test the theory that less skilled pilots use Flarm as a way to up their performance - and that Stealth mode helps negate this strategy. I compare the last two 15 Meter Nationals - 2014 at Montague (Flarm without Stealth Mode) and 2015 at Harris Hill (Flarm with Stealth Mode). I compared each pilot's performance as measured by final score as a percentage of winner's score to their PRL percentage, which is the longer-term average of the same metric and is used for US Team selection so it's the best available measure of overall pilot skill. The metric I used was average absolute error for each contest - on average how closely did contest performance match PRL percentage. I dropped from the analysis any pilots who withdrew from either contest.

The theory we are testing is that with unrestricted Flarm you should get more inexperienced pilots higher in the scoresheet with higher scores and potentially some of the more highly skilled pilots dropping down as a result..

The average absolute error for the 2014 15-Meter Nationals at Montague was 7.4%. That is, on average pilots scored only 7.4% off of their PRL. Four pilots were competing in Standard Class gliders at Montague without handicaps. If I give them each a 4% handicap the average absolute error drops to 7..1%.

The 2015 15-Meter Nationals at Harris Hill had an average absolute error of 11.8%. That a significant difference and not in the direction you'd expect. It turns out that at Harris Hill a number of highly ranked pilots under-performed their PRL and a few lower-ranked pilots out-performed their PRL.

This is 35 pilots flying a total of 259 contest flights, so it's not a super big sample size, but not terrible given the stark difference. I also looked for any potential sources of sample bias - average pilot skill, contest length or devalued days. The two contests were pretty similar. Average PRL for Montague was .899 and Harris Hill was .906. Montague was 8 days and Harris Hill was 7 days. The average daily winner's score at Montague was 869 points and Harris Hill was 844 points. Not perfect, but pretty close.

So, if the theory is that Flarm allows lower skilled pilots to leech their way to the top of the score sheet and Stealth mode puts a lid on this behavior, the data suggests the opposite. In fact a number of long-time (30-40 years) highly ranked pilots under-performed their PRL with stealth mode. One potential reason for this is that more experienced pilots are actually better at making use of Flarm information than inexperienced pilots. The other theory is that Flarm helps reduce the random landouts that tend to scramble the scoresheet and actually reinforces stability and order.

Discuss.

9B

P.S. I already said that it's only 259 data points.

Weather variability is most likely a bigger factor than Flarm. To wit- SM, who never lands out got snookered multiple times at Elmira. Same with KS. Only 3 pilots completed all tasks assigned. I suspect the weather had a much bigger effect on performance variability from the expected than Stealth- or not- ever would.
It is also worth noting that FS, who has a lot of Flarm expertise relative to the rest of us, won 15M in Stealth. Would he have enhanced his advantage? I think so.
UH