Failure to Fly--Summary

[erstad]

Since, oddly, I consider statistics fun, I’m choosing to weigh in.

I’m not a PhD nor do I lecture in statistics, but I am a certified six sigma black belt, use statistical methods almost daily in my job, and not infrequently mentor others in various techniques.

IMHO, herwin is right… mostly….

First, the criticism that you need some fixed number of samples to be statistically significant is not exactly right. The sample size needed to establish statistical significance is a function of the difference between the populations being sampled. If the populations are very different, it’s likely that a much smaller set of samples will demonstrate statistical significance. If the populations are only slightly different, you may need a huge number of samples to prove it. (Although in real life, this isn’t usually a problem because if the populations are that close, the fact that they are different is probably not practically significant).

So, is seven samples enough for this case? That answer can be provided mathematically. I would approach the problem differently; I would do a Chi Test to see if there is a statistically significant difference. (Not saying this is better than herwin’s approach, a chi test is just the hammer I bring out for anything that looks like a nail. As a practicing engineer rather than a theoretician I’ve found that even the “wrong” tools often yield useful results as long as one understands the assumptions the tool makes about the data).

Doing this analysis on the data (1/5 fly for case 1, 2/2 fly for case 2), I get statistical significance at the 0.053 level. Pretty close to what herwin reported with his methods. This basically means that getting a sampled difference as large as that observed is a 20 to 1 long shot if there is truly no difference.

However, if the data were 1/5 fly for case 1, 1/2 fly for case 2 (20% versus 50%), that would have a p-value of about 0.43, way above any valid argument that a relationship exists. But you also can’t prove a relationship doesn’t exist, hence illustrating that sample size of 7 may or may not yield a statistically significant result.

However, there is one key limitation that herwin didn’t mention. All of these methods make the assumption that the trials are independent. For example, I assume most of the readers know that if you just run the exact same turn you get the exact same result. You could run 10 identical trials, get some result A, set some bombers on “city attack”, run 10 more trials, get different result B, and erroneously conclude that “putting bombers on city attack causes unrelated results to change from A to B” - because the assumption of independence wasn’t valid. The second through tenth trials were not independent.. Examples of bias that might creep in: Sounds like the first five attacks were five sequential days, and the next two followed those, so perhaps something else was temporally correlated. Ships are more likely to attack when lower on fuel; or the DL of the target increased; probability of attack is linked to phase of the moon and the latter tests were at the right phase, or… I am NOT arguing whether or not herwin’s results are flawed due to such a factor, only pointing out the underlying assumption of his math (and mine).

The best test would be probably be one where the same day is run over and over, and the number /size of attacks that preceed this one are modulated to move the random number sequence). Although even that doesn't guarantee independence. (Actually, the best test would be a code review... but I assume that's not an option)

Also, there is no selection bias by only looking at the carriers, because that was determined before the test starts. If I want to find the average number of heads a coin flips, I can get equivalent results whether I flip the coin 100 times, or whether I flip it 200 times and only look at the even numbered flips. What I can’t do is flip the coin 200 times and then look at the data and decide where to take the 100 samples.

[rtrapasso]

Also, there is no selection bias by only looking at the carriers, because that was determined before the test starts.

Not sure about this... it was only AFTER the carriers failed to fly 3 times that the test was started. IF you pick them AFTER this, this eliminates the first three failures.

Then we have 5 tests- 3 secondary (fly) and 2 primary (one fly, one failure).

[erstad]

ORIGINAL: rtrapasso

Also, there is no selection bias by only looking at the carriers, because that was determined before the test starts.

Not sure about this... it was only AFTER the carriers failed to fly 3 times that the test was started. IF you pick them AFTER this, this eliminates the first three failures.

Then we have 5 tests- 3 secondary (fly) and 2 primary (one fly, one failure).

Ah, I missed that part. Must have been in the original thread (I recall seeing that one, but only read this one in preparing the response.) If it's true that the days that led to the "hmm, I wonder if something is odd here" are also in the seven day sample, then I agree.

[herwin]

ORIGINAL: erstad

ORIGINAL: rtrapasso

Also, there is no selection bias by only looking at the carriers, because that was determined before the test starts.

Not sure about this... it was only AFTER the carriers failed to fly 3 times that the test was started. IF you pick them AFTER this, this eliminates the first three failures.

Then we have 5 tests- 3 secondary (fly) and 2 primary (one fly, one failure).

Ah, I missed that part. Must have been in the original thread (I recall seeing that one, but only read this one in preparing the response.) If it's true that the days that led to the "hmm, I wonder if something is odd here" are also in the seven day sample, then I agree.

Isn't exploratory data analysis fun?

The WTF reaction was triggered by the first failure to fly. However, there may be an order dependence and a non-linearity.

> wake<-data.frame(attack=c(0,0,0,1,0,1,1,1),mission=c("P","P","P","S","P","S","P","S"),
+ day=1:8)
> wake
attack mission day
1 0 P 1
2 0 P 2
3 0 P 3
4 1 S 4
5 0 P 5
6 1 S 6
7 1 P 7
8 1 S 8

> attach(wake)
> modelm<-glm(attack~mission*day,binomial)
Warning message:
fitted probabilities numerically 0 or 1 occurred in: glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,
> modela<-glm(attack~mission+day,binomial)
Warning message:
fitted probabilities numerically 0 or 1 occurred in: glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,
> modelx<-glm(attack~mission,binomial)
> modely<-glm(attack~day,binomial)
> modeln<-glm(attack~1,binomial)
> anova(modelm,modela,test="Chi")
Analysis of Deviance Table

Model 1: attack ~ mission * day
Model 2: attack ~ mission + day
Resid. Df Resid. Dev Df Deviance P(>|Chi|)
1 4 5.0678e-10
2 5 2.8000e-10 -1 2.2678e-10

--Ouch!--

I'd like to find the time to rerun the same turn over and over again, but then the game doesn't advance.