Failure to Fly--Summary

[herwin]

I now have some data on this problem. For the 5th Carrier Division, I have seven good (well-controlled) samples, five on days when the airgroup squadrons had a primary attack mission against a designated nearby base, and two on days when the squadrons had a secondary attack mission against the base combined with a primary naval attack mission (with no nearby enemy surface units). On one of the five days in the first set, the airgroup flew; on both of the days in the second set, it flew. I did a analysis of these statistics using R and a general linear model with binomial family. The analysis of variance comparing the null hypothesis with the effect of the mission being factored in was able to reject the null hypothesis at the 3% level.

That is, the probability that these statistics would be generated by a common process--the choice of mission not being significant in determining whether the airgroup flies--is about 3%.

Translating into human-speak, that says the difference between the two cases is probably important. Given that the airgroup is significantly less likely to fly when the mission against the base is primary, it highlights a problem with the game engine.

[DuckofTindalos]

What was your null hypothesis?

And seven samples isn't statistically significant. Seven HUNDRED might help.

[herwin]

ORIGINAL: Terminus

What was your null hypothesis?

That the mission being primary or secondary did not affect the probability of it being flown.

And seven samples isn't statistically significant. Seven HUNDRED might help.

Read what I wrote. Seven samples can give a statistically significant result in a well-controlled experiment. In this case, it allows me to reject the null hypothesis with about 3% chance of being wrong.

[DuckofTindalos]

And I'm still avowing that seven samples isn't enough in a game model that potentially can fly hundreds upon hundreds of missions every turn.

[herwin]

ORIGINAL: Terminus

And I'm still avowing that seven samples isn't enough in a game model that potentially can fly hundreds upon hundreds of missions every turn.

Isn't enough for what?

It's enough to be able to falsify a null hypothesis involving a binary response variable when the experiment is well-controlled. Do the analysis; I've given you everything you need. Granted, if I was reporting on samples randomly selected from the population, seven would not be enough, but seven samples differing in the specific missions and the weather at Wake Island on seven successive days is a different story. The null hypothesis is that it's a coin flip (of an unfair coin, yes) whether the attack takes place on a given day. The alternative hypothesis is that the rates are different depending on which of the two missions is ordered, and the ANOVA indicates that the null hypothesis will produce the pattern seen about 3% of the time.

The choice of making the mission primary or secondary affects whether it launches, and in a completely unexpected direction--introducing a requirement that the air group check for a naval target first increases the probability that it will attack the base. That suggests the logic executed for the two cases is disjoint, with differing endpoint probabilities. The right way to code would have been to do the naval target check as a branch on the base attack logic, with the code returning to the base attack once it has decided not to do the naval attack, either because the naval attack is not ordered or because there's no need. That's not what the numbers say is occurring.

[rtrapasso]

Try this thought experiment:

Flip each of 100 coins (numbered 1-100) each 7 times (for a total of 700 flips.)

There will be a few coins in there that come up tails 6 times in the 7 flips.

If you pick out one those few coins and claim that there is a bias (i.e. flaw) in the randomness of the coin that is picked out, is this a good conjecture?

[DuckofTindalos]

Your method is obviously correct, from a "mechanical" standpoint, but the data you're putting into the method isn't extensive enough, and this skews the result in the direction you want it to go. Sorry, but I've seen it before.

[rtrapasso]

ORIGINAL: Terminus

Your method is obviously correct, from a "mechanical" standpoint, but the data you're putting into the method isn't extensive enough, and this skews the result in the direction you want it to go. Sorry, but I've seen it before.

Were you replying to me, or Herwin? [&:]

[herwin]

ORIGINAL: rtrapasso

Try this thought experiment:

Flip each of 100 coins (numbered 1-100) each 7 times (for a total of 700 flips.)

There will be a few coins in there that come up tails 6 times in the 7 flips.

If you pick out one those few coins and claim that there is a bias (i.e. flaw) in the randomness of the coin that is picked out, is this a good conjecture?

We go non-linear when a researcher does that--that's cherry-picking the data. (I consult and teach statistics.) There are 128 possible sequences of seven coin flips, of which one will be all tails and seven will have a single heads, so the expected number of times we get six out of seven tails in 100 sequences with a fair coin is about 5.46. On the other hand, suppose we randomly select one of the 100 sequences, and discover that it has six tails. If we reject the hypothesis that the coin is fair, we will err only about 5.46% of the time. That's what I'm doing. The coin is unfair (and unfair in the wrong direction). Statistics tells me I'll have to eat my words about 3% of the time.

[rtrapasso]

ORIGINAL: herwin

ORIGINAL: rtrapasso

Try this thought experiment:

Flip each of 100 coins (numbered 1-100) each 7 times (for a total of 700 flips.)

There will be a few coins in there that come up tails 6 times in the 7 flips.

If you pick out one those few coins and claim that there is a bias (i.e. flaw) in the randomness of the coin that is picked out, is this a good conjecture?

We go non-linear when a researcher does that--that's cherry-picking the data. (I consult and teach statistics.) There are 128 possible sequences of seven coin flips, of which one will be all tails and seven will have a single heads, so the expected number of times we get six out of seven tails in 100 sequences with a fair coin is about 5.46. On the other hand, suppose we randomly select one of the 100 sequences, and discover that it has six tails. If we reject the hypothesis that the coin is fair, we will err only about 5.46% of the time. That's what I'm doing. The coin is unfair (and unfair in the wrong direction). Statistics tells me I'll have to eat my words about 3% of the time.

OK - here is the crux then: since the carriers are important, and you are focusing on them (and not the other umpteen air units), wouldn't picking that particular unit out (because it is important) bias the selection?

It isn't really a random sample because the particular operation is important (whereas you might not have noticed some units in China not flying.)

From my experience in the game, i'll notice that a particular unit may not fly for many turns in a row. i fiddle with them and they start to fly again (eventually). Did my fiddling with them help, or not? It's hard to know in a PBEM game although i can (sometimes) trace down effects in an AI game. Do i think it is a bug? There is really no way for me to know because there are just too many die rolls involved, and the tools i have don't allow me to know WHY the units don't fly.

Do i wish there was some way to know why unit X didn't fly on any particular turn? Yes! Do i think i'll be able to find out why? Alas, no.... and it might not be from any ONE factor, but a combination of factors (i.e. morale combined with leadership effects combined with weather combined with fatigue combined with... etc.)

[herwin]

ORIGINAL: Terminus

Your method is obviously correct, from a "mechanical" standpoint, but the data you're putting into the method isn't extensive enough, and this skews the result in the direction you want it to go. Sorry, but I've seen it before.

I'm continuing to sample.

Let's assume the null hypothesis. The maximum likelihood estimate of the probability of an airstrike being launched is 3/7. When the mission is primarily naval strike, the probability that both times will launch is 9/49 or about 19%. When the mission is primarily base attack, the probability that one out of five missions will launch is 3840/16807 or about 23%. The combination we see has a probability of about 4.2%. That's small enough that most scientists would dig a lot deeper.

[jwilkerson]

Well despite all the "hulla-baloo" one way or the other ... herwin may be on to something here.

In my game with Moses, way back in Feb 42 I conducted the only real sustained set of attacks by IJN carriers against landbases that I've done in this game. For this game at least, I made the doctrinal decision before the game started that I would use the KB as an "anti-shipping" weapon and not as an "air superiority" weapon. I made this decision mostly due to the fragile nature of the KB vis-a-vis the American counterpart. Oh and I also made the pre-war decision never to split KB but instead to use it "enmass" primarily to offset the American carriers - at least for as long as possible.

But anyway, back in Feb 42 my plan was to use the entire KB in support of airbase suppression on Java. This being the one and only exception to the pre-war doctrinal decision. From air reconnaissance, operating patterns and deduction I believed that Moses had substantial air assets on board Java and that it would take a full court press to be able to overcome these - and Full Court Press meant all LBA and all CBA that could possibly be brought to bear. I had captured and was building up and stocking up airbases on Borneo at both "ends" so Pontiac and Banjarmasin and Balikpapan, and I think Kuching was used as well. So Sallys and Zeros out of Banjarmasin and Betty's and Zeros out of the rest.

There was a point at which the land based air was not ready and the invasion troops were still proceeding to their assembly areas at Kendari and Balikpapan. So I had about a week to try to use KB to attrite the air on Soerabaja end of Java. So I went in three times from Kendari to a point about 5-6 hexes away from the nearest L4 airbase on Java and launched sweeps from the carriers. Using about half the fighters on the sweeps and the other half as CAP. The first time I tried this nothing flew. When I checked weather after the turn, everything in the area was "T" so I figured those were the breaks. Being a careful fellow I then pulled back and refueled to break the Allied spot on KB. Then I went in and tried again. Same result. No fly. I don't recall what the weather was this time, but it wasn't "T". But I pulled back again and refueled and went in a third time. Still same result, no attacks. By this point the land based air was ready and the troops were ready, so we launched land based attacks and combined those with carrier based attacks and this time everything flew and after several turns of sustained pounding we forced Moses to pull back his air from the bases around Soerabaja and the "liberation" of java could begin.
While I wasn't trying to conduct "deep analysis" on why the KB planes did not fly, I was of course curious. At the time I believe I explained it - in my head anyway - by deciding that the threat to my carriers from the land based air was a factor. I can't even recall what my orders were, but either half all sweep and half all cap - or perhaps all of the fighters on 50% cap with all also sweeping. Can't recall.

I have looked many times at the chunk of code known as "CreateStrikeFlight" and this code contains about a dozen "fly" checks, involving various factors. But I don't see anything in there that would be specifically related to what herwin (or I) have experienced. So there may be some carrier specific "AI like" code somewhere else that influences this situation.

Within our current division of responsibility, I'd probably ask MichaelM if he wants to look at this to see if he can find anything that looks out of wack. If herwin has a save he'd like to send us, he could send it to rtrapasso and we could take it from there and get it to Michael (assuming it reproduces for us).

[herwin]

ORIGINAL: rtrapasso

ORIGINAL: herwin

ORIGINAL: rtrapasso

Try this thought experiment:

Flip each of 100 coins (numbered 1-100) each 7 times (for a total of 700 flips.)

There will be a few coins in there that come up tails 6 times in the 7 flips.

If you pick out one those few coins and claim that there is a bias (i.e. flaw) in the randomness of the coin that is picked out, is this a good conjecture?

We go non-linear when a researcher does that--that's cherry-picking the data. (I consult and teach statistics.) There are 128 possible sequences of seven coin flips, of which one will be all tails and seven will have a single heads, so the expected number of times we get six out of seven tails in 100 sequences with a fair coin is about 5.46. On the other hand, suppose we randomly select one of the 100 sequences, and discover that it has six tails. If we reject the hypothesis that the coin is fair, we will err only about 5.46% of the time. That's what I'm doing. The coin is unfair (and unfair in the wrong direction). Statistics tells me I'll have to eat my words about 3% of the time.

OK - here is the crux then: since the carriers are important, and you are focusing on them (and not the other umpteen air units), wouldn't picking that particular unit out (because it is important) bias the selection?

It isn't really a random sample because the particular operation is important (whereas you might not have noticed some units in China not flying.)

From my experience in the game, i'll notice that a particular unit may not fly for many turns in a row. i fiddle with them and they start to fly again (eventually). Did my fiddling with them help, or not? It's hard to know in a PBEM game although i can (sometimes) trace down effects in an AI game. Do i think it is a bug? There is really no way for me to know because there are just too many die rolls involved, and the tools i have don't allow me to know WHY the units don't fly.

Do i wish there was some way to know why unit X didn't fly on any particular turn? Yes! Do i think i'll be able to find out why? Alas, no.... and it might not be from any ONE factor, but a combination of factors (i.e. morale combined with leadership effects combined with weather combined with fatigue combined with... etc.)

I agree. This PBEM is a beta test of an RHS scenario, so I'm watching for these things. I've noticed that my other two carrier forces (which have naval attack as their primary mission and base attack as a secondary mission) launch a base attack every turn they're within range of an enemy base. Kwaj launches an airstrike every turn against Wake--in fact, all my land-based air can be relied upon to launch. What makes the 5th Carrier Division different?--when the primary mission is base attack, rather than naval attack, it launches about 20% of the time. (And when the primary mission is naval attack, it behaves like the other carrier forces.)

[herwin]

Saves are available at <http://scat-he-g4.sunderland.ac.uk/~harryerw/witp/index.html>.

When the orders are Naval Attack, CreateStrikeFlight should probably first check whether it should interrupt the secondary mission for a naval attack, and if that doesn't happen, process the secondary mission as if it were a primary mission. If it is possible to have a naval attack after the secondary mission, CreateStrikeFlight should probably come back and do a second check for the pm naval attack.

Here's a question for String--I think you did some base attacks as a primary mission in our PBEM. Did you see the same effect?

[jwilkerson]

Our preference would be to have a save pack (zip) sent that includes the save and a description of how to execute that save and see what is of interest.

There are actually two "sub-phases" within each airphase at which air units are checked to see if they will fly. This influences the sequencing you see in the game. So an air unit can actually be checked twice in the .am. and then twice again in the .pm.

[DuckofTindalos]

ORIGINAL: rtrapasso

ORIGINAL: Terminus

Your method is obviously correct, from a "mechanical" standpoint, but the data you're putting into the method isn't extensive enough, and this skews the result in the direction you want it to go. Sorry, but I've seen it before.

Were you replying to me, or Herwin? [&:]

Herwin.

[herwin]

ORIGINAL: jwilkerson

Our preference would be to have a save pack (zip) sent that includes the save and a description of how to execute that save and see what is of interest.

There are actually two "sub-phases" within each airphase at which air units are checked to see if they will fly. This influences the sequencing you see in the game. So an air unit can actually be checked twice in the .am. and then twice again in the .pm.

I'll find some time to address this tomorrow.

[herwin]

I now have eight well controlled samples, and there won't be any more as Wake fell.

> wake<-data.frame(attack=c(0,0,0,1,0,1,1,1), mission=c("P","P","P","S","P","S","P","S"))
> attach(wake)
> wake
attack mission
1 0 P
2 0 P
3 0 P
4 1 S
5 0 P
6 1 S
7 1 P
8 1 S

Attack is 1 if the mission flew. P and S describe whether the attack mission was primary or secondary (to naval attack)

> model1<-glm(attack~mission,binomial)
> model0<-glm(attack~1,binomial)
> anova(model1,model0,test="Chi")
Analysis of Deviance Table

Model 1: attack ~ mission
Model 2: attack ~ 1
Resid. Df Resid. Dev Df Deviance P(>|Chi|)
1 6 5.0040
2 7 11.0904 -1 -6.0863 0.0136

This translates into the difference between the hypothesised relationship (Model 1) and the null hypothesis (Model 2) being significant at the 1.36% level.

Have fun!

[rtrapasso]

ORIGINAL: herwin

I now have eight well controlled samples, and there won't be any more as Wake fell.

> wake<-data.frame(attack=c(0,0,0,1,0,1,1,1), mission=c("P","P","P","S","P","S","P","S"))
> attach(wake)
> wake
attack mission
1 0 P
2 0 P
3 0 P
4 1 S
5 0 P
6 1 S
7 1 P
8 1 S

Attack is 1 if the mission flew. P and S describe whether the attack mission was primary or secondary (to naval attack)

> model1<-glm(attack~mission,binomial)
> model0<-glm(attack~1,binomial)
> anova(model1,model0,test="Chi")
Analysis of Deviance Table

Model 1: attack ~ mission
Model 2: attack ~ 1
Resid. Df Resid. Dev Df Deviance P(>|Chi|)
1 6 5.0040
2 7 11.0904 -1 -6.0863 0.0136

This translates into the difference between the hypothesised relationship (Model 1) and the null hypothesis (Model 2) being significant at the 1.36% level.

Have fun!

As i said some time ago, i have not been able to replicate your results from the beginning using stock ... since i am NOT running your mod, i suggest the problem is in your mod.

Good luck.

[herwin]

Wouldn't be completely surprised.