Matrix Games Forums

ORIGINAL: wwengr

Five replications is enough to establish statistical significance.

5 replications can only establish significance for normally distributed data that is randomly sampled. I do not believe that either criteria is met here.

Most of the variables in the game are single sided distributions, not normal distributions.

Thought I read somewhere that the game engine was designed to give you identical results, to preclude "rolling for best result". To disturb the outcome, you have to make changes in the inputs, but these are not really random even though they give differeing results.

Many thanks, excellent list.

Test runs require making a new scenario for the test and starting a new game for each run. It is not sufficient to re-run a saved turn. As I understand it, the game applies many combat modifiers using random number generators. Given the multiple parameters, my guess would be that the distribution would be normal if it is principally summed functions or log-normal if combat results are multiplicative products of these independent randomized variables. There are different tests for different probability distributions and sample sizes, but... I've already admitted that my test was weak... I made unverified assumptions about the probability distribution. Even then, I simply failed ot reject the null hypothesis.

Thank you!  You just saved me a ton of work and questions.
 

ORIGINAL: wwengr

Test runs require making a new scenario for the test and starting a new game for each run.

That should address the randomness, at least, as I understand what Gary initially created for the combat rolls.

As for the non-normal distribution, I'm not sure I agree. Most of the variables are bounded on at least one side.
Example: max damage is to destroy the ship. It cannot also destroy a second ship. Minimum is a miss. These are hard boundaries that are not assymptotic. There are a lot of misses. Most other variable appear similar on at least one side and occasionally both as exampled.

Agreed. The only way to get to a really reliable result is to do a whole lot of runs. 5 x 5 like in my little test says that it is most likely random variation, but leaves a lot of room for it to just be a low probability result.

ORIGINAL: wwengr

Agreed. The only way to get to a really reliable result is to do a whole lot of runs. 5 x 5 like in my little test says that it is most likely random variation, but leaves a lot of room for it to just be a low probability result.

Agreed. And a lot of runs could be a LOT of runs. Almost everything we observe is multi-variant, and that just takes the sample size up and up.

For these leadership variables, unless one of the devs shares some more insight, I don't think we are going to get very far on our own. I think of the large variance in any battle outcome .. just too much involved. How many times have you run 7 Dec and just looked at PH results? Or the Force Z result? Huge variation. Separating the leadership effects out the battle result? What, there must be +20 independent variables involved in just a surface combat result. I don't even want to look at my charts for sample size to attain 90% confidence level. Scary sample size.

Which gets back to our original question to the devs: can we at least get some directional relationship definition of the leadership variables to game situations? We do not need quantitative, just qualitative statements. Confirm or rebut the response to the OP would be great.

I was reviewing the leader statistics for ASW task forces:

Are you sure that aggression has no influence on the performance of an ASW Task Force?
IIRC, ASW TFs react to detected subs the same way as a surface combat TF reacts to detected surface TFs, so I would suspect that aggression does matter?

Cheers,

Moss

Shouldn't the naval value be always also considered for finding submarines? I guess that is not only important for an ASW, Transport/Replenish & Escort TF.

Also is each test made on the values that the AG leader has?
Check my notes in brackets:
***
Pilot experience affects operational losses. (Sounds to me like each pilots value is important, not leader only)
Air groups with morale < 50 must pass a morale test before flying an offensive mission. If the test is failed, no aircraft will fly. (And for this the AGs overall morale is used I guess)
Air groups flying Naval Search, ASW Patrol, or CAP must pass two morale tests before flying. Each morale test failed reduces the number of aircraft flying by 25%. (Same as before)
Level Bombers must pass three tests before flying an offensive mission. Each test failed reduces the number of aircraft flying by 25%:

Experience test. (I guess again on overall experience level, not leader himself)
Test against the leader’s Air Skill.
Morale test. (I guess here overall morale is tested)

Pilot experience affects the chances to find the target in a strike mission. (Sounds to me like each pilots value is important, not leader only)
Pilot experience affects air-to-air combat results. (Sounds to me like each pilots value is important, not leader only)
Leader’s air skill affects results in air-to-air combat.
***
So what of this is really tested on the leader only?

ORIGINAL: PaxMondo
For these leadership variables, unless one of the devs shares some more insight, I don't think we are going to get very far on our own.

I think you folks have gotten very far on your own. wwengr has defined the space very clearly. And Mike Wood, although not on the AE team, has very significant insights into the game system. He is as much a dev as anyone and his input should be put in your 'book'.

You guys are trying to make this way too hard. It's really very simple. So far as Naval stuff goes, if you look at wwengr's post, it's pretty clear, if you have any kind of "Surface" TF (and that includes subs), then pick a leader with good Naval stats. If you have any kind of "Air" TF, then pick a leader with good Air stats. If you have a mixed TF with CVs and BBs, then decide which is more important and go with that kind of leader but pick one with good secondary skills for the secondary ships. It's really that simple.

The sidebar crap doesn't matter much in the combat environment. Aggression helps mostly in responding to detection and reaction thingys. It does not help for airplanes, it does not help for gunnery, it only helps for things that require a response to something; is the dude aggressive? will he "hie to the sound of the hounds"? It's really that simple.

Admin, inspiration, leadership, work within sub, and sub/sub, routines that probably aren't of interest to you folks anyway, since they have little to no impact on detect, react, or combat, so I'll leave them alone.

wwengr's stuff, and an intelligent application thereof, is the best you all will ever get. I trust you all will use it wisely.

Wow, this is something I've wanted more info on for some time. Very informative and I see I have some checking to do.

I have one question. In ground combat what exactly is determining a (+) or (-) leadership modifier for each force. Is it the skill or leadership rating of the commander that determines this. Should I be picking a leader that has a higher land skill rating than their leadership rating for troops in combat? Just what factors actually determine the (+) or (-) modifier? I tend to pick high land skill, yet in a lot of combat my forces suffer a (-) leadership modifier...so I'm questioning whether I should be picking the higher leadership rating?

BUMP!

Good to know its LEADERSHIP that effects pilot training, not INSPIRATION which I've been using... Time to spend another 100 PPs reassigning air unit leaders. [:D]

Well, is anyone willing to update it to AE based on experinces/patches ? Or it didnt changed at all from original Witp ?

Bump a bumpy bump!

ORIGINAL: wwengr

ORIGINAL: Smeulders

No offense, but 5 replications seems a bit low to really draw conclusions from. Not that this result would really surprise me, I'm guessing whether or not the DD spot the sub is of a much greater importance.

You are both correct and incorrect. I won't go into the math, as it can be fairly tedious for all but mathematicians, engineers, and statisticians. Five replications is enough to establish statistical significance ...

AKA a trend?

Thanks very much for the info. I'm just wondering the empirical basis of this list. I can't understand why the system provides two streams for decision, if they are not required. The first is the "leadership - inspiration" one. The second is the "skill - aggression" one. I don't understand why the designers set this up if the second is irrelevant. The list here says to ignore the paradigm established by the designers, but go with a "skill - inspiration" balance instead. Now this makes intuitive sense, but begs the question, on what basis are the recommendations made?

Think of this thread, and more so the discussion following the first post, as a baseline for your tests. But make sure you remain aware of two things:

1) The list at the beginning is an analysis of WitP, not WitP AE data.

2) The list is incomplete (in a skill - consequence relational sense), and contains errors.

The ensuing discussion started by the first post (in here and in other threads) was more important than the fist post. A significant conclusion is, that considering all variables influencing results, 5 iterations is not enough repetition for establishing statistical significance as long as certain (variable) attributes about the environment remain unknown.

wwengr´s, PaxMondo´s, and John´s post are the most important in here. Use as reference.

I've been using this list as a guide for a long time and very useful it has been.&nbsp; I do have one query regarding the leadership rating being used as the main attribute for most ships.&nbsp;&nbsp; Naval should be higher priority than leadership.&nbsp; What are others thoughts on this?

ORIGINAL: Chris H

I've been using this list as a guide for a long time and very useful it has been.  I do have one query regarding the leadership rating being used as the main attribute for most ships.   Naval should be higher priority than leadership.  What are others thoughts on this?

Hi Chris

I have lots of opinions but no statistical data to back them. In a multi-variate analysis sense I would like to know the principle component and the amount of variance that could be assigned to that component.

Cheers
Guy