Leo's ASW TEST (and scenario) - problems spotted!

Gary Grigsby's strategic level wargame covering the entire War in the Pacific from 1941 to 1945 or beyond.

Moderators: Joel Billings, wdolson, Don Bowen, mogami

User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

Leo's ASW TEST (and scenario) - problems spotted!

Post by Apollo11 »

Hi all,

Here is Leo's ASW TEST (and scenario) made from scratch in new WitP v1.70 BETA (i.e. with OOB of v1.60 since after that no OOB changes were done).

I wanted to do this for log time bi only today managed to get some free time to do it...


Download Link:

Leo ASW TEST Scenario #120.zip (10-16-2005)


The ASW TEST scenario is made to explore various tactics and possibilities!


NOTE:
All ships in scenario have their default EXP rating whilst all air units have their EXP at 75 and morale at 75 (you can change those in scenario editor rather easy if you wish).


Possible serious issue with Air ASW search

I did found one rather disturbing fact already...

The ASW air search is _EXTREMELY_ effective and out of 36 submarines 33 were discovered.

IMHO this is way way way exaggerated and needs to be toned down...


Please look at the following pictures:

Image

Image

Image


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
rtrapasso
Posts: 22655
Joined: Tue Sep 03, 2002 4:31 am

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by rtrapasso »

Possible serious issue with Air ASW search

I did found one rather disturbing fact already...

The ASW air search is _EXTREMELY_ effective and out of 36 submarines 33 were discovered.

IMHO this is way way way exaggerated and needs to be toned down...

This has been a problem for some time now (and many a bitter complaint has graced the forum about it) - it did NOT originate with this patch. Alas, AFAIK, they did not say they were going to address the problem in this patch - just how FAST you could sink the subs was issue addressed.
User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Apollo11 »

Hi all,
ORIGINAL: rtrapasso

This has been a problem for some time now (and many a bitter complaint has graced the forum about it) - it did NOT originate with this patch. Alas, AFAIK, they did not say they were going to address the problem in this patch - just how FAST you could sink the subs was issue addressed.

I know... it was set several WitP versions ago (the WitP v1.70 BETA only chnaged ship based ASW)...


But, unfortunately, nobody did comprehensive test to actually see how good/bad Air ASW search is until I just did it!


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
aletoledo
Posts: 827
Joined: Fri Feb 04, 2005 6:51 pm
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by aletoledo »

72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.

perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.
User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Apollo11 »

Hi all,

I just spotted "tiny" mistake in my TEST ASW scenario - all TF slots had "9999" delay set so no new TF were able to create - I fixed this and uploaded the fixed file insetad of old one

Please note that I left the file name and version same (in order not to confuse users more).

So, if you downloaded my scenario in past 60 minutes (since itw as out) please reload - sorry for inconvinience...


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Apollo11 »

Hi all,
ORIGINAL: aletoledo

72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.

perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.

The scenario is available for all so all kind of tests can be made (with all kind of alterations).


BTW, I on purpose set 100% ASW in order to get best possible ASW Air search case.


Also note that 72 aircraft is not much if you look how big area covered with submarines is. We are talking about _HUGE_ piece of sea - the submarines roughly take 60 (sixty) HEXes of deep sea and each HEX is 60x69 nautical miles!!!

60x60x60 nmi = 216000 square nautical miles!


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
jwilkerson
Posts: 8110
Joined: Sun Sep 15, 2002 4:02 am
Location: Kansas
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by jwilkerson »

Since July 2004, there have been many comments that subs are way too detectable by air search - thus far M2b3 have chosen other means to address the overall issue of subs sinking too fast - not by making subs less detectable - but by changing the way the attacks work - and I think again in 1.7 this has been the approach. Not sure why search hasn't been addressed but must be because it is thought to be more diffcult to make the change from that direction.

AE Project Lead
SCW Project Lead
User avatar
Dereck
Posts: 3127
Joined: Mon Sep 06, 2004 10:43 pm
Location: Romulus, MI

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Dereck »

ORIGINAL: aletoledo

72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.

perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.

True. If you're going to run a test make the test with a reasonable level of planes instead of going overboard like this and other "tests" that you've run. Going overboard you're just ensuring you get off-kilter results.
PO2 US Navy (1980-1986);
USS Midway CV-41 (1981-1984)
Whidbey Island, WA (1984-1986)
Naval Reserve (1986-1992)
madmickey
Posts: 1336
Joined: Wed Feb 11, 2004 6:54 pm
Location: Calgary, Alberta

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by madmickey »

Ron is Right
User avatar
rtrapasso
Posts: 22655
Joined: Tue Sep 03, 2002 4:31 am

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by rtrapasso »

ORIGINAL: madmickey

Ron is Right
[:D][:D]
User avatar
Nomad
Posts: 7273
Joined: Wed Sep 05, 2001 8:00 am
Location: West Yellowstone, Montana

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Nomad »

ORIGINAL: madmickey

Ron is Right

We know that, now convince Matrix Games and 2x3 of that. [:D]
User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Apollo11 »

Hi all,
ORIGINAL: dereck

If you're going to run a test make the test with a reasonable level of planes instead of going overboard like this and other "tests" that you've run. Going overboard you're just ensuring you get off-kilter results.

Please explain what you mean under "test" I have made?

What is "overboard" about this and other test I made?

Who else made any other tests worth wile that were available to general WitP population (an what tests you made so that you can complain about what I did)?


BTW, do you really think that putting 72 aircraft (and out of those 36 are biplanes) on AWS is overkill?

Have you read the AARs (in those number on ASW aircraft can be counted in hundreds)?


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
Dereck
Posts: 3127
Joined: Mon Sep 06, 2004 10:43 pm
Location: Romulus, MI

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Dereck »

Yes I think your testing is flawed. What you're basically doing is overkill and ensuring that if the ASW search routine IS changed it's going to be based on flawed testing and force people in future version to have MORE than your 72 planes on ASW just to be able to spot submarines.

If you want to do a honest and decent test have three levels of searching. A low value, a medium value and a high value - in both number of planes used AND the amount assigned to ASW search. You'd be doing a lot of tests but that way you have a RANGE of spotting instead of a high value which will obviously result in sub spottings.

If, as you say, people in AARs are using hundresds of planes in ASW patrol then YES they're going to spot submarines. That is obvious - all your test did was to show the obvious.

You want to do a serious test vary the test data so if it's the number of planes, the percent assigned to ASW, a combination of both, or an unrelated variable can be determined. It could show that 72 planes at 10% ASW may do just as well as 18 planes at 100% for example. You may also want to run the same tests as both the Japanese and Allied to see if the results are the same or different.

As it is your test is only showing one side and, in my opinion, not objective enough to really be taken seriously since you can have people like me questioning it.

And as far as testing, I've done plenty of program and system testing and test plan writting in my 16 years in IT to know that you have to be objective and plan for a lot of possibilities - you've only planned and tested for one at most.
PO2 US Navy (1980-1986);
USS Midway CV-41 (1981-1984)
Whidbey Island, WA (1984-1986)
Naval Reserve (1986-1992)
User avatar
jwilkerson
Posts: 8110
Joined: Sun Sep 15, 2002 4:02 am
Location: Kansas
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by jwilkerson »

Unfortunately in some of the games I've played you wouldn't get into overboard until you'd gotten up towards 500+ planes !!! ( for better or worse, if we give the players a loop hole - they will exploit it !!! House rules have subsequently fixed in that game )


AE Project Lead
SCW Project Lead
User avatar
Nomad
Posts: 7273
Joined: Wed Sep 05, 2001 8:00 am
Location: West Yellowstone, Montana

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Nomad »

I don't have a lot of time for testing but since Leo gave us a good tool to use I tried a few tests. I took the bettys off of any search. Put the Mavis on 100% Naval search at 6000 feet. and the two 9 aircraft float planes on 100% ASW at 1000 ft. I think this is pretty standard( it would be for me ). In three runs I got 25, 22, 21 subs detected. So, anyone want to venture a guess on what the detected number should be? [:)]
User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Apollo11 »

Hi all,
ORIGINAL: dereck

Yes I think your testing is flawed. What you're basically doing is overkill and ensuring that if the ASW search routine IS changed it's going to be based on flawed testing and force people in future version to have MORE than your 72 planes on ASW just to be able to spot submarines.

If you want to do a honest and decent test have three levels of searching. A low value, a medium value and a high value - in both number of planes used AND the amount assigned to ASW search. You'd be doing a lot of tests but that way you have a RANGE of spotting instead of a high value which will obviously result in sub spottings.

If, as you say, people in AARs are using hundresds of planes in ASW patrol then YES they're going to spot submarines. That is obvious - all your test did was to show the obvious.

You want to do a serious test vary the test data so if it's the number of planes, the percent assigned to ASW, a combination of both, or an unrelated variable can be determined. It could show that 72 planes at 10% ASW may do just as well as 18 planes at 100% for example. You may also want to run the same tests as both the Japanese and Allied to see if the results are the same or different.

As it is your test is only showing one side and, in my opinion, not objective enough to really be taken seriously since you can have people like me questioning it.

And as far as testing, I've done plenty of program and system testing and test plan writting in my 16 years in IT to know that you have to be objective and plan for a lot of possibilities - you've only planned and tested for one at most.

Have you read what I wrote when I first published this?

I think not..


I asked that WitP community (i.e. all of us who are interested in better WitP) test and check for _THEMSELVES_!

I _ALWAYS_ post scenario and _NOT_ results _ONLY_ (i.e. where scenario is "hidden" or unavailable so the results can't be verified).

I also wrote that I made one test - more testing can be done by me (when I get time) or by any other interested WitP player who thinks current way of WitP ASW is not ideal.

You can test if you want to help.


BTW, I also have 15+ years in IT and know what testing is - testing (in essence) is:

- controlled environment (this we have - I made scenario that can be tested and tested)
- inclusion of all possible variations and alterations (this we have - I made scenario that can be changed easily)
- need of many runs to establish pattern


So who/what is flawed here?


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
Dereck
Posts: 3127
Joined: Mon Sep 06, 2004 10:43 pm
Location: Romulus, MI

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Dereck »

Testing SHOULD be in a controlled environment Leo so you can have results that will hold water. By doing multiple tests using something like the grid below you could show both a pattern and/or trend and the results couldn't be called skewered by people.



Image
Attachments
ASWTestSearchGrid.jpg
ASWTestSearchGrid.jpg (20.68 KiB) Viewed 615 times
PO2 US Navy (1980-1986);
USS Midway CV-41 (1981-1984)
Whidbey Island, WA (1984-1986)
Naval Reserve (1986-1992)
User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by Apollo11 »

Hi all,
ORIGINAL: dereck

Testing SHOULD be in a controlled environment Leo so you can have results that will hold water. By doing multiple tests using something like the grid below you could show both a pattern and/or trend and the results couldn't be called skewered by people.

Image

I agree 100% - I always agreed (and I always run several test runs with each test being reload of scenario in order to get new "seed" each time - just saving game and reloading save is wrong for testing because you will always have same "seed").


So... who volunteers to help to see what is really going with WitP Air and ASW search?


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
Apollo11
Posts: 25218
Joined: Thu Jun 07, 2001 8:00 am
Location: Zagreb, Croatia
Contact:

Another discovery (now proven fact)...

Post by Apollo11 »

Hi all,

Another discovery (now proven fact)...

In ASW Air search the range is halved (just as it is written in manual).

So if you set range (via dialer) to 6 HEXes you get ASW search to 3 HEXes out!


Leo "Apollo11"
Image

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE
User avatar
jwilkerson
Posts: 8110
Joined: Sun Sep 15, 2002 4:02 am
Location: Kansas
Contact:

RE: Leo's ASW TEST (and scenario) - problems spotted!

Post by jwilkerson »

ORIGINAL: Nomad

I don't have a lot of time for testing but since Leo gave us a good tool to use I tried a few tests. I took the bettys off of any search. Put the Mavis on 100% Naval search at 6000 feet. and the two 9 aircraft float planes on 100% ASW at 1000 ft. I think this is pretty standard( it would be for me ). In three runs I got 25, 22, 21 subs detected. So, anyone want to venture a guess on what the detected number should be? [:)]

Well if there was a " move to station " mode ( sub operating mostly on the surface in waters believed to be mostly free of enemy air search while in route to patrol area ) ... and a "patrol mode" ... submarine mostly submerged during the day to avoid air search ... and the above searches were with the sub in patrol mode and made without radar .. and during the day ... then the detected number should be some small number between about 0 and 2 ... ( even sub commanders make mistakes and surface sometimes when they shouldn't .. or they have engine trouble ... or they see a juicey target .. or they go aground ... )

AE Project Lead
SCW Project Lead
Post Reply

Return to “War In The Pacific - Struggle Against Japan 1941 - 1945”