Page 1 of 4
Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 1:18 pm
by Apollo11
Hi all,
Here is Leo's ASW TEST (and scenario) made from scratch in new WitP v1.70 BETA (i.e. with OOB of v1.60 since after that no OOB changes were done).
I wanted to do this for log time bi only today managed to get some free time to do it...
Download Link:
Leo ASW TEST Scenario #120.zip (10-16-2005)
The ASW TEST scenario is made to explore various tactics and possibilities!
NOTE:
All ships in scenario have their default EXP rating whilst all air units have their EXP at 75 and morale at 75 (you can change those in scenario editor rather easy if you wish).
Possible serious issue with Air ASW search
I did found one rather disturbing fact already...
The ASW air search is
_EXTREMELY_ effective and out of 36 submarines 33 were discovered.
IMHO this is way way way exaggerated and needs to be toned down...
Please look at the following pictures:
Leo "Apollo11"
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 1:35 pm
by rtrapasso
Possible serious issue with Air ASW search
I did found one rather disturbing fact already...
The ASW air search is _EXTREMELY_ effective and out of 36 submarines 33 were discovered.
IMHO this is way way way exaggerated and needs to be toned down...
This has been a problem for some time now (and many a bitter complaint has graced the forum about it) - it did NOT originate with this patch. Alas, AFAIK, they did not say they were going to address the problem in this patch - just how FAST you could sink the subs was issue addressed.
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 1:49 pm
by Apollo11
Hi all,
ORIGINAL: rtrapasso
This has been a problem for some time now (and many a bitter complaint has graced the forum about it) - it did NOT originate with this patch. Alas, AFAIK, they did not say they were going to address the problem in this patch - just how FAST you could sink the subs was issue addressed.
I know... it was set several WitP versions ago (the WitP v1.70 BETA only chnaged ship based ASW)...
But, unfortunately, nobody did comprehensive test to actually see how good/bad Air ASW search is until I just did it!
Leo "Apollo11"
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 2:10 pm
by aletoledo
72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.
perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 2:16 pm
by Apollo11
Hi all,
I just spotted "tiny" mistake in my TEST ASW scenario - all TF slots had "9999" delay set so no new TF were able to create - I fixed this and uploaded the fixed file insetad of old one
Please note that I left the file name and version same (in order not to confuse users more).
So, if you downloaded my scenario in past 60 minutes (since itw as out) please reload - sorry for inconvinience...
Leo "Apollo11"
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 2:22 pm
by Apollo11
Hi all,
ORIGINAL: aletoledo
72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.
perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.
The scenario is available for all so all kind of tests can be made (with all kind of alterations).
BTW, I on purpose set 100% ASW in order to get best possible ASW Air search case.
Also note that 72 aircraft is not much if you look how big area covered with submarines is. We are talking about
_HUGE_ piece of sea - the submarines roughly take 60 (sixty) HEXes of deep sea and each HEX is 60x69 nautical miles!!!
60x60x60 nmi = 216000 square nautical miles!
Leo "Apollo11"
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:10 pm
by jwilkerson
Since July 2004, there have been many comments that subs are way too detectable by air search - thus far M2b3 have chosen other means to address the overall issue of subs sinking too fast - not by making subs less detectable - but by changing the way the attacks work - and I think again in 1.7 this has been the approach. Not sure why search hasn't been addressed but must be because it is thought to be more diffcult to make the change from that direction.
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:17 pm
by Dereck
ORIGINAL: aletoledo
72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.
perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.
True. If you're going to run a test make the test with a reasonable level of planes instead of going overboard like this and other "tests" that you've run. Going overboard you're just ensuring you get off-kilter results.
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:18 pm
by madmickey
Ron is Right
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:27 pm
by rtrapasso
ORIGINAL: madmickey
Ron is Right
[:D][:D]
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:28 pm
by Nomad
ORIGINAL: madmickey
Ron is Right
We know that, now convince Matrix Games and 2x3 of that. [:D]
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:29 pm
by Apollo11
Hi all,
ORIGINAL: dereck
If you're going to run a test make the test with a reasonable level of planes instead of going overboard like this and other "tests" that you've run. Going overboard you're just ensuring you get off-kilter results.
Please explain what you mean under "test" I have made?
What is "overboard" about this and other test I made?
Who else made any other tests worth wile that were available to general WitP population (an what tests
you made so that you can complain about what I did)?
BTW, do you really think that putting 72 aircraft (and out of those 36 are biplanes) on AWS is overkill?
Have you read the AARs (in those number on ASW aircraft can be counted in hundreds)?
Leo "Apollo11"
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:54 pm
by Dereck
Yes I think your testing is flawed. What you're basically doing is overkill and ensuring that if the ASW search routine IS changed it's going to be based on flawed testing and force people in future version to have MORE than your 72 planes on ASW just to be able to spot submarines.
If you want to do a honest and decent test have three levels of searching. A low value, a medium value and a high value - in both number of planes used AND the amount assigned to ASW search. You'd be doing a lot of tests but that way you have a RANGE of spotting instead of a high value which will obviously result in sub spottings.
If, as you say, people in AARs are using hundresds of planes in ASW patrol then YES they're going to spot submarines. That is obvious - all your test did was to show the obvious.
You want to do a serious test vary the test data so if it's the number of planes, the percent assigned to ASW, a combination of both, or an unrelated variable can be determined. It could show that 72 planes at 10% ASW may do just as well as 18 planes at 100% for example. You may also want to run the same tests as both the Japanese and Allied to see if the results are the same or different.
As it is your test is only showing one side and, in my opinion, not objective enough to really be taken seriously since you can have people like me questioning it.
And as far as testing, I've done plenty of program and system testing and test plan writting in my 16 years in IT to know that you have to be objective and plan for a lot of possibilities - you've only planned and tested for one at most.
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 3:58 pm
by jwilkerson
Unfortunately in some of the games I've played you wouldn't get into overboard until you'd gotten up towards 500+ planes !!! ( for better or worse, if we give the players a loop hole - they will exploit it !!! House rules have subsequently fixed in that game )
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 4:09 pm
by Nomad
I don't have a lot of time for testing but since Leo gave us a good tool to use I tried a few tests. I took the bettys off of any search. Put the Mavis on 100% Naval search at 6000 feet. and the two 9 aircraft float planes on 100% ASW at 1000 ft. I think this is pretty standard( it would be for me ). In three runs I got 25, 22, 21 subs detected. So, anyone want to venture a guess on what the detected number should be? [:)]
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 4:19 pm
by Apollo11
Hi all,
ORIGINAL: dereck
Yes I think your testing is flawed. What you're basically doing is overkill and ensuring that if the ASW search routine IS changed it's going to be based on flawed testing and force people in future version to have MORE than your 72 planes on ASW just to be able to spot submarines.
If you want to do a honest and decent test have three levels of searching. A low value, a medium value and a high value - in both number of planes used AND the amount assigned to ASW search. You'd be doing a lot of tests but that way you have a RANGE of spotting instead of a high value which will obviously result in sub spottings.
If, as you say, people in AARs are using hundresds of planes in ASW patrol then YES they're going to spot submarines. That is obvious - all your test did was to show the obvious.
You want to do a serious test vary the test data so if it's the number of planes, the percent assigned to ASW, a combination of both, or an unrelated variable can be determined. It could show that 72 planes at 10% ASW may do just as well as 18 planes at 100% for example. You may also want to run the same tests as both the Japanese and Allied to see if the results are the same or different.
As it is your test is only showing one side and, in my opinion, not objective enough to really be taken seriously since you can have people like me questioning it.
And as far as testing, I've done plenty of program and system testing and test plan writting in my 16 years in IT to know that you have to be objective and plan for a lot of possibilities - you've only planned and tested for one at most.
Have you read what I wrote when I first published this?
I think not..
I asked that WitP community (i.e. all of us who are interested in better WitP) test and check for
_THEMSELVES_!
I
_ALWAYS_ post scenario and
_NOT_ results
_ONLY_ (i.e. where scenario is "hidden" or unavailable so the results can't be verified).
I also wrote that I made one test - more testing can be done by me (when I get time) or by any other interested WitP player who thinks current way of WitP ASW is not ideal.
You can test if you want to help.
BTW, I also have 15+ years in IT and know what testing is - testing (in essence) is:
- controlled environment (this we have - I made scenario that can be tested and tested)
- inclusion of all possible variations and alterations (this we have - I made scenario that can be changed easily)
- need of many runs to establish pattern
So who/what is flawed here?
Leo "Apollo11"
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 4:26 pm
by Dereck
Testing SHOULD be in a controlled environment Leo so you can have results that will hold water. By doing multiple tests using something like the grid below you could show both a pattern and/or trend and the results couldn't be called skewered by people.
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 4:33 pm
by Apollo11
Hi all,
ORIGINAL: dereck
Testing SHOULD be in a controlled environment Leo so you can have results that will hold water. By doing multiple tests using something like the grid below you could show both a pattern and/or trend and the results couldn't be called skewered by people.
I agree 100% - I always agreed (and I always run several test runs with each test being reload of scenario in order to get new "seed" each time - just saving game and reloading save is wrong for testing because you will always have same "seed").
So... who volunteers to help to see what is really going with WitP Air and ASW search?
Leo "Apollo11"
Another discovery (now proven fact)...
Posted: Sun Oct 16, 2005 4:35 pm
by Apollo11
Hi all,
Another discovery (now proven fact)...
In ASW Air search the range is halved (just as it is written in manual).
So if you set range (via dialer) to 6 HEXes you get ASW search to 3 HEXes out!
Leo "Apollo11"
RE: Leo's ASW TEST (and scenario) - problems spotted!
Posted: Sun Oct 16, 2005 4:51 pm
by jwilkerson
ORIGINAL: Nomad
I don't have a lot of time for testing but since Leo gave us a good tool to use I tried a few tests. I took the bettys off of any search. Put the Mavis on 100% Naval search at 6000 feet. and the two 9 aircraft float planes on 100% ASW at 1000 ft. I think this is pretty standard( it would be for me ). In three runs I got 25, 22, 21 subs detected. So, anyone want to venture a guess on what the detected number should be? [:)]
Well if there was a " move to station " mode ( sub operating mostly on the surface in waters believed to be mostly free of enemy air search while in route to patrol area ) ... and a "patrol mode" ... submarine mostly submerged during the day to avoid air search ... and the above searches were with the sub in patrol mode and made without radar .. and during the day ... then the detected number should be some small number between about 0 and 2 ... ( even sub commanders make mistakes and surface sometimes when they shouldn't .. or they have engine trouble ... or they see a juicey target .. or they go aground ... )