Introducing Intel Monkey (Another New Version: 0.4.1b)

[witpqs]

OK. In essence "getting something useful out there" has been my approach, both from a delivery standpoint and to get more (and better) information on what things should really look like. You've given me some very useful feedback on how battles information is being used (even prior to IM, of course), which in turn instructs me in how to store, process, and present battles information.

The other piece is more along the lines of 'what to do first?' In terms of the basic functionality we have on the plate:

- Improving the capture of LCU information from the combat report. Currently all LCUs involved in ground combat are captured. But LCUs that are the recipients of naval or air bombardments are not captured, nor are those noted in amphibious landings, etc. Only ground combat (whether bombardment, deliberate, or shock attacks). This could require significant attention, depending on what is required to actually solve the problem. Might be a bit of a quagmire.

- Implementing the capture and display of Troop information from the combat report.

- Implementing the capture and display of Aircraft information from the combat report.

- Implementing the capture and display of information from the operations report.

- Implementing the capture and display of information from the combat events report.

- Implementing a much improved, and specific, Ground Battles tab (or window), including:
* Ability to filter on locations.
* Ability to filter on dates.
* Ability to show groups of battles.
* Ability to show summaries of key data (Unadjusted Assault value, Adjusted Assault Value, Troops, Guns, casualties, etc.).
* Ability to show calculations (ratio of Unadjusted AV to Adjusted AV, ratios, given the current battle's odds and AV ratios - what the next Odds result would look like if all factors were the same, etc.)

- A natural complement of Ground Battles tab would be a Naval Battles tab.

The first item, additional LCU data, is somewhat speculative. The next four, troop and aircraft data from the combat report, the operations report, and the combat events report, will certainly provide less return than the items implemented so far. The Ground Battles tab could be the first one to start providing some advanced capabilities of both reporting and analysis.

Opinions? Advice?

[RogerJNeilson]

I'm going to chuck a mighty, 'what if' into the discusion.

The game, because its a game and is nice and neat formats all the information in different 'information streams'. You have ops reports, sigint reports, combat reports etc etc.

In reality in HQ there's somewhere a file or index card with the various fragments being added all the time. Some will be hard evidence, some will be sigint, some will be FOW stuff. But the game keeps this stuff separate. I am doing a 'fun' job of pulling some info from Combat Reports, but that does not capture the sigint or the ops stuff.

In essence all activity happens somewhere to something, at some time. So the best tool is one that allows the combination of all the information together in one big database that then can be searched/filtered according to:

Where it happens
When it happens
What participates
What happened (less essential)

So if I am interested in lets say Koepang, I want to know the 'dossier' on Koepang which will involve any combat, and bombardments, any landings, any flights, any air intercepts, and sigint, etc etc. Some intel guy will bring that to my desk in HQ..... I won't have to go and ask for five different files....

So the best tool is one that takes all the information and stick it all into a growing database that is searchable as above.

There may be a danger in developing IntelMonkey as a whole series of interesting information streams that don't easily give the big picture....

Just MHO.... lets have other's views.

Roger

[RogerJNeilson]

Only you know how much time and effort any of this is taking you. It might be worth playing with the ground one and seeing what level of detail can be extracted, if that's the way forward, and the lessons learned from that, and the feedback would then make the implementation of other aspects simpler and more 'shaped' by user feedback.

In essence develop one aspect as far as you choose and see how useful it is, a narrow thrust rather than a broad Front assault.

Roger

[guytipton41]

Hi Witpqs,

1st let me say how grateful I am that you have built this tool. I went out and bought a book on Java just so I could start analyzing the intel reports. An then a week later Intel Monkey comes out.. sweet!

So I'm just trying it for the 1st time today. I was surprised to see that the 'E' boat "astern Army" was spending all of it's time in Tokyo.

1942-03-04 E astern Army - - is located at Tokyo 114,60
1942-04-03 E astern Army - - is located at Tokyo 114,60
1942-04-16 E astern Army - - is located at Tokyo 114,60
1942-05-05 E astern Army - - is located at Tokyo 114,60
1942-05-21 E astern Army - - is located at Tokyo 114,60
1942-05-30 E astern Army - - is located at Tokyo 114,60
1942-06-04 E astern Army - - is located at Tokyo 114,60
1942-07-06 E astern Army - - is located at Tokyo 114,60

Must be the Navy equivalent of a hanger queen. I guess any boat that only goes backward and is operated by the Army is an oxymoron.

Cheers,
Guy javascript:void(AddText('[:D]'))

[witpqs]

ORIGINAL: guytipton41

Hi Witpqs,

1st let me say how grateful I am that you have built this tool. I went out and bought a book on Java just so I could start analyzing the intel reports. An then a week later Intel Monkey comes out.. sweet!

So I'm just trying it for the 1st time today. I was surprised to see that the 'E' boat "astern Army" was spending all of it's time in Tokyo.

1942-03-04 E astern Army - - is located at Tokyo 114,60
1942-04-03 E astern Army - - is located at Tokyo 114,60
1942-04-16 E astern Army - - is located at Tokyo 114,60
1942-05-05 E astern Army - - is located at Tokyo 114,60
1942-05-21 E astern Army - - is located at Tokyo 114,60
1942-05-30 E astern Army - - is located at Tokyo 114,60
1942-06-04 E astern Army - - is located at Tokyo 114,60
1942-07-06 E astern Army - - is located at Tokyo 114,60

Must be the Navy equivalent of a hanger queen. I guess any boat that only goes backward and is operated by the Army is an oxymoron.

Cheers,
Guy javascript:void(AddText('[:D]'))

Good job! I just noticed that myself and was on the way here to post it.

Second one, also in SigInt: If the 'actor' is "Radio call sign of CV Akagi" or any ship, that is being missed as a ship. This SigInt message is in a different format from the others, with what I interpret as the 'action', namely "Radio call sign of" being at the beginning and then followed by the actor. I'll have to code to catch it farther upstream when the records are first parsed.

Please post any others you find! [:)]

Will fix up both of those.

[witpqs]

OK, I think I have both of them fixed. I just want to scout around and see if in doing so I broke anything else.

[witpqs]

Look at the previous post. See the first line?

The actor is "a W-17 class DMS is".

The rest of it - moving to, Georgetown, hex # - is fine. But the actor part is rough.

What's the best way to handle that? My first thought is:
- Catch the ship type (DMS) and put it in the Type column.
- Make the Actor column blank so it will sort differently than specific named ships.
- Put the class in the Optional #1 column (in this example as "W-17 class").

What do you think?

[witpqs]

Here is another issue, one that I need help with. Does anyone have samples of similar messages received by the Japanese player?

[guytipton41]

ORIGINAL: witpqs

Look at the previous post. See the first line?

The actor is "a W-17 class DMS is".

The rest of it - moving to, Georgetown, hex # - is fine. But the actor part is rough.

What's the best way to handle that? My first thought is:
- Catch the ship type (DMS) and put it in the Type column.
- Make the Actor column blank so it will sort differently than specific named ships.
- Put the class in the Optional #1 column (in this example as "W-17 class").

What do you think?

Good Evening,

In my humble opinion the output should be like:

Date DMS W-17 Class is - - moving to Georgetown

Which would involve parsing the "W-17 Class" as the actor. All that class would sort together and superb human eye would do the rest of the correlation required.

Cheers,
Guy

[witpqs]

ORIGINAL: guytipton41

ORIGINAL: witpqs

Look at the previous post. See the first line?

The actor is "a W-17 class DMS is".

The rest of it - moving to, Georgetown, hex # - is fine. But the actor part is rough.

What's the best way to handle that? My first thought is:
- Catch the ship type (DMS) and put it in the Type column.
- Make the Actor column blank so it will sort differently than specific named ships.
- Put the class in the Optional #1 column (in this example as "W-17 class").

What do you think?

Good Evening,

In my humble opinion the output should be like:

Date DMS W-17 Class is - - moving to Georgetown

Which would involve parsing the "W-17 Class" as the actor. All that class would sort together and superb human eye would do the rest of the correlation required.

Cheers,
Guy

Extending that logic to the next post, "a Japanese xAK" would be Type "xAK", Actor "Japanese". Make sense?

[witpqs]

How does this one look:

[quertice]

very interesting tool [&o]

[guytipton41]

ORIGINAL: witpqs

How does this one look:

<snip>

I think that is the best that can be done parsing the intel strings. In fact it looks real good. Thanx again!

Cheers,
Guy

[guytipton41]

Good Morning Witpqs,

Downloaded the most recent version at 0932 CDT. I extracted over the top of the previous install.

1942-03-04 E astern Army - - is located at Tokyo 114,60
1942-02-03 E iko-wan Fortress - - is located at Genzan 105,48

On this record it would be useful to recognize that there are two actors, double the record and parse it both ways.

1942-03-28 - 14th Shipping Engineer Regiment is loaded on xAK Hokumai Maru moving to Port Moresby -

For this record the 2nd actor is a generic so there would be no need to double the records. HOWEVER it would be very difficult to encode a set of rules that would reliably tell the difference between a generic and actual actor. Since having actor/actor records doubled would be very handy it is most likely worth the small cost of doubling the actor/generic records.

1942-06-11 - 16th Recon Regiment is loaded on a Japanese xAK moving to Manila -

This bunch of records calls out for a more sophisticated parsing but the correct approach is not obvious.

1942-02-15 - a Aden Cargo class xAK is - - moving to Ominato 119,54
1942-05-10 - a Ansyu xPB class PB is - - moving to Truk 112,108
1942-01-03 - a CHa-1 class SC is - - moving to Babeldaob 90,97
1942-05-12 - a Daigen Cargo class xAKL is - - moving to Osaka/Kyoto 109,59
1942-06-12 - a Japanese AM - - is located at Lunga 114,138
1942-04-25 - a Japanese AP - - is located at Niigata 114,57
1942-01-12 - a Japanese CM is - - moving to Nagasaki/Sasebo 102,58
1942-02-12 - a Japanese LB is - - moving to Lae 99,126
1942-01-11 - a Japanese PG is - - moving to Keijo 103,50
1942-04-30 - a Japanese SC is - - moving to Singapore 50,84
1942-03-17 - a Japanese SS is - - moving to Kwajalein Island 132,115
1942-05-26 - a Japanese SS is - - moving to Kwajalein Island 132,115
1942-06-16 - a Japanese SS is - - moving to Amchitka Island 158,52
1942-06-21 - a Japanese xAK is - - moving to Normanton 86,138
1942-01-17 - a Japanese xAKL is - - moving to Osaka/Kyoto 109,59
1942-01-29 - a Japanese xAKL is - - moving to Foochow 86,60
1941-12-21 - a Japanese xAP - - is located at Takao 84,65
1942-02-26 - a Manzyu TM class TK is - - moving to Takao 84,65
1942-02-27 - a Shokaku class CV is - - moving to Sabang 44,70
1942-07-06 - a Std-F Cargo class xAKL is - - moving to Osaka/Kyoto 109,59
1941-12-22 - a To'su Wa-1 class AMc is - - moving to Truk 112,108
1941-12-10 - a Type A1 class SS is - - moving to 64,99
1942-03-14 - a Type C1 class SS is - - moving to Rabaul 106,125
1942-06-12 - a Type L4 class SS - - is located at Efate 120,154
1942-04-18 - a Yusen A Cargo class xAK is - - moving to Hirosaki/Aomori 118,54

Perhaps this:

1942-02-15 AK Aden Cargo class is - - moving to Ominato 119,54
1942-05-10 PB Ansyu xPB class is - - moving to Truk 112,108
1942-01-03 SC CHa-1 class is - - moving to Babeldaob 90,97
1942-05-12 xAKL Daigen Cargo class is - - moving to Osaka/Kyoto 109,59
1942-06-12 AM Japanese - - is located at Lunga 114,138
1942-04-25 AP Japanese - - is located at Niigata 114,57
1942-02-26 TK Manzyu TM class is - - moving to Takao 84,65
1942-02-27 CV Shokaku class is - - moving to Sabang 44,70

Cheers,
Guy

[witpqs]

Oh wait - the screenshot was a work in progress, not the release! Sorry for being unclear. I'll get it in a bit with a post.

ORIGINAL: guytipton41

Good Morning Witpqs,

Downloaded the most recent version at 0932 CDT. I extracted over the top of the previous install.

1942-03-04 E astern Army - - is located at Tokyo 114,60
1942-02-03 E iko-wan Fortress - - is located at Genzan 105,48

On this record it would be useful to recognize that there are two actors, double the record and parse it both ways.

1942-03-28 - 14th Shipping Engineer Regiment is loaded on xAK Hokumai Maru moving to Port Moresby -

For this record the 2nd actor is a generic so there would be no need to double the records. HOWEVER it would be very difficult to encode a set of rules that would reliably tell the difference between a generic and actual actor. Since having actor/actor records doubled would be very handy it is most likely worth the small cost of doubling the actor/generic records.

1942-06-11 - 16th Recon Regiment is loaded on a Japanese xAK moving to Manila -

This bunch of records calls out for a more sophisticated parsing but the correct approach is not obvious.

1942-02-15 - a Aden Cargo class xAK is - - moving to Ominato 119,54
1942-05-10 - a Ansyu xPB class PB is - - moving to Truk 112,108
1942-01-03 - a CHa-1 class SC is - - moving to Babeldaob 90,97
1942-05-12 - a Daigen Cargo class xAKL is - - moving to Osaka/Kyoto 109,59
1942-06-12 - a Japanese AM - - is located at Lunga 114,138
1942-04-25 - a Japanese AP - - is located at Niigata 114,57
1942-01-12 - a Japanese CM is - - moving to Nagasaki/Sasebo 102,58
1942-02-12 - a Japanese LB is - - moving to Lae 99,126
1942-01-11 - a Japanese PG is - - moving to Keijo 103,50
1942-04-30 - a Japanese SC is - - moving to Singapore 50,84
1942-03-17 - a Japanese SS is - - moving to Kwajalein Island 132,115
1942-05-26 - a Japanese SS is - - moving to Kwajalein Island 132,115
1942-06-16 - a Japanese SS is - - moving to Amchitka Island 158,52
1942-06-21 - a Japanese xAK is - - moving to Normanton 86,138
1942-01-17 - a Japanese xAKL is - - moving to Osaka/Kyoto 109,59
1942-01-29 - a Japanese xAKL is - - moving to Foochow 86,60
1941-12-21 - a Japanese xAP - - is located at Takao 84,65
1942-02-26 - a Manzyu TM class TK is - - moving to Takao 84,65
1942-02-27 - a Shokaku class CV is - - moving to Sabang 44,70
1942-07-06 - a Std-F Cargo class xAKL is - - moving to Osaka/Kyoto 109,59
1941-12-22 - a To'su Wa-1 class AMc is - - moving to Truk 112,108
1941-12-10 - a Type A1 class SS is - - moving to 64,99
1942-03-14 - a Type C1 class SS is - - moving to Rabaul 106,125
1942-06-12 - a Type L4 class SS - - is located at Efate 120,154
1942-04-18 - a Yusen A Cargo class xAK is - - moving to Hirosaki/Aomori 118,54

Perhaps this:

1942-02-15 AK Aden Cargo class is - - moving to Ominato 119,54
1942-05-10 PB Ansyu xPB class is - - moving to Truk 112,108
1942-01-03 SC CHa-1 class is - - moving to Babeldaob 90,97
1942-05-12 xAKL Daigen Cargo class is - - moving to Osaka/Kyoto 109,59
1942-06-12 AM Japanese - - is located at Lunga 114,138
1942-04-25 AP Japanese - - is located at Niigata 114,57
1942-02-26 TK Manzyu TM class is - - moving to Takao 84,65
1942-02-27 CV Shokaku class is - - moving to Sabang 44,70

Cheers,
Guy

[witpqs]

OK, here you go. Intel Monkey V0.4.1b


Link: https://docs.google.com/open?id=0ByVleYydeDVcZTJuWDhMVUU5OVE

Release Notes:


V0.4.1b

SigInt:
- Added that a ' ' has to follow a ship type at the front of Actor. 'E' at the beginning of LCU names was being picked up as a ship type.
- Added logic to handle 'Radio call sign of CV Akagi...' (for example).
- Added 'is moving to' as a parsing term as the term 'moving to' was leaving ' is' as part of ship names.
- Added logic for 'a xxx class xxx' as the Actor.
- Added logic for 'a Japanese xxx' and 'a(n) Allied xxx' as the Actor.

[bk19@mweb.co.za]

HI witpqs,

You may be pleased to know that I have just installed Intel Monkey and run it without any problem under "Python 3.2.2 Stackless 3.1b3 (32bit)".

Initial processing details for your info are:


E:\Extended Disk\Matrix Games\War in the Pacific (Admirals Edition)\tools\Intel Monkey>IntelMonkey-J.py

289 Days of SigInt Processed.
1,715 SigInt Records Processed.

290 Combat Days Processed.
5,936 Battles Processed.
10,095 Individual Combat Records Processed.

6,587 LCU Combat Records Processed.
3,508 Ship Combat Records Processed.
5,936 Combat Blocks Recorded.

I think that should be enough of a stress test to demonstrate your code currently behaves normally with the Stackless Python variant.

Nicely done!

[RogerJNeilson]

Any further developments yet?

Roger

[USSAmerica]

This tool looks very handy, witpqs! Thanks for your time and efforts! I look forward to trying it out soon. [8D]

[Numdydar]

Doing my part and bumping [:)]