I understand the scenario data (eg CVO) comes in the files labelled "wp?070.dat". These files are in some kind of hex format.
The tool witpload is supposed to convert this format to .csv, which is fine for putting the data into spreadsheets or using other tools (eg witpdecoder). I did the extraction of scenario 70 with witpload and tried to work a bit on the data.
I understand that there are lots of slots with special (hardcoded) functions etc, but nevertheless I believe it should be possible to get the raw unit data correct in spreadsheets. In the example above I noticed that many dates in the 'delay' field, which controls the date of reinforcement, is plain wrong. Meaning its different from the data I see in the game. For instance some units appeared to be already available but weren't, others just have a wrong date set.
Question is why it is wrong in the extracted spreadsheets. It could be that witpload is generally not compliant with RHS scenario files due to technical issues. Fine, but then there must be another way to extract the data correctly from the scenario files into spreadsheets. I'm not talking about processing the data further with other tools (eg witpdecoder), I'd be happy if I could get correct spreadsheet data.
Am I right to assume that the data the RHS team puts into the scenario files comes from spreadsheets in the first place? If so, maybe if you could make those files available to the community as they would provide a big help for playing?
Scenario data extraction
Moderators: wdolson, Don Bowen, mogami
Scenario data extraction
If you gained knowledge through the forum, why not putting it into the AE wiki?
http://witp-ae.wikia.com/wiki/War_in_th ... ition_Wiki
http://witp-ae.wikia.com/wiki/War_in_th ... ition_Wiki
-
el cid again
- Posts: 16983
- Joined: Mon Oct 10, 2005 4:40 pm
RE: Scenario data extraction
I don't think what you are seeing is RHS specific. That is, I think the identical things should be present in all forms of WITP - if you dumped them in the same way. It is very unusual for us to use hidden fields - and those few exceptions are documented (and generally something Matrix revealed - such as the code for an airborne unit - or the code that calls a cavalry symbol).
Amazing as it may seem, there are few tools even at Matrix - I am told "we use essentially the same tools you do." And RHS is NOT developed on special spreadsheets - but on Matrix databases - which are spreadsheets in a way.
However, we DO convert the data FROM the Matrix spreadsheets to MS Excel format - and sometimes we make that data available for players to use. For example, the Economic Utilities - which actually are just the first turn dump for any given scenario - and we can build a similar page for any later turn. There is, however, a security problem with doing this using tools other than published dump utilities - and we don't do stuff unpopular with Matrix - which after all is the hand that feeds us and the code owner (owners have rights to decide what is acceptable use practice).
It is not clear what you are looking at - or what is bothering you? If you could be very technical and specific, we can probably decode it - and possibly advise ways to get wherever you want to go? We also are at a stage where I plan to cause a number of utilities to be built - and we are working on installer/pwhex modification utilities right now. [We because I am not doing it myself - RHS truly is a team] But understand that WITP is not documented, that the original designers are not around, and that more than a few things are undefined, and not understood by anyone. One programmer wrote "after a year I thought I understood how this function worked; then I saw a line that completely changed my understanding, and I am no longer sure what else there may be that might affect this?" We may NOT be able to say more than "something in the code".
Amazing as it may seem, there are few tools even at Matrix - I am told "we use essentially the same tools you do." And RHS is NOT developed on special spreadsheets - but on Matrix databases - which are spreadsheets in a way.
However, we DO convert the data FROM the Matrix spreadsheets to MS Excel format - and sometimes we make that data available for players to use. For example, the Economic Utilities - which actually are just the first turn dump for any given scenario - and we can build a similar page for any later turn. There is, however, a security problem with doing this using tools other than published dump utilities - and we don't do stuff unpopular with Matrix - which after all is the hand that feeds us and the code owner (owners have rights to decide what is acceptable use practice).
It is not clear what you are looking at - or what is bothering you? If you could be very technical and specific, we can probably decode it - and possibly advise ways to get wherever you want to go? We also are at a stage where I plan to cause a number of utilities to be built - and we are working on installer/pwhex modification utilities right now. [We because I am not doing it myself - RHS truly is a team] But understand that WITP is not documented, that the original designers are not around, and that more than a few things are undefined, and not understood by anyone. One programmer wrote "after a year I thought I understood how this function worked; then I saw a line that completely changed my understanding, and I am no longer sure what else there may be that might affect this?" We may NOT be able to say more than "something in the code".
RE: Scenario data extraction
El Cid, I totally understand that some tools for security reason or company policy or other reasons are not published. It's also clear that you're working/interfacing with some black magic here (aka code), which clearly causes many problems and inaccuracies.
Nevertheless the issue remains that witpload for instance is not able to handle the delay field correctly (among other information). Obviously it would be very nice to have the date for reinforcements on a handy sheet but this is by now not possible.
You wrote 'published dump utilities', this indicates there are others I'm not aware of. Could you please let me know which ones you know of or even better use and where I can get them?
What I'm trying to achieve is not an option to obtain data from an ongoing game (aka security issue), I'd just like to have a dump of the first turn. This dump should include all the information I can get when I fire up a game and write everything off the screen. Again I can live with some inaccuracies, missing data or such, but this should be the exception not the rule as it is atm in the case of RHS and witpload.
EDIT:
Seems I found a case where a unit is in the game but not in the extracted files of witpload. Might be related to the fact that another unit appears twice in the game...
In CVO (scen70) there are two units with the same name. The name is "BA 77th LAA Regiment", both units apper Apr/15/42 in Karachi. One has the Twin .50in AAMG, the other has the 40mm Bofors in its TOE. They appear in the witpload dump as unit number 2496 and 2405.
The "BA 77th HAA Regiment" (HAA not LAA!), appearing mid january/42 in Aden is missing in the dump completely.
Nevertheless the issue remains that witpload for instance is not able to handle the delay field correctly (among other information). Obviously it would be very nice to have the date for reinforcements on a handy sheet but this is by now not possible.
You wrote 'published dump utilities', this indicates there are others I'm not aware of. Could you please let me know which ones you know of or even better use and where I can get them?
What I'm trying to achieve is not an option to obtain data from an ongoing game (aka security issue), I'd just like to have a dump of the first turn. This dump should include all the information I can get when I fire up a game and write everything off the screen. Again I can live with some inaccuracies, missing data or such, but this should be the exception not the rule as it is atm in the case of RHS and witpload.
EDIT:
Seems I found a case where a unit is in the game but not in the extracted files of witpload. Might be related to the fact that another unit appears twice in the game...
In CVO (scen70) there are two units with the same name. The name is "BA 77th LAA Regiment", both units apper Apr/15/42 in Karachi. One has the Twin .50in AAMG, the other has the 40mm Bofors in its TOE. They appear in the witpload dump as unit number 2496 and 2405.
The "BA 77th HAA Regiment" (HAA not LAA!), appearing mid january/42 in Aden is missing in the dump completely.
If you gained knowledge through the forum, why not putting it into the AE wiki?
http://witp-ae.wikia.com/wiki/War_in_th ... ition_Wiki
http://witp-ae.wikia.com/wiki/War_in_th ... ition_Wiki
-
el cid again
- Posts: 16983
- Joined: Mon Oct 10, 2005 4:40 pm
RE: Scenario data extraction
That is an inheritance from CHS - where there are two 77th LAA Regiments. We should kill one of them. In RHS it should be 2405 - since there is no such thing as a LAA regiment with .50 cals. This does not explain one being missing however - both should appear.
I can publish any start of game dump you like - and sometimes I do this in the form of a WITPExcel upload. You should be able to see scenario start data in Editor X, Matrix editor 6.0, WITPExcel, using WITP Dump, and probably other formats. But I can dump other fields if you need them. Thing is - if you cannot now see them - how do you know they are there and you want to see them?
I can publish any start of game dump you like - and sometimes I do this in the form of a WITPExcel upload. You should be able to see scenario start data in Editor X, Matrix editor 6.0, WITPExcel, using WITP Dump, and probably other formats. But I can dump other fields if you need them. Thing is - if you cannot now see them - how do you know they are there and you want to see them?
-
jwilkerson
- Posts: 8253
- Joined: Sun Sep 15, 2002 4:02 am
- Location: Kansas
- Contact:
RE: Scenario data extraction
ORIGINAL: Mistmatz
El Cid, I totally understand that some tools for security reason or company policy or other reasons are not published. It's also clear that you're working/interfacing with some black magic here (aka code), which clearly causes many problems and inaccuracies.
Nevertheless the issue remains that witpload for instance is not able to handle the delay field correctly (among other information). Obviously it would be very nice to have the date for reinforcements on a handy sheet but this is by now not possible.
You wrote 'published dump utilities', this indicates there are others I'm not aware of. Could you please let me know which ones you know of or even better use and where I can get them?
What I'm trying to achieve is not an option to obtain data from an ongoing game (aka security issue), I'd just like to have a dump of the first turn. This dump should include all the information I can get when I fire up a game and write everything off the screen. Again I can live with some inaccuracies, missing data or such, but this should be the exception not the rule as it is atm in the case of RHS and witpload.
EDIT:
Seems I found a case where a unit is in the game but not in the extracted files of witpload. Might be related to the fact that another unit appears twice in the game...
In CVO (scen70) there are two units with the same name. The name is "BA 77th LAA Regiment", both units apper Apr/15/42 in Karachi. One has the Twin .50in AAMG, the other has the 40mm Bofors in its TOE. They appear in the witpload dump as unit number 2496 and 2405.
The "BA 77th HAA Regiment" (HAA not LAA!), appearing mid january/42 in Aden is missing in the dump completely.
As far as scenario file editing utilities ... I know about witpload.exe ... this was written by MichaelM years ago and has been generally available (like on Spookys) for years. I use it exclusively.
As far as save file editing ... I use HexEdit ... a freeware product ... I'm sure there are many equivalents ... but this one is probably as good as any ...
So I there are no "exclusive" editors for any of this stuff ... no "matrix" or "2 y 3" specific editors ... they are no such things ... we do all use the same editors .. and we have used these for years.
WITP Admiral's Edition - Project Lead
War In Spain - Project Lead
War In Spain - Project Lead
RE: Scenario data extraction
Silly question: are you playing with "variable reinforcements"?ORIGINAL: Mistmatz
In the example above I noticed that many dates in the 'delay' field, which controls the date of reinforcement, is plain wrong. Meaning its different from the data I see in the game.
RE: Scenario data extraction
As the Japanese player in the game in question I can tell you that no we are not playing with variable reinforcements.ORIGINAL: VSWG
Silly question: are you playing with "variable reinforcements"?ORIGINAL: Mistmatz
In the example above I noticed that many dates in the 'delay' field, which controls the date of reinforcement, is plain wrong. Meaning its different from the data I see in the game.
"Square peg, round hole? No problem. Malet please.
RE: Scenario data extraction
Good news! WitPExcel seems to extract the data correctly. I only checked some devices and delay dates and they all look good so far. So the problem seems to be related to WitPload which is unfortunately a prerequisite for the WiTPdecoder. Maybe I can find a workaround here.
Nevertheless having an Excel sheet available (40MB!) is the most important thing for planning the war IMO. I'm glad I, as a user, can generate this now.
Nevertheless having an Excel sheet available (40MB!) is the most important thing for planning the war IMO. I'm glad I, as a user, can generate this now.
If you gained knowledge through the forum, why not putting it into the AE wiki?
http://witp-ae.wikia.com/wiki/War_in_th ... ition_Wiki
http://witp-ae.wikia.com/wiki/War_in_th ... ition_Wiki
RE: Scenario data extraction
ORIGINAL: jwilkerson
As far as save file editing ... I use HexEdit ... a freeware product ... I'm sure there are many equivalents ... but this one is probably as good as any ...
Hey, Joe - which HexEdit do you use? It seems there are several out there by the same name.
Intel Monkey: https://sites.google.com/view/staffmonkeys/home
RE: Scenario data extraction
ORIGINAL: witpqs
ORIGINAL: jwilkerson
As far as save file editing ... I use HexEdit ... a freeware product ... I'm sure there are many equivalents ... but this one is probably as good as any ...
Hey, Joe - which HexEdit do you use? It seems there are several out there by the same name.
Bump. [8D]
Intel Monkey: https://sites.google.com/view/staffmonkeys/home
RE: Scenario data extraction
ORIGINAL: witpqs
ORIGINAL: witpqs
ORIGINAL: jwilkerson
As far as save file editing ... I use HexEdit ... a freeware product ... I'm sure there are many equivalents ... but this one is probably as good as any ...
Hey, Joe - which HexEdit do you use? It seems there are several out there by the same name.
Bump. [8D]
Bump. [8D]
Intel Monkey: https://sites.google.com/view/staffmonkeys/home
RE: Scenario data extraction
ORIGINAL: witpqs
ORIGINAL: jwilkerson
As far as save file editing ... I use HexEdit ... a freeware product ... I'm sure there are many equivalents ... but this one is probably as good as any ...
Hey, Joe - which HexEdit do you use? It seems there are several out there by the same name.
I can't speak for Joe, only myself. I do things exactly the same way, and use HexEdit v. 1.03 from Ohio State. It's pretty good and it's freeware. You can get it from:
http://www.physics.ohio-state.edu/~prewett/hexedit/
Ciao.