help desk requested post here for developer

[Shadrach]

That's good - I'm pretty sure a memory leak is what's going on here. But it will probably take a long time to get it fixed with the rate of patches we're seeing (about six months since last). In the meantime, not sure what to do to avoid the crashes.

The shot I posted is from the tool "Process Explorer" and you can download it here:
https://docs.microsoft.com/en-us/sysint ... s-explorer
It's basically a "Task Manager on steroids" and allows you to see details of every process in a lot of detail.

On multi-processor: The game is multi-threaded, but by default the game logic itself runs in a single thread on a single processor/core. The audio playback runs in a couple more threads. You will see this in Process Explorer under the "threads" view of the process.

There is actually a flag in "Opart 4.ini", by default false:
multicpu=N
I've tried setting this to 'Y' - and it does spread the main game load over several cores, but it seems to have no noticeable effect on game speed when I've tested it, so I've kept it to 'N'.

Best of luck with support - but don't keep your fingers crossed they will come up with anything soon [:D]

[Lobster]

The scenario did crash towards the end of the turn during combat planning. Computer vs computer. Like watching grass grow or paint dry. Has someone modified this scenario from the original since TOAWIV?
Forgot to say, my system's memory was no where near maxed out.
And yes, fixing stuff is glacial. [:(]

[Shadrach]

Lobster; it's hard to tell from that image how much memory the TOAW process itself was taking up. It's strange that you don't see any increase in memory usage. Maybe the graph just isn't detailed enough, don't know. Try the same with Process Explorer (if you're interested in these things [;)] ), and double-click "Opart 4.exe" when it runs, then open the Performance Graph tab.

Anyway, I did some more digging, since I wanted to find out if my theory of audio playback was causing the leak. And apparently not - just moving units around the map doesn't increase memory much, as I would expect if the leak was there.

However - and this is really interesting - just *moving* the map increases memory by a massive amount!
Especially around the heavily urbanised areas in the north, around Bruxelles - Gelsenkirchen, the game will become really sluggish and memory use will increase in leaps and bounds.

Once memory reaches short of 1.9GB, the game will hang and crash.



Of course, as the player moves around, or during the AI play, the map will move and memory will slowly build.

I tested with a couple of other huge scenarios, D21, FITE2, and saw nothing. Memory stayed a sane 3-400MB even during a full turn of FITE2.

I tried with another similar scenario, "France 1944 D-Day", and the same happens there, as the terrain is the same. So there is something up with how the terrain tiles are modelled in that scenario around that area.

[Lobster]

Yes, I suspected it was the scenario otherwise there would have been many more crashes reported. Another one that someone has modified and causes crashes is here: tm.asp?m=4601852 Next War 1979 - Expanded (Beta 1.0)

I wouldn't be surprised if there are other TOAWIII scenarios that have been 'converted' that also crash the game. Because of the changes in TOAW between 3 and 4 not everything works as it once did. Simply saving an older scenario in the newer version will not guarantee success.

BTW, memory usage did increase about 6% as more combats were planned until the game crashed. But with all of the units and combats that isn't surprising.

I wonder if France 1944 D-Day will crash.

[Shadrach]

France 1944 D-Day will crash, it's pretty much assured. Once memory increases like that and never gets collected, it's only a matter of time.
Well, depends a bit on how much the player moves the map around I guess, and how much AI movement will move the map around as well during its turn.

I tested the "Next War 1979 - Expanded (Beta 1.0)" scenario, the same thing happens there.

Some other observations:
- Doesn't appear to happen in the editor.
- Doesn't appear to happen in the tiny/ultra tiny map sizes.
- Also happens in 3D view, which is weird...

For all I know, it happens on all scenarios, and that's actually more likely, it's just that some scenarios have a bigger concentration of whatever causes the memory to increase...

[fastfrank]

For 3 years I thought I was the only player having this problem. Thought it was specific to my PC configuration. Smart people have said the first step to solving a problem is identifying/validating the problem. Thank you very much.

Although I was a programmer/system analyst in the 60's and early 70's, that was before we even had operating systems. So for now I will download the tool you used, and try to study tactics that release memory and permit play while awaiting a fix, i.e. save & exit game mid-term or mid-round, or when memory consumption nears a TBD threshold.

Thanks again.

[Shadrach]

No worries, happy to have been of some help - I actually like doing digging like this, feels like being a detective [8D]

And hey, computers haven't really changed that much since the 70's. The basic concepts are the same, you have CPU, Memory and I/O. Of course, back then memory was a very expensive resource and programmers learned to stay within very small limits. These days memory is cheap and plentiful and it seems programmers have lost the old skills of memory management. Even the browser tab I have this page open on takes a whopping 43MB of memory - for ONE page [8|]

Let us know if you hear anything at all from support - it will be interesting to know what they say!

[fastfrank]

Ran Turn 1-16 again to gather what information might be useful. This is really "black box analysis" without any tools or insight of the internals. Nonetheless kept Task Manager/Performance open to track available memory as well as Performance Monitor to track private bytes consumption. What I found:
A. Consumed private bytes dropped back to ~300-400MB at start of each player Turn. Growth was slow but steady during Turn.
B. Although consumed private bytes looked like (on the System Monitor graph) to be approaching the ceiling many times, Win10 increased allocation each time. The memory consumption chart here can be deceiving since the vertical axis is apparently computed automatically, and as the size of the allocated private bytes is increased, the entire graph is immediately rescaled. See, e.g. the Allied progression in Turn 16 from 809MB to 827MB, from 966 to 989MB, Perhaps significant is the available memory during Turn 16 only decreased from 11GB to 10.3GB at program crash. Not sure memory leak can be confirmed as cause of the crash.
C. Since 3 of us have now had crashes with the identical scenario, chances of this being caused by a specific PC configuration, considering how many variations are possible, must be most unlikely.
D. When I saved this game run for analysis, I set PO random seed to -0, to eliminate any stochastic variability during the analysis. TOAW IV blew up 7 of 7 times identically before I started the thread. Nonetheless, last night while gathering my data, TOAW completed Turn 16 and continued onwards. [I was tempted to add an exclamation mark to the prior sentence or even one of those omicons, but recovered my senses.] Nonetheless, either this software is subject to the Heisenberg Uncertainty Principle, or we don't fully understand the system design or I've had too much Cabernet, but when software produces different results while repeating then the end of the world is not far off.

[Shadrach]

There's definitely a memory leak if memory keeps increasing during gameplay, with no attempt to clean it up before the memory reaches the 32-bit limit. Like I said, I'm able to make it crash within a couple of minutes by just running the map around.

The memory 'ceiling' shown in Process Explorer is just computed to make the graph look better and it will scale as it increases. The limit of a 32-bit application running under Windows is 2GB, and it apparently triggers a crash even before it hits the limit.

https://docs.microsoft.com/en-gb/window ... s-releases
https://superuser.com/questions/1163749 ... 39#1163939

I made a quick video showing how quick it is:
https://drive.google.com/open?id=1xmYj7 ... P9x37fsrQU
At this point no crash dump was generated, but I've had them generated before.

That being said, you might sometimes get 'lucky' like you said in point D. Maybe there's not enough movement of the map during the turns to make it crash, and at the start of a new turn, the memory is cleared.

And like you said, this is "black box analysis". The only ones who can give an answer at this point is the developer or support, by recreating the crash on their side, obtaining crash dumps and analysing them for the cause of the crash.

At the very least, an acknowledgement from them that "we are aware of the issue" would be good.

[fastfrank]

he is reply from helpdesk

Hi fastfrank,
The game producer confirmed me, that he have sent an email to the game developer asking him to reply in that thread.
Kind Regards,

Paulo Costa
The Slitherine Group

[fastfrank]

I've got a 64-bit PC running Win10 Home and 16GB memory> Same result?

[Shadrach]

Yes, I have the same OS. If it's what I think, I doubt the OS will make any difference... [:)]

[fastfrank]

Just ran another experiment while I await a fix. Ran turn 16 Allied with P/O at a reasonable map scale and tracked Private Bytes. Started at 383.3MB and grew to 1.5GB just before crash. Changed map scale so entire map fit on screen. Map didn't move during play. Started at 351.3MB and never grew. Turn 16 ran to completion. Not sure I always had TOAW4 and MS Project Monitor(?) running simultaneously but clearly consumption of Private Bytes didn't occur on a map scale large enough to fit on display.

Of course this is not a work-around since that map scale only shows red and blue ants running around and no tactical or operational information. Does suggest that map re-centering is consuming some vital memory. Have a vague memory of a switch to invoke automatic map re-centering, but found no reference in the TOAW IV game manual. Perhaps I need to look for a 3M by 4M display.

[cathar1244]

Shadrach,

Thanks for the tip re: Process Explorer.

How does one see how much "private bytes" capacity one's PC has?

Cheers

ETA. Sorry for the ? re: Private bytes. I was just reading up on it, and it looks variable. Messing around with PE, I could see TOAW using 400MB but it would increase the amount if required. The upper limit on my PC may be the same as yours. Definitely something inefficient going on with memory management as you pointed out.

[Shadrach]

ORIGINAL: cathar1244
I was just reading up on it, and it looks variable. Messing around with PE, I could see TOAW using 400MB but it would increase the amount if required. The upper limit on my PC may be the same as yours. Definitely something inefficient going on with memory management as you pointed out.

Basically, like I posted earlier, a 32-bit process, even under a 64-bit OS with plenty of RAM, will always have a hard limit of 2GB, unless a special flag (LARGE_ADDRESS_AWARE) is set. Even then it wouldn't help much for this case, it would still crash (just after a bit more time).

The field being "filled" under the "Private Bytes" in the PE process view is not really relevant, what matters is the total used and how the graph looks.

Figuring out Windows memory management is something akin to Black Magic, but it's certainly interesting to learn about - if you're into that kind of thing [;)]
I am nowhere near an expert on the topic but have worked with figuring out plenty of memory leaks before (in Java), so I certainly know what one looks like [8D]

Here's a good explanation of the main types of memory available to view in PE:
https://stackoverflow.com/a/1986486/202114

In addition, looking at it a bit more today, I think I got a bit closer to the culprit. The cause of the leak seems to be the text overlay on the map. Which makes sense, as places with a lot of urban tiles also have a lot of place names, at least in these specific scenarios. Which kind of leads me in the direction that this might be related to Windows Fonts, and font smoothing/scaling, possibly only in Windows 10. Just a shot in the dark, I've tried turning off both DPI scaling and font aliasing in Windows and it didn't help. But definitely something with the place names. If I turned off "Windows Fonts" in the game for the Huge fonts, the issue was gone, but the fixed-size fonts are basically unreadable...

So a workaround if you have crashes is to turn off place names (Options > Place Names > Invisible). Scrolling the map will be MUCH smoother, and no memory increase. Just hard to remember to do on every turn...[8|]

Frank; this might be something you could update your support case with, that it seems to be related to the place name fonts. Might help the developer nail the leak.

[cathar1244]

Shadrach,

Great comments. Following on your earlier probes, I also scooted around the FITE2 map. Fixed bytes never went much over 260MB or so.

Ditto "Campaign for South Vietnam" -- low fixed bytes usage.

But "Russo-China 2000" (not even a very big map) -- usage really takes off as one scrolls around the map.

I wonder if some fonts are more 'compatible' with TOAW than others.

Cheers

[sPzAbt653]

So a workaround if you have crashes is to turn off place names (Options > Place Names > Invisible). Scrolling the map will be MUCH smoother, and no memory increase. Just hard to remember to do on every turn...

Thanks for suggesting this, hopefully those with similar troubles will see this and find it useful.

[Shadrach]

ORIGINAL: cathar1244
I wonder if some fonts are more 'compatible' with TOAW than others.

Maybe, but I've tried to change font (to Times New Roman/Consolas), removed drop shadow, added background, nothing changed. Possibly it's the "Huge Placename Font", but this is also the one used most often. The fonts are Windows TrueType fonts, and they are all the same, so I doubt a specific font is causing it.

I think the game might be loading fonts into memory every time a place name is scrolled into view, and then it's never reclaimed, apart from on new turn beginning.

I wish I'd find a way to see what the increased memory contains, it would explain a lot, but I don't have the tools to do that, probably need some serious debugger.

At least we have a workaround, but at this point the only ones who can figure this out is the developer.

[fastfrank]

Just had a reply from the help desk, they have asked the "producer" to try and contact the "developer" to review this string and try to debug. Not sure how this all works, not sure if the developer is also the programmer or whether the developer is the designer and will need to then contact the programmer.... I'm trying to keep the help desk in the loop since they are contactable and seem to accept some ownership to move this process forward. Help desk has asked the developer to contact us via this forum string.

[fastfrank]

Just ran Allied turn 16 with place names invisible. Private Bytes started at 344MB end successfully ended the turn at 385MB. If someone occasionally needs place names, hovering the mouse pointer over the hex will show the place name. Congratulations sir and well done.