Page 76 of 325
RE: need to shrink DB
Posted: Tue Jun 15, 2010 4:18 pm
by floydg
ORIGINAL: herwin
ORIGINAL: floydg
ORIGINAL: herwin
That's a standard problem with JDBC systems. You need to bug the developers to allow people to use alternative high-performance DBMSs.
Okay, I'm game - if it can improve performance, can you recommend something?
There are two problems--big databases are unavoidably slower, and the memory fragmentation you get in the JVM as you load a lot of data means you start thrashing. There was this FAA system I had some involvement with that ran into just that problem. I've asked the DBMS expert who sits two desks over for some suggestions.
For a starter, is the database normalised?
I believe the database is in 3NF.
The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.
Floyd
RE: WitPTracker AE Release 1.0
Posted: Wed Jun 16, 2010 3:25 am
by n01487477
ORIGINAL: n01487477
ORIGINAL: jcjordan
I minor thing I noticed on the alert tab was that in my game as Allies vs AI IJ that on the new month I get the msgs about a/c research being moved forward to x month....
Let me get this right ... you're seeing your opponents R&D on the first of the month ?
Oops ... I think I wrote this bit of code I forgot to filter for side ... found, done and fixed
RE: need to shrink DB
Posted: Wed Jun 16, 2010 5:29 am
by herwin
ORIGINAL: floydg
I believe the database is in 3NF.
The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.
Floyd
What's probably killing the user is memory fragmentation during the load. You might look at advanced memory management techniques. Unfortunately, Java (and C#) aren't really designed for user control of memory usage. Do a Google search on Java memory fragmentation, and you'll find a number of articles on the subject. My colleague may get back to me later today.
RE: need to shrink DB
Posted: Wed Jun 16, 2010 11:28 am
by n01487477
ORIGINAL: herwin
ORIGINAL: floydg
I believe the database is in 3NF.
The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.
Floyd
What's probably killing the user is memory fragmentation during the load. You might look at advanced memory management techniques. Unfortunately, Java (and C#) aren't really designed for user control of memory usage. Do a Google search on Java memory fragmentation, and you'll find a number of articles on the subject. My colleague may get back to me later today.
Just spent a little bit of time doing some stats on Tracker ... jconsole.exe (JDK) -heap dump-> Mem Analyser(eclipse). Gave me what I thought in the first place, the map is a memory hog ! This was with about 10 turns loaded, I'll compare with 100+
RE: need to shrink DB
Posted: Wed Jun 16, 2010 2:49 pm
by Zebedee
Sent an e-mail to the techsupport address. Just can't get 64-bit Win 7 on this new AMD rig to recognise 32-bit Java
RE: need to shrink DB
Posted: Wed Jun 16, 2010 11:16 pm
by floydg
ORIGINAL: n01487477
ORIGINAL: herwin
ORIGINAL: floydg
I believe the database is in 3NF.
The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.
Floyd
What's probably killing the user is memory fragmentation during the load. You might look at advanced memory management techniques. Unfortunately, Java (and C#) aren't really designed for user control of memory usage. Do a Google search on Java memory fragmentation, and you'll find a number of articles on the subject. My colleague may get back to me later today.
Just spent a little bit of time doing some stats on Tracker ... jconsole.exe (JDK) -heap dump-> Mem Analyser(eclipse). Gave me what I thought in the first place, the map is a memory hog ! This was with about 10 turns loaded, I'll compare with 100+
Next release will have the ability to hide the map. It is a huge image read in...
RE: need to shrink DB
Posted: Thu Jun 17, 2010 4:20 am
by herwin
Yes, but the map is fixed size--it shouldn't be a big issue. You could consider using a proxy and only loading it as needed, but then you might run into out of memory problems later.
Replays Question
Posted: Thu Jun 17, 2010 6:34 am
by herwin
I've offered my opponent a 'mulligan' back about 5 days. I assume I need to reload all the .pws files up to that date...
RE: Replays Question
Posted: Thu Jun 17, 2010 8:42 pm
by floydg
ORIGINAL: herwin
I've offered my opponent a 'mulligan' back about 5 days. I assume I need to reload all the .pws files up to that date...
Yeah, no way to "unload" turns.
RE: need to shrink DB
Posted: Thu Jun 17, 2010 8:44 pm
by floydg
ORIGINAL: herwin
Yes, but the map is fixed size--it shouldn't be a big issue. You could consider using a proxy and only loading it as needed, but then you might run into out of memory problems later.
The loading takes a huge amount of time in addition to huge amount of memory. I'd need to profile it a little deeper to see where the bottleneck is, but for now, as I said, the option to turn it off will be available.
RE: need to shrink DB
Posted: Thu Jun 17, 2010 9:08 pm
by witpqs
Floyd, personally I've stopped using the map and just use the 'Z' key in-game. Having the search arcs on the map would be nice, but (reported a while back) they don't work right. Having the Z key, while not as good an overview as Tracker, is good enough. So an option to turn off the map would be great.
To recap, the problem with the search arcs is this (made up example to illustrate the key point):
Take a group with 9 planes and put it on 100% search. Tell it to search from 0d to 180d. Do 'show arcs' and you will see a morning search for half of the search (say 0d to 90d) and an afternoon search for the other half (say 90d to 180d).
Instead tell it to search 0d to 90d. Do 'show arcs' and you will see that 0d to 90d is searched both morning and afternoon.
Tracker - in both cases - will show a morning search for half and an afternoon search for half. In other words, it seems like Tracker just takes the starting place (0d), takes the number of planes set to search (planes and percentage set to search), and then counts out morning and afternoon searches. The ending place (90d) is not recognized.
RE: need to shrink DB
Posted: Thu Jun 17, 2010 9:26 pm
by floydg
ORIGINAL: witpqs
Floyd, personally I've stopped using the map and just use the 'Z' key in-game. Having the search arcs on the map would be nice, but (reported a while back) they don't work right. Having the Z key, while not as good an overview as Tracker, is good enough. So an option to turn off the map would be great.
To recap, the problem with the search arcs is this (made up example to illustrate the key point):
Take a group with 9 planes and put it on 100% search. Tell it to search from 0d to 180d. Do 'show arcs' and you will see a morning search for half of the search (say 0d to 90d) and an afternoon search for the other half (say 90d to 180d).
Instead tell it to search 0d to 90d. Do 'show arcs' and you will see that 0d to 90d is searched both morning and afternoon.
Tracker - in both cases - will show a morning search for half and an afternoon search for half. In other words, it seems like Tracker just takes the starting place (0d), takes the number of planes set to search (planes and percentage set to search), and then counts out morning and afternoon searches. The ending place (90d) is not recognized.
First time I've heard about this bug. But I see where in the code I've gone wrong: not checking the end position and just blowing through it.
Floyd
RE: need to shrink DB
Posted: Fri Jun 18, 2010 6:17 pm
by Zebedee
Just wanted to thank Floyd and the WitPtracker team for helping me get up and running.
Nice one
Base data
Posted: Sat Jun 19, 2010 2:58 pm
by Roger Neilson II
Have just tried an export of the Bases material - very useful potentially, but whilst it has supply data there is no fuel data? Oe of the key things i was looking for?
Roger
RE: Base data
Posted: Sat Jun 19, 2010 4:13 pm
by floydg
ORIGINAL: Roger Neilson II
Have just tried an export of the Bases material - very useful potentially, but whilst it has supply data there is no fuel data? Oe of the key things i was looking for?
Roger
The export function only exports what's shown in the table. I can add fuel to the table so next release you'll be able to carefully count your barrels.
Floyd
RE: Base data
Posted: Sat Jun 19, 2010 4:35 pm
by Roger Neilson II
That would be really useful.
Roger
RE: need to shrink DB
Posted: Sat Jun 19, 2010 6:07 pm
by witpqs
ORIGINAL: floydg
First time I've heard about this bug. But I see where in the code I've gone wrong: not checking the end position and just blowing through it.
Floyd
Thanks!
RE: need to shrink DB
Posted: Sat Jun 19, 2010 8:17 pm
by vonTirpitz
version 1.5.1
I have observed a minor nuisance that occurs when trying to load the Industry screen after some of the other screens have been viewed. For example, if I load and review the events, check out the victory screen and perhaps the Intel screen then go to Industry it won't display. I usually have to select another screen such as mines or whatnot and then the Industry screen again. It generally displays properly when selected after that. Kind of feels like perhaps a flag isn't being initialized set or cleared but just guessing.
Great utility though. Saves a player a lot of time trying to figure out what is where.... Thanks for all the hard work!
Marc
RE: need to shrink DB
Posted: Sun Jun 20, 2010 1:11 am
by erstad
version 1.5.1
Very minor issue, but might be easy to address if it seems worthwhile.
On the LCU screen, when filtering for "Lack supply", all the LCU fragments loaded on ships show up, because the LCU drops its supply when it loads.
Since all loaded LCUs have no supply, and there's a lot of fragments, seems like it would be better to only list the LCUs on the ground.
RE: need to shrink DB
Posted: Mon Jun 21, 2010 12:26 am
by floydg
ORIGINAL: erstad
version 1.5.1
Very minor issue, but might be easy to address if it seems worthwhile.
On the LCU screen, when filtering for "Lack supply", all the LCU fragments loaded on ships show up, because the LCU drops its supply when it loads.
Since all loaded LCUs have no supply, and there's a lot of fragments, seems like it would be better to only list the LCUs on the ground.
Fixed.