How To: Save a table or nested tables to keys

All discussions & material related to Command's Lua interface

Moderators: RoryAndersonCDT, michaelm75au, angster, MOD_Command

Post Reply
KnightHawk75
Posts: 1850
Joined: Thu Nov 15, 2018 7:24 pm

How To: Save a table or nested tables to keys

Post by KnightHawk75 »

So you have some custom data, or global variables you want to save.
But by default you can't save them all to single key right? You have to save each one to separate key, and in doing you also lose the data type and have to always remember to restore the type when loading it back from the key store. Bummer right? Sometimes limiting or causing you to do things you would rather not to cram things into different keys with like incremental name sometime and stuff right?

Well I just wanted to remind everyone no, you don't have to do that. You can 'stringify' tables, and restore them just how they were including keeping simple data types, and nested tables. The two attached scene's are just a simple..or maybe not so simple demos depending on your lua-fu, that show how to do it using an included library gKH.State.

Lua knowledge needed: Medium to Low
- You need to know what a table is and how to use and access the basics of them.
- You need to understand the basic difference between global vars and functions and local ones.
- You need to have a general understanding of how CMO events\triggers\actions operate.
- You need to have some concept of load-order in terms of assigning scripts 'on scenario load' so that you understand where\when to re-load your data or other code on startup (ie conceptually know hey script b depends on script a having run first type thing).
- You need to be aware that CMOINSTALLLFOLDER\Lua\Lua_History_thedate.txt logs exist for help with troubleshooting.

What do I need to do to use this?
All you really need to know to use it is:
- Copy paste either the Events\LuaAction named "Load_gKH_State_library", OR the contents of the also include gKH_State_Standalone.lua file into a CMO Lua Action similarly named in your own scene (they are identical).
- Then setup a scenario loaded trigger if one does not exist yet, and an Event that uses said trigger and points to the lua script action. Alternatively if you already have a scenario loaded startup type, just add the new action to the list, ideally listed first, unless you know what your doing.

Now you might ask, can I just add this to existing code that I already have for loading a bunch of global functions and stuff at sceneload, or via some attachment, or via RunScript and the file? If you know what you're doing, sure, you absolutely can. That said I know what I'm doing, and I still keep each of my modules separate as it's far easier to debug and make changes.

What are the commands?
There are two, one to save a table and one to load a table, they are simple in and of themselves.
-If you want to save a table to a key anywhere in your code call the follow global functions that are available:

Code: Select all

-- first, parameter is the table variable, second is the KeyStore name you want to use, third is optional no-log option, true disables some logging which is on by default. 
-- returns true or false to indicate success or a failure.
gKH.State.SaveTableToKey(theTableVariableHere,"theNameOfTheKeyStoreKeyHere", [ true | false ]);
-If you want to load a table that has been stringified\serialized to a key back to a table call:

Code: Select all

-- first, parameter is the KeyStore name you want to load the table from, second is an optional no-log option, true disables some logging which is on by default. 
-- returns either a table with your data or nil upon error, or simply that the key you asked for doesn't exist yet.
yourTableVarName = gKH.State.LoadTableFromKey("theNameOfTheKeyStoreKeyHere",[ true | false ]);
Yup that's it. But for those that want a walk though of some examples in two different scenes, using both console and gui based testing, examples I threw together the following. It's probably overkill and too verbose and written while I'm very tired but it's what your getting for now. :D

Scene 1 (Sample_StringifyTableToKeys.scen) -
Sample_StringifyTableToKeys.zip
(15.98 KiB) Downloaded 32 times
(just the .scen file)

This is just a demo scene to work with to show simple save and restore of some gKH.SceneGlobals stored table data that stores some config values and an extra sub table of data to play around with, and an environment for the examples below.
It could be your data stored in your namespace or tables, this just uses my own standard one.
All data is stored into one KeyStore key named "MySceneGlobals". How original I know ... come on it's a sample ... I actually use "gKHSceneGlobals" or "gKH_SceneGlobals" normally, but I think that would have been more confusing to follow in this first example. :)

Lets get to playing around with the scene loaded in the editor to see things via the console:
-Load up your game and enter the editor, either via Create Scene followed by load, or just Edit Scene and select the file to load.
- Don't press play yet, just open you're Lua console, and ideally turn off "echo input" via the checkbox.
- First lets print the gKH.SceneGlobals table just to see what if anything is there. (part0):

Code: Select all

print(gKH.SceneGlobals);
- Ok we see the table has some data, we see there is no "gKH.SceneGlobals .Table1, .SomeStringVar, or .SomeNumericVar yet. Probably all you need to take note of at the moment.

- Next lets add some data that does not exist yet by entering the following and running it (part1):

Code: Select all

--populate the a new Table1 entry with a stored table and some entry data.
gKH.SceneGlobals.Table1 = {[1]={name="something",guid="some-thing",num=111.11},[2]={name="something 2",guid="soooom-thing",num=222.22}};
--create some new vars with values in addition to what exists
gKH.SceneGlobals.SomeStringVar="something else";
gKH.SceneGlobals.SomeNumericVar=123.456;
--save all the changes you just did.
gKH.State.SaveTableToKey(gKH.SceneGlobals,'MySceneGlobals'); --stringify and save the above table state.
print(gKH.SceneGlobals);  -- reprint the whole table to the console again.
- Now lets make some changes but not save them, just so you can see the values are changed. (part2):

Code: Select all

--make some changes but don't save them...
gKH.SceneGlobals.Table1[2].name= "blablabla"; --edit the second entry's name field in Table1
gKH.SceneGlobals.Table1[2].num= 333.0; --edit the second entries num field in Table1
gKH.SceneGlobals.SomeStringVar= "doodly doodly do"; -- change the value
gKH.SceneGlobals.SomeNumericVar= 976.54; -- change the value
print(gKH.SceneGlobals); -- see and note the changes.
- Now lets restore from the key store what you saved during part 1. (part3)

Code: Select all

--reload what was originally saved atm.
gKH.SceneGlobals = gKH.State.LoadTableFromKey('MySceneGlobals'); --load saved table and de-stringify
print("SomeStringVar: " .. gKH.SceneGlobals.SomeStringVar .. "  type:" .. type(gKH.SceneGlobals.SomeStringVar));
print("SomeNumericVar: " ..gKH.SceneGlobals.SomeNumericVar.. "  type:" .. type(gKH.SceneGlobals.SomeNumericVar));
print("Table1 type: " .. type(gKH.SceneGlobals.Table1) .. "  table data:");
print(gKH.SceneGlobals.Table1);  --note the preserved typed inside the table data as well.
- Note the preserved types ie strings have "" around them numbers do not etc.
- Next do a "file\save AS " on the scene and call it ...well anything but the original name! Lets say "Sample_Strinify_WithMyChanges.scen"
- Next exit the game and reload game. (ok you don't have too exit the game, it's just to prove nothing is still in ram, because CMO does not clean up the Lua environment between re-loads of scenes, but you can skip this if you want)
- Next reload the file you just saved (it should be in your recent list).
- Next goto the Lua console and run print( gKH.SceneGlobals ). See all your last saved data was restored on scene load, Table1's data is there, and your added vars.


Now lets move on to showing samples of some of this data being changed and resaved on the fly.
- RELOAD the orginal scen file. (ie not the one you saved.)
- Open the Special Actions window.
- Press the "Counter Example" like 3-4 times, enter whatever you want for each time. After each entry a special message will popup showing you the contents of data that has been added to the gKH.SceneGlobals.MyTableData table that previously existed but defaulted to empty. Close the special message window after each press.
- Next from the Special Actions menu select "Show ElapsedSeconds". You should see a popup message saying "0".
- Next press play, and let time advance ~5 seconds.
- Now from the Special Actions menu select "Show ElapsedSeconds". You should get a popup message box telling you how many seconds have elapsed in game-time in total. This is done by the event and triggers that run every game second which updates the counter and re-saves the whole gKH.SceneGlobals table each time.
- Next go ahead and "file\save as" the present state of things as "Sample_Stringify_Counters.scene"
- Now go ahead and reload the file you just saved.
- Go back to Special Actions and run "Show ElapsedSeconds" again, you should see whatever the last value was before the save, "5" if say it was 5 seconds. Again just shows your data was tracked between sessions, no biggy, normal key saves would do that, but no needing to write tonumber() or tostring() a bunch of times converting data, and we're saving and restoring all data in one call. That few lines of code you can find in the Lua Action named "Startup_GlobalVars".

- Note the value add here where a table object gKH.SceneGlobals contains itself, a table of objects in MyDataTable and our Table1 sample as well, so nested tables saved and restored, a factor has been quite enabling. There are caveats with saving the tables though that I'll discuss later, but the TLDR is, you can't save ones that have functions or "userdata" types, attached, just plain strings,boolean,numeric|integer,nil,arrays and tables,nesting of all those is fine.


Scene 2 (Sample_StringifyTableToKeys-ContactNotes.scen) -
Sample_StringifyTableToKeys_ctnotes.zip
(37.85 KiB) Downloaded 37 times
(just the .scen file)

This is the same as the first, except I added in addition a little module (we can't actually use true lua modules in non-professional, but that's what I'm going to call it anyway) that I have for storing notes for contacts, it is at least sort of useful and it shows off basically the same things we've already covered but we're not going to be in the console, we're just going to use the added menu items in the Special Actions menu and be selecting contacts in the gui. Instead of gKH.SceneGlobals, for this scene we're going to be focused on changing and storing additional data in gKH.ContactNotes and really gKH.ContactNotes.Data mostly as we're saving just the .Data table to a KeyStore key named "gKHContactNotes". All the code for which is executed on scene load via the Lua script action "Startup_ContactNotes". It would be a good idea to eye ball it. Lets get started.

- Load the scene ( select side 'Test' ), and make sure you are in unit view (hotkey "v" to change back and forth, bottom left of the screen shows the view status).
- Note there are some existing contacts on the map, please select the contacts named "Bravo__Runway1 and Civilian #11". (shift key is used to select multiple units).
- Open up the Special Action menu window, and press "Show contact notes for selected contacts". You should get a special message window showing the contact name, it's side if known, and the contents of the "note" that the note taker has associated with the selected contact(s). These are all pulled from the data table gKH.ContactNotes.Data that already has been restored. I believe Civilian #11's note should read "Doug Masters", yup totally original. Now close the special message window.
- Now Press "Show all stored contact notes". You should get a special message window dumping the full list of contacts for which the note take has valid records and associations. Note that by default the scene has been setup so that upon a detection trigger for a new contact for side "Test" it will add a blank entry to the data for that new contact. This is why you will see some entries exist and have "None." as text for the "note".

- Now with the same two contacts selected, press "Add Notes to selected contacts". You should receive an input box asking for the note you would like to apply to the given contact, and if it has an existing note, it will also display it. Yes, it sucks we can't pre-fill it with a value, so to change things you have to retype stuff again. Go ahead and enter something different for each and press ok, you will get a confirmation that your additions were made. Should you decide to cancel on one it's fine, you will get told that one will be skipped, and it will move on to the next if any.
- Now save the game file as "ThisShouldShowRunway1Civ11Changed.scen".
- Now reload the just saved game file, and ope the Special Actions again.
- Now invoke the "Show all stored contacts" special action. You should see that your changes were saved and restored at startup. (They also are scavanged and validated automatically at startup, but more on that later.)

- Now select contact "Civilian #12" and "Civilian #13". from the contacts menu select "drop contact(s)" or right click each contact and drop them individually.
- In the Special Actions menu select "show all stored contacts" again. You should see that "civilian #12", and "civilian #13" entries still exist. Yup, just cause you dropped a contact doesn't mean your datastore or code knows that. Thus enters the scavenger process which runs by default about every 2 minutes, but is configurable via the gKH.ContactNotes.ScavengerCycletime configuration field\property.
- Press play and let the simulation run ~15-20 seconds, then pause. (you are waiting for the deleted contacts to get re-detected)
- In the Special Actions menu select "show all stored contacts" again. You should see that there are new entries for "Bogey XX", and of course you still see the dated "civilian" entries.
- Press play and let the simulation run ~2 minutes, then pause, you can use like 5x if you want.
- In the Special Actions menu select "show all stored contacts" again. You should see that "civilian #12", and "civilian #13" entries have been cleaned up, and probably by now the contacts have changed from "Bogey XX" to "civilian XX" because of classification updates. The data at this point should probably reflect that if further classification happened. The data probably should show the most up to date names now as well, if they don't give it a two or 3 minutes more of playtime, they should by that time (1307.3 much slower to classify at range).

- What happened? There is a "Scavenger" event and code that is setup by default to run every minute and check if it is time to run itself, if it is time or past time, all existing ContactNotes.Data table entries are checked if they still have matching live contacts still, if they do then their most current name and side (if known) is updated within the .Data record entry. If an entry's stored contact guid is no longer found to exist, then that record is deleted. We could take additional steps to do the same when ever there are different classification changes by adding more triggers and events, which would help keep data insync at a faster rate, but we don't cause it's a simple example.
- Go ahead and select some contacts, and use the Special Actions menu command "Add Notes to selected contacts" to add or update a few entries. Just vaguely remember what you change.
- Now go ahead and "file\save as" the scene as "NoCivilian1213_MyAdditions.scen" , we're going to check that our changes persist across saves again.
- You guessed it, now reload the same file you just saved again. Then go to the Special Actions menu command "Show all stored contacts" yet again. You should see all your notes and changes are restored to their last state. If you don't you did something wrong or skipped a step along the way maybe?
- You are welcome at this point to screw around as much as you like adding, deleting, droping,forcing instant scavenging, the other available Special Actions for ContactNotes not discussed do what they hint and the descriptions say they do. :)
- I encourage you to eyeball the code for the special events, and event lua actions, you'll see 95% of the code points back to all the global function in the action "Startup_ContactNotes", basically everything is in one place.

Ok we're technically done with the second walk though. But if I didn't drone on some more it wouldn't be me. :D so get a fresh cup of coffee if you want to keep reading.

Why did I maybe waste your time with the scene 2 example walk through?
I assume at least a slim majority, might want the sort of walk through. One that sort of shows things being used in a somewhat real way, vs 'ok here is abstractly what you could do and say 'just read the code'. Also I think this has value showing examples of some of the things used and things to consider when storing data that is desired to persist. In this case where the need doesn't have to be totally in-sync with the game but is close enough for the context (instant or near instant not needed,overhead/perf considerations vs gain, little ramification of out of sync data,etc).

-------
Caveats on using gKH.State.SaveTableToKey( ) and gKH.State.LoadTableFromKey( )
-------
It saves strings, numerics,integers,booleans,nils, arrays and tables, and if present nested sub strings,numerics,booleans,nils, arrays,and tables ..etc..etc. It does NOT SAVE table metadata info, functions,"userdata", or other types. For instance if you 'getunit()' a unit, and then you store that unit into a table in memory, you can keep using that as a wrapper\object, but you can't save it that way. If for example you attempted to stringify it you will get "Unjsonifiable type" errors because it's of "userdata" type. The same is true in our second example if we tried to save "gKH.ContactNotes" as a whole instead of just the .Data sub table, because the gKH.ContactNotes namespace (table) contains our functions too. But if you were say storing a unit's .sensor table into one of your own tables or fields, that would be fine because what you get back from unitwrapper.sensors is a table, not a "userdata" object. You dig? consider the following:

Code: Select all

table1
          .AnotherTableWithJustData={[1]={entryid=1,somevar="sdfasdf"}};
          .somesetting = 0;
          .someflag = true; 
          .AnotherTableWithJustData2={[1]={entryid=1,somevar="sdfasdf"},[2]={entryid=2,somevar="klklklkl",mysubtable={[1]={entryid=1}}}};

table2
          .AnotherTableWithJustData={[1]={entryid=1,somevar="sdfasdf"}};
          .somesetting = 0;
          .someflag = true; 
          :somefunction() = print('hello world.'); end OR .somefunction() = print('hello world.'); end
          .AnotherTableWithJustData2={[1]={entryid=1,somevar="sdfasdf"},[2]={entryid=2,somevar="klklklkl",mysubtable={[1]={entryid=1}}}};
table1 - Can be saved in it's entirety to a key and restored using top level table1,as it's all raw data, and all it's nested tables all only containing the same.
table2 - Can NOT be saved to a key using table2 , it contains functions, however you could separately save and restore .AnotherTableWithJustData1 and 2, to separate respective keystore keys, this is equally true of table1 by the way if you wanted to do it for performance reasons or segregation vs actually having too.

For example if I wanted to make what are now hard coded default values that must be changed in the code for certain ContactNotes.xyz configuration settings, I would probably move all the settings to a ContactNotes.Config subtable, and change all the code that referenced the past values self.settingname or gKH.ContactNotes.settingname to gKH.ContactNotes.Config.settingname and add some code to save and restore it at scene load.

--- end caveats ---

Here are some other things worth noting or explaining up front, maybe for the newer to Lua folks looking through the ContactsNotes code.

Q: Why are some somethings "." vs ":"?
A: "." is used to create\reference an entry off whatever precedes it.
But function declaration can also use the ":" separator, so you can use either . The ":" is syntax sugar for auto adding a first sort of hidden parameter that gets filled and can be referenced enabling the use of the self keyword. I use tend to use ":" particularly when the function is part of what I consider a "object", or something I could have more than 1 of, or where the functions themselves operate mainly over data self contained to things in that namespace, or when I know one function will be calling other functions from within the same namespace (or table level if you like) so I can use self without having to manually set it up. It makes mass-renaming things easier. Pro tip, if you like me and sometimes use both, just pick one at least for the particular level or namespace you're working in. You will see I use self often in the ContactNotes:Whatever() functions, where ever you see self.xxx or self:xxx it, for this example, can be considered to be read "gKH.ContactNotes.xxxx or gKH.ContactNotes:xxxxx" . But hey some book or whatever told me always use "." or ":", look my it's party and I'll "." or ":" if I want to. :D I think I detailed some of this in a post here probably long buried by now if you want all the details - use whatever\which ever works best for you.

Q: Why am I using a string indexed table for the ContactNote.Data table vs a numeric index.
A: The upside is it makes checking if somethings exists and accessing\replacing\removing it by contactguid just about as fast as it can be made, and I don't have the requirements that necessitate an integer, sequential or otherwise index. Down side is one can't use ipairs or for i = 1, #tblname etc (least not without doing your own count first), and key pairs have nondeterministic ordering when enumerating. However for the context I was working with this isn't needed, and there are potentially thousands of "notes" possible in a scene so speed primary concern. The difference between if tbl [ "ctguidhere" ] ~= nil, vs running through a for looking for if tbl.thisentry.ctguid == "somevalue" is quite large at scale (thousands).

Q: How often should I be invoking SaveTableToKey()?
A: Good question. Ideally, only as often as you make changes or can reasonably assume some have occurred necessitating it, and yet as little has possible, especially with tables of any significant size.

It can be a tricky question so I'll keep on being insanely verbose. :)

It's basically a massive string parser, that means overhead, the more data you are saving to a single key the more that must be processed and the more overhead, the more nested, also I think the more overhead. That said if you doing a dozen settings, and couple kilobytes of data, it's usually an afterthought even on my ancient rig. UNLESS you are calling every things every second or two or less, or it's aligned on a timescale (same 'regular time') where you are also doing lots and lots of other expensive work in the same second\pulse. The bottom line is when you make changes, those changes aren't saved 'to what will get auto-saved, or saved with a manual save', until after you call SaveTableToKey(). You have to plan and design accordingly for that.

So if your data can be out-of-sync, you could choose to just save on some periodic schedule. That could be a better option if you're making changes say every second or two to a ton of data but don't want take a perf hit of re-saving to a key say 100kb of data every second - which would be less then ideal. Assuming you could live with say doing it every 15sec, and the data is likely not be too out of sync with autosaves if you have them enabled, combined with say a function that runs just after loading/restoring your data back "on scene load" that re-validates or refreshes the data is possible, and I've had that work well in some instances, again depends on circumstances. In that case your basically trading more (and probably little tedious) code writing (the validator\re-syncing routines) and taking a single hit on startup to run it, in exchange for less overhead while playing... what's 2-3 more seconds during during the loading process vs say sealing 100ms out of every 1,2 or five seconds, right?

Another option there, that is available, is flat warning the user if they want solid saves, breif them up front asking the user to pause, and press some Special Action before saving which you provide that forces everything to save at the point in time - which is not great (who wants remember to pop into SA area before saving right?), but it would ensure no data being out of sync. You could do both now, or combination of things, and in Tiny+ you can even invoke the Save process yourself in code, they could for instance turn off auto saves and let your own autosave sub-system take care of things. Maybe that's a project to add to my list, gKH.CustomAutoSaver now that devs added all the plumbing needed over the last year for it.

In the scene 2 example, it can stand to have some stale data around to still reasonably function without losing data (worst case you have to edit a contact with it's two minute old name, or wait the two minute to update the note), so it doesn't run the scavenger constantly but only every two minutes. Even there it only even checks if it's time once per minute, we could check that every second or every 15, but ask yourself, what's the maximum delay for something your code can live with and start there.

The opposite example would be in the examples where the "Elapsed Seconds" counter updates and saves every second, and really nothing else will do to be accurate in the case of a manual save at any given second. Ideally that's a candidate for not using SaveTableToKey vs having it's own KeyStore key because we're saving the whole of SceneGlobals table each second, which for our example of a dozen or so entries with a couple small nested test tables is not a big deal - but could be if larger. Often I will segregate table data into 2-3 areas for something, the meat (say .Data), the settings (.Config), and values that are constantly updated (say .Volatile). This way the settings that don't change much are baked into one keystore key updated only when they infrequently change or are invoked by the player, volatile to another perhaps update very often and kept to a few entries, and .Data holding larger data storage table(s) to another IF i deem it advantageous to save each separately and at different rates.

For example if you look at the tail end of the gKH.ContactNotes : AddDetectedContact ( ct ) function code you will see I call SaveTableToKey at the end of each selected units note updating. Optimally I wouldn't do that, I'd create a didAnytingChange flag and set that instead, and move the save to the end conditioned upon didAnythingChange, that way if you selected 100 contacts and wanted to add notes to them all at once (why?) the save would only happen once when needed, instead of right now where it would actually be called all 100 times. Does it matter for our example...not really but it's something I would probably change anyway, I left it that way so I could have something to point at, and because if things were running in the background when playing, I was thinking I wanted an auto-save during the time it would take say entering 20 notes to capture the changes. Least that's what I was thinking when writing it, looking back I question it.


Q: Ok so how much data, how big a table can I store in a single key, 10kb? 100kb? 10MB, 100MB?
A: Technically a lot. How much memory do you have to spare? I mean in the command.exe 32-bit process space, not your system. Just cause you can, doesn't mean you should, or often is even need, at least stored inside the save anyway. That said I have stored a couple hundred kb in keys for years now without issues, and in testing I believe I stored 14MB once, and 47MB total across 4-5 keys back in late stage CMNAO days, and I think I ran similar test in early stage CMO with 17MB. HOWEVER, generally speaking don't do that,first off CMO file sizes have already grown themselves over time, for example a year or two ago we didn't have quality lastdetection data tables for all contacts, great feature, I asked for it, but I knew it would also bloat the save files, and it has done it's share of that. :D (worth it!)
But CMO increasingly is butting heads as it is with OOM issues parsing large game files and overall process address space issues, with larger and larger scenes, devs have done yeoman work there but only so much they can do. Your couple hundred kb extra isn't going to change much, but your couple MB may in an already beefy seen might. Besides if you're storing that much data you're probably on Professional Edition anyway don't need to be reading most if any of this and probably stopped reading this an hour ago. :)

Q: What is the Dependency Check code about?
A: It's simply to try to ensure you're loading things in the wrong order. IE ContactNotes uses gKH.State, so both gKH and gKH.State must exist before ContactNotes can create its namespace,and it's functions for use without error since it depends on other global functions existing. While you should understand that, you don't really have too, so long as you just remember the gKH.State library needs to run at "scene loaded" before gKH.ContactNotes library, or whatever code you write that uses the two gKH.State functions.

Q: What happens if execute LoadTableFromKey("somekeyname"), and somekeyname does not exist yet?
A: Well instead of a table returned like you expect you will get nil, which is why when calling it I suggest you always check for nil. Like is done in both the "Startup_Globals" action and the "Startup_ContactNotes" action, the later being down in the bottom in : bootstraper(), but to spell it out again:

Code: Select all

local tbl = gKH.State.LoadTableFromKey("somekeyname");
if (tbl ~= nil) and type(tbl) == "table" then 
  print("successfully got a table back."); 
else
  print("something wrong! using a default");
  tbl = {};
end

Q: Can you write something for me that re-creates the events and triggers for the basics of this?
A: - Err I could, but really why, outside the on-scenario load event, which you're likely to have already and just need to add an action script to the existing list. That's the preferred way since you're more guaranteed the correct execution order of scripts upon scene load vs multiple events each with their own sceneload trigger. All the rest are just specific to the example setup, and you probably don't need. I suppose I could for gKH.ContactNotes scavenger and detection events if there are people who actually think the tool is useful, but even there the detection event I added was just for an example to highlight a trigger updating\adding data. But I also think it's important for folks to setup the stuff manually, such that they understand at least to some degree how things work, or don't work. IDK I guess if there is demand.

Q: The sample files are saved with DB 479a and In build 1147.52 version stamp\format, why, it's 2023?
A: Yup they are, I still use 1.04.1147.52 a lot for stability purposes, don't get wrong, lots of good stuff in Tiny+ (1.05.13xx+), but some days I don't want to fight the teething problems, and most of the code and example concepts for this in particular have been around since CMANO. That said loaded them in 1307.3, they should work ok with one exception. If in "group view" and selecting the special contact "Bravo__AFB1" and trying to add a note too it, it will fail because both ScenEdit_GetContact and VP_GetContact can't find the contact guid even though it exists. Same code works fine in 1.04.1147.52 however, report has been submitted.


Q: Should SaveTableToKey and LoadTableFromKey like functions like this be added to the game itself?
A: I don't know to be honest.
Over the years I thought about it and always come back to "why?, this works pretty well implemented in pure Lua as it is right?". Then the next thought is "of all the things on the dev's to-do list, do I really want to add this has another?", followed by "the pro users already probably have access to outside .net modules so how many non-pro users are really storing enough data for that to be worth it?". Also I don't know that having .net native do the stringifying\destringing would up performance a large amount,it might, I don't have data on that, but even if it did, is it enough juice for the dev squeeze?

Q: I really want to serialize some table object of mine that has data and functions, can't we just tell it when it see's a function to skip it and continue on do everything else it can, and just skip what it can't do?

A: Now that you specifically mention it, technically I think I tried that once by removing the call to error( 'Unjsonifiable type' ) in gKH.json.stringify( ) and replacing it with return 'null'. OR I might have just added an extra " elseif kind == 'function' then return 'null' ; ". The result as I recall, and it's been some time, was the entry for the function names got restored each having an empty table {} as their value. Which in theory could/would then be overwritten by your actual function declarations if order of execution was right. Meaning you 'restored' your data first, and then moved on to definitions, where usually it's the opposite. I'm not suggesting you do it, I'm not even suggesting it will always work. I am suggesting it's probably possible with some tinkering if you really wanted to play around.


Q: Why gKH as root of namespace, why namespace extra table levels at all?
A: To help organize my own stuff and de-conflict\ avoid name collisions and overall limit my pollution the global namespace to just gKH and below. Why that name, cause it's short to type,conforms with the existing rules (starts lowercase, and isn't a keyword or base lua library or function name), at a bit unlikely to collide with anyone else's namespace in ram during a single play session of a few different maps from a few different authors. Also it hard for me to forget or stop using at this point. So I kinda claimed it. I suggest people do the same for there own custom stuff, pick a g+2-6 character prefix that works for you that doesn't otherwise conflict. I do similar with KeyStore key names just in case people are gonna combine stuff from different people into one scene unchanged - far less likely to run into a collision using "gKH_SomethingSomthing", then say "Data". Both involve some more typing lol, but i don't mind it's clearer to me. Speaking of more typing you'll often I still use if something == true then, instead of if something then, compiles to same thing almost always in lua, but I find reading it more clear, too each there own.


Q: Isn't too much nested table depth, namespace depth, a performance hit?
A: Well,it can be, and occasionally more than you think, but it depends on how much one transverses the tree, and how often it is. Here is my arbitrary guideline to myself, if I've got more than 3 dots it's time to maybe re-think, or start testing performance difference, so seeing if local copy instance helps performance. What do I mean by that?

Check out the following code, dosomething1 and dosomething2, dosomething3 function do effectively the same thing.
The first accesses via full global path, the second grabs a local instance at a certain level, the third, uses local instance at closest level it can.

Code: Select all

--assumes a gKH table already exists. 
gKH.House={};gKH.House.Kitchen={};gKH.House.Kitchen.Fridge={};gKH.House.Kitchen.Fridge.Freezer={};
gKH.House.Kitchen.Fridge.Const = {Celsius=0,F=32};
function gKH.House.Kitchen.Fridge.Freezer:GetTemp(a) return(a + 1); end
function gKH.House.Kitchen.Fridge.Freezer:GetTemp2(a) return(a + 1); end
local function doSomething(limit)
  starttime = os.clock(); local count=0;
  for i = 1,limit do
    local tmp = gKH.House.Kitchen.Fridge.Freezer:GetTemp(gKH.House.Kitchen.Fridge.Const.Celsius);
    count = count +1;
    local tmp = gKH.House.Kitchen.Fridge.Freezer:GetTemp2(gKH.House.Kitchen.Fridge.Const.F);
  end
  timetaken = (os.clock() - starttime) * 1000;
  print(string.format('1 timetaken (ms): %.8f',timetaken));
end

local function doSomething2(limit)
  local fridge = gKH.House.Kitchen.Fridge;  local c = gKH.House.Kitchen.Fridge.Const
  starttime = os.clock(); local count=0;
  for i = 1,limit do
    local tmp = fridge.Freezer:GetTemp(c.Celsius);    count = count +1;    local tmp = fridge.Freezer:GetTemp2(c.F);
  end
  timetaken = (os.clock() - starttime) * 1000;
  print(string.format('2 timetaken (ms): %.8f',timetaken));
end

local function doSomething3(limit)
  local freeze = gKH.House.Kitchen.Fridge.Freezer;  local c = gKH.House.Kitchen.Fridge.Const
  starttime = os.clock();local count=0;
  for i = 1,limit do
    local tmp = freeze:GetTemp(c.Celsius);    count = count +1;    local tmp = freeze:GetTemp2(c.F);
  end
  timetaken = (os.clock() - starttime) * 1000;
  print(string.format('3 timetaken (ms): %.8f',timetaken));
end
doSomething(10000);
doSomething2(10000);
doSomething3(10000);
gKH.House = nil;

On my ancient system the first takes like on avg 6-7ms and the second 2.5-3ms, and the third rounded up ever so slightly 2ms. Put another way the third is about 3x faster than the first, the second a little over 2x fast than first. Since these aren't doing much actual work, most of it is just the access over overhead, so yeah if doing tight loops over thousands of items be considerate of > 3 levels.

Ok that's enough for now, if anyone needs help etc just ask.
If you made it this far you get a golden star sticker. :D

Full file both sample .scen files and the 2 main .lua components in txt files, and this raw cut and paste from this forum post in .txt form.
Stringify_Samples.zip
(74.63 KiB) Downloaded 33 times
Last edited by KnightHawk75 on Sun Feb 19, 2023 11:02 pm, edited 3 times in total.
Parel803
Posts: 972
Joined: Thu Oct 10, 2019 3:39 pm

Re: How To: Save a table or nested tables to keys

Post by Parel803 »

Wow,
No time to read it now but many thanks for the time you spend on sharing this sort of stuff.
Learning all the time, much appriciated.
best regards GJ
KnightHawk75
Posts: 1850
Joined: Thu Nov 15, 2018 7:24 pm

Re: How To: Save a table or nested tables to keys

Post by KnightHawk75 »

Parel803 wrote: Sun Feb 19, 2023 9:16 pm Wow,
No time to read it now but many thanks for the time you spend on sharing this sort of stuff.
Learning all the time, much appriciated.
best regards GJ
No problem. it's funny I was just about to reply to you in the tech support thread pointing you here since you seemed interested in sample used for the bug report, posted there even before I could get around to posting it here (this one does have updated files\code though).


FYI my comments in the OP about the bug in "Tiny" build of the game should be resolved in the next release or two, in addition all the extra code needed in 1147.52 falling back to VP_Contact, sounds like Michael found the issue and resolved it for both SE_GetContact and VP_GetContact so certain sorts of base 'contacts' will be able to see and obtained by both in near future. But I digress.
Parel803
Posts: 972
Joined: Thu Oct 10, 2019 3:39 pm

Re: How To: Save a table or nested tables to keys

Post by Parel803 »

KH,
Thx again. My first thought if I'm able to have an options to add some free text to a contact which I can see later on, later maybe the same thing on units. One of my defects is to compare stuff to a CMS, can't help it :? Realizing C:MO isn't. So lots to read and try out. The whole keyValue is still fuzzy material but with slow pace learning.
In the first chapter the position of lua history is on my pc in Logs, not important, might be optional. Just something I noticed when start reading.
best regards GJ
KnightHawk75
Posts: 1850
Joined: Thu Nov 15, 2018 7:24 pm

Re: How To: Save a table or nested tables to keys

Post by KnightHawk75 »

Parel803 wrote: Mon Feb 20, 2023 8:08 am KH,
Thx again. My first thought if I'm able to have an options to add some free text to a contact which I can see later on, later maybe the same thing on units. One of my defects is to compare stuff to a CMS, can't help it :? Realizing C:MO isn't. So lots to read and try out. The whole keyValue is still fuzzy material but with slow pace learning.
In the first chapter the position of lua history is on my pc in Logs, not important, might be optional. Just something I noticed when start reading.
best regards GJ
You mean seeing note on the contact in the map or on mouseover? No way to do that.
I guess you could set it up to optionally 'bark' the notes at you in intervals via the newer bark feature. IDK how useful that is..but could be a per-entry option.
As for taking notes on your own unit's, that's an easy change, just have to re-work the selectedunits part to search in units after contacts if no contacts are found.
Parel803
Posts: 972
Joined: Thu Oct 10, 2019 3:39 pm

Re: How To: Save a table or nested tables to keys

Post by Parel803 »

Not really the mouse over. For ideally a free textfield with the contact status but being able to add, read and delete remarks/notes is a great alternative. In the process of reading and after that try to copy your notes on contacts.
regards GJ
Post Reply

Return to “Lua Legion”