SANDMAN - a fatigue/effectiveness modeling framework

[musurca]

This is a graph of micronap risk per minute vs time without sleep for a pilot. In the next release I'll halve those 3AM peaks at 56+ hours (although keep in mind we're talking about people who have not slept at all for 3 days). Once a nap starts, it's twice as likely to continue as stop in the next minute. This model is based on Miller & Melfi, as well as scattered accounts of pilots talking about nodding off who gave some estimate of how long they'd been awake at that point. The general idea is that this phenomenon is incredibly common, but data from which one might build up a model is scarce.

[center][/center]

This the graph for crash risk per minute while landing at an airbase, based on the default 'normal' chance of crashes set to 3.5 per 100K flight hours. (We're talking about hundredths of a percent here.) Landing on a ship will multiply this risk by anywhere from 5-20x, depending on weather and time of day. Boltering & go-arounds are currently 100x more likely. This model is based on the FMCSA's study of transportation safety which suggests a polynomial relationship between crash likelihood and hours without sleep, as well as a SAFTE data point which noted that pilots at effectiveness ~50% were 65x more likely to be in an accident. Increased risks based on time of day, weather, land vs. sea, etc. are purely speculative on my part.

[center][/center]

EDIT: Note that currently crashes can ONLY happen during approach for landing.

[boogabooga]

Okay, every time I launch this scenario, one of the F-16s begins a "nap" almost immediately after takeoff, and the "nap" seems permanent.

[musurca]

Thanks for finding this. The culprit, as feared, is the stock math.random(), which is returning the exact same small improbable value over and over for that plane, so that the nap never ends. (If you change the random seed, notice that the bug goes away.) In the next release, I'll replace it with the Xorshift32 implementation I use in IKE.

I haven't had time to report this in the Tech Support forum and create a save that demonstrates the issue, but I think what's happening is that math.random() resets its seed value every frame.

[boogabooga]

I mean, if math.random() is bugged, is that a CMO problem or a lua problem?

BTW, I recommend that you put those graphs in the readme. Perhaps try to document the model itself as well as possible.

[musurca]

The problem relates to the implementation of math.random() in the embedded Lua library and how it interacts with the game. Beyond that I can't speculate without doing a deep dive to produce a save file that reproduces the conditions under which it happens. (I'll probably use the one you sent as a starting point.)

However, it's basically the same issue I noticed while working on the PBEM mod, which stores and resets the current random seed 'state' to prevent save-scumming random events. math.random() started returning the same sequence of values over and over, so that under certain conditions, e.g., the downed pilots in "North Pacific Shootout" all had the same name. You wouldn't notice this being a problem in most scenarios, as different calls to math.random() don't necessarily line up with the same context in the code. However, because the SANDMAN update loop just iterates through the same list of units every minute, entries in the pseudo-random sequence tend to line up with the same conditions, which is why that one F-16 kept napping over and over. I verified that this is indeed happening by printing out the values in the save file you posted.

What I suspect is happening is either that, under certain conditions...
a) math.random() is resetting its own state, either every frame or else within a period less than the 60 seconds between SANDMAN updates; or,
b) math.random()'s underlying PRNG is rolling over MUCH faster than it should.

I wouldn't blame the CMO team, as you can find Stack Overflow questions about this same problem happening in other embedded Lua contexts. The advice is always some black magic suggestion like "depending on your platform, read a passage from the Necronomicon and call math.random() three or four times to prime it before use," but that of course is total nonsense, as that is not how pseudo-random number generators should work in production. The Lua users wiki provides a series of increasingly insane suggestions before recommending that you just import the third-party lrandom library.

My solution to minimize time spent debugging arcane issues, both here and in IKE, is to reimplement a proper PRNG in pure Lua, and avoid math.random() entirely.

EDIT:

BTW, I recommend that you put those graphs in the readme. Perhaps try to document the model itself as well as possible.

Sure, that's a good idea. Since this is all still in flux as I make changes based on the excellent suggestions so far, the graphs may fall behind as the model changes.

[boogabooga]

Sorry, can't resist...

https://xkcd.com/1277/

[musurca]

LOL. Nothing turns me into a ranting Galtian lunatic faster than the mere mention of math.random!!

[nukkxx5058]

Wow ! Just wow !!! [&o][&o][&o][&o][&o][&o][&o][&o]

[boogabooga]

If I change the plane set after an initial SANDMAN install, do I just reinstall, or do I need to delete all of the events / special actions first? What about the global key/value register?

[musurca]

If I change the plane set after an initial SANDMAN install, do I just reinstall, or do I need to delete all of the events / special actions first? What about the global key/value register?

You just have to reinstall. Everything else is handled by the SANDMAN wizard.

(This will be even smoother in v0.2.0, which is coming as soon as the new-forum-dust settles a bit.)

[musurca]

Hi everyone — happy to say that SANDMAN v0.2.0 is out now. Thanks for all of your great suggestions. Here’s what I was able to add in this release:

RESERVE CREWS

Bases may now store reserve crews per unit class, which can be swapped in to replace exhausted pilots. Planes that start the game as “Maintenance (Unavailable)” or “Reserve (Available)” automatically add reserve crews to the local pool. Scenario designers may also add reserve crews via the Lua API:

Code: Select all

Sandman_AddReserves({
	guid = '', — guid of the base to add the reserve crew
	dbid= 437, — DBID of the unit type
	proficiency=“Veteran”, —base proficiency level of the reserve crew
	num=4,	—number of reserve crews to add
	min_hoursawake=6, — (optional) minimum # of hours awake, if different from side default
	max_hoursawake=15 — (optional) maximum # of hours awake, if different from side default
})

Placing pilots in reserve is also the most effective way to rest them. You can manually do this by switching the loadout of a plane to “Reserve (Available)”.

You can set your automatic replacement thresholds from the Special Actions menu.

MULTICREW MODELING

The effectiveness level of planes with multiple crew members is the composite of their individual effectiveness levels. They are also proportionally less likely to micronap.

GLOBAL TIMEZONES + CIRCADIAN RHYTHM SHIFTING

There’s no need to calibrate a local timezone anymore — this is handled automatically by unit longitude. Crews start with their circadian rhythms synchronized to their starting timezones, but these rhythms will slowly shift as they move. That means you’ll now end up with jetlagged crews if you transfer them long distances.

PEAK AWARENESS TIME (P.A.T.)

The Fatigue Awareness Tool now shows you each crew’s Peak Awareness Time (P.A.T.) in Zulu time, so that you can synchronize your operations with your crew’s circadian rhythms.

ENABLE FATIGUE TRACKING BY SIDE

In the SANDMAN wizard, you now choose to enable fatigue tracking for each side. You can also customize the fatigue modeling values per side.

API CHANGES

The API now follows the CMO standard more closely:

Code: Select all

Sandman_Disable()
Sandman_Enable()
Sandman_SetRandomSleepDeficit({guid, min_hoursawake, max_hoursawake, longitude})
Sandman_GetEffectiveness({guid})
Sandman_GetCrashRisk({guid})
Sandman_AddReserves({guid, dbid, proficiency, num, min_hoursawake, max_hoursawake})

MISCELLANEOUS

-Proficiency reduction is now modeled as a delta value, not a multiplier. That means more experienced pilots will stay effective longer.
-The built-in pseudo-random number generator has been replaced with a pure Lua implementation, which should cut down on weird results (see my above rant for context on this).
-UAVs automatically get 2x reserve crews, and will never micronap.
-The SANDMAN update function is now called on prime number intervals between 47 and 71 seconds, so that events no longer happen on exact minute boundaries.

IMPORTANT NOTE: For best results, install this on a fresh copy of your scenario instead of installing it over the previous (0.1.x) version of SANDMAN.

[musurca]

I've just put out a new release out to fix a bug involving much higher-than-intended crash incidence. Thanks to @KJohnston for finding that one by wrecking a bunch of SR-72s.

DOWNLOAD SANDMAN v0.2.1

[nukkxx5058]

I was wondering what parameters you guys are using (if not the default ones) ?
Because I noticed that too many of my pilots started as "cadet" so I tried to limit the awake time by using 4-10 instead of the 6-14 suggested. With 3.5 crash rate unchanged. It's also partly because my scenario is using various proficiencies for pilotes but not for all so I cannot change prof for all to 'veteran' for example.

Other point: I was wondering whether it could be possible -or rather if it would make sense- to have fatigue activated for ground units too ?

What do you think ?

[vettim89]

I was wondering if you have tested SANDMAN's effects on a computer run side. Does the AI break down with the limitations in place? In other words, is it best to only apply SANDMAN to sides that are playable?

[musurca]

Because I noticed that too many of my pilots started as "cadet" so I tried to limit the awake time by using 4-10 instead of the 6-14 suggested. With 3.5 crash rate unchanged. It's also partly because my scenario is using various proficiencies for pilotes but not for all so I cannot change prof for all to 'veteran' for example.

Other point: I was wondering whether it could be possible -or rather if it would make sense- to have fatigue activated for ground units too ?

I suspect the default starting hours-awake currently in SANDMAN (6-14hrs) are a little high. In a future release I'll probably bring it closer to the 4-10hrs you're currently using. However, it's hard to say that either one is "right" as it all depends on the given circumstances of your scenario.

Regarding ground units, it's certainly possible to activate fatigue for them, but I haven't done enough research on the issues involved to be able to know whether the current model (which is based on papers written specifically about fatigue in aviation) would apply to them as is. If someone is interested and motivated to try extending the model to other domains, the code for SANDMAN is all up on GitHub, and I'd be more than happy to merge pull requests.

I was wondering if you have tested SANDMAN's effects on a computer run side. Does the AI break down with the limitations in place? In other words, is it best to only apply SANDMAN to sides that are playable?

It should be fine running on an AI side, although by default the AI sides will not rest their pilots as intelligently as a player might. There are some ways in which scenario authors can improve this situation; you could, for example, adjust the AI's threshold for swapping out active crews with reserves by switching to the AI side in the editor and running the Special Action as if you were playing that side. You could also write some rudimentary logic in Lua using the SANDMAN API (as documented above) to check unit effectiveness and order units to RTB past some threshold. But even without all that, you shouldn't notice the AI side break down completely, outside of really extreme circumstances.

[nukkxx5058]

In fact, after doing more tests, it seems that I cannot get my pilots to go below 75-80%, even after 30+ hours on duty. After a while, the % effectiveness stopped deteriorating. For example, a pilot on AAW patrol with refueling is blocked on 73% after 35hours on duty (!!) Not sure what's going on...
So I might return to 6-14 setup.

*EDIT: in fact, effectiveness is increasing. I tried with 6/14/3.5 and same. After 15 hours or so, many pilots are back to 100% (they might had a rest) but very few seem exhausted (<70%)
But I think I will keep the 6/14/3.5 setup in fact.

EDIT2: In fact with default 6/14/3.5 it's working just fine ! Don't know why but with previous numbers (4/10/3.5) it apparently didn't. So I'll def keep 6/14.

[ClaudeJ]

Hey there,

what an awesome script you have there.

I wouldn't be surprised to see it become an embedded feature of CMO at some point

Is it a way to apply a multiplier to speed up the appearance of fatigue ?
I would use it in a scenario of quite short duration.

PS : I was reading a boot about Human Factors in Transportation and there was a chapter about "night owls", ie. circadian rhythm disorder, which could still be useful to have.
Is this something you would consider ?

PSbis : What do you think about the aircraft's generation having an impact too ?
My reasoning here is that a contemporary fighter is built to reduce workload and fatigue on the pilot, whom can fly endure longer mission. Eg. a Rafale CAS/strike mission was flown over Mali from an airbase in France. I assume that, besides the aircraft itself, it would have been much more taxing on the pilot to perform with a, say, F-100.
I think that it would allow to somewhat simulate one of the benefit of operating "high end" platforms instead of 40 years old airframes.
However, I don't see a variable that could be referred to check which "gen" is the aircraft of. It would thus require to maintain a custom-made list, and that would be a pain.

[musurca]

Is it a way to apply a multiplier to speed up the appearance of fatigue ?
I would use it in a scenario of quite short duration.

Not currently, but I can definitely add that as a variable in the next release. Good idea, thanks!

"night owls", ie. circadian rhythm disorder, which could still be useful to have. Is this something you would consider ?

This is a reasonably rare condition, no? But if you can send me some data on it, I can take a look at how it might fit in to the model.

What do you think about the aircraft's generation having an impact too ? My reasoning here is that a contemporary fighter is built to reduce workload and fatigue on the pilot, whom can fly endure longer mission.

It's an interesting question. While I would guess that next-gen fighters with, e.g., glass cockpits do indeed reduce the workload on a pilot, I'm not sure they necessarily reduce fatigue as modeled in this framework, which is mostly a function of how long the pilot's been awake while working on a reasonably complex task (flying a combat jet being a complex task regardless of automation level, I think). I suspect the best way to model the effect of a next-gen airforce would be to simply reduce the accident rate per 10,000 hours variable in SANDMAN, which you can do when you install the framework into the scenario.

I could be wrong, though! Again, if you can provide any data, I'll definitely take a look.