SANDMAN - a fatigue/effectiveness modeling framework

[musurca]

Hi everyone. After reading comments on the CMO Discord about the lack of pilot fatigue (@KJohnston: "I just think to myself that the E-3 operators have a mini fridge full of red bull or something"), I went down a bit of a rabbit hole wondering what it might look like to model fatigue in a plausible, exhaustive way—and ended up creating this framework for adding fatigue to your scenarios, which I call SANDMAN.

DOWNLOAD SANDMAN v0.2.1

SANDMAN simulates the current fatigue level of your pilots using a realistic approach derived from the SAFTE model (see Hursh, et al. in the bibliography) based on their sustained activity, sleep debt, and circadian rhythms, then degrades their proficiency accordingly. You may also see exhausted pilots taking short, dangerous "micronaps" in which their plane goes out of comms, and becomes vulnerable to enemy action. Extremely exhausted pilots may even begin to bolter/go-around on landings, or even (under very, very rare conditions) crash.

You can monitor the effectiveness of your pilots, and attempt to stand down particularly exhausted pilots for rest, using the Fatigue Avoidance Scheduling Tool (available from the Special Actions menu). Note that leaving planes "parked" will be the most effective way to rest your pilots. The pilots of "readying" planes will still rest somewhat, but very ineffectively.

While major penalties to pilot effectiveness will mostly occur in longer scenarios (24hrs+), you will always see effectiveness vary +/- 7-16% based on circadian rhythms during the daily cycle, making the time of day at which you launch operations a significant factor in planning. (In this circadian model, alertness peaks at 2000hrs, and reaches a minimum at 0300hrs.)

How to install into your scenarios:

1) Download the latest release.
2) Open your scenario in the Scenario Editor.
3) Go to Editor -> Lua Script Console
4) Paste the contents of the file sandman_min.lua into the white box, then click Run.
5) Complete the SANDMAN wizard and voila! You now have sleepy pilots.

Hope you find it useful! I've also put the source code up on GitHub if anyone wants to contribute/make a pull request to the framework. In particular, the risk functions I wrote for crashes and micronaps are more speculative; if anyone has better insight or hard data on those subjects it would be very helpful.

Thanks to @stww2, @Fido81, @Tempest, @Gunner98, @AndrewJ, @KnightHawk75, @boogabooga, and @KJohnston for contributing to the discussion as I was developing this, as well as @rbsj, whose thread on fatigue-related crashes during carrier landings was an excellent starting point.



Bibliography:

BRANFORD et al. Biomathematical Fatigue Models Guidance Document. Civil Aviation Safety Authority (CASA) Australia, March 2014.

MILLER & MELFI. Causes and Effects of Fatigue in Experienced Military Aircrew. Air Force Research Laboratory, Jan 2006.

HURSH et al. Fatigue models for applied research in warfighting. Aviat Space Environ Med 2004; 00(3, Suppl.):A000 – 00.

ROMA et al. Flight Attendant Work/Rest Patterns, Alertness, and Performance Assessment: Field Validation of Biomathematical Fatigue Modeling. Federal Aviation Administration/Office of Aerospace Medicine, Sept 2012.

[stww2]

what it might look like to model fatigue in a plausible, exhaustive way

I see what you did there!

[RockPaperScissors]

Thank you ... this is a similar approach to fatigue modelling as in the (excellent) SimEx scenario(s) ... looking forward to enriching scenarios with this.

KR, RPS

[BDukes]

Wow nice add. Another step toward longer game models.

Mike

[Gunner98]

This looks interesting. I haven't downloaded it or played with it yet but have a couple questions:

-Is this meant for the scenario designer to setup as default in the scenario or for a player to add if desired?
-I suppose a designer could just auto run the script at scenario start
-If for the designer, a Special Action might be nice so a player can turn the feature on/off at the start of the game
-Can a designer set a fatigue level for the scenario to start at? Is this customizable? I.e. Sqn XXX has been in the fight for 3 days and is bagged but Sqn YYY is just arriving and is fresh.
-Can/are spare pilots factored in? I know that many nations have about 1.2 to 1.4 pilots (bit of a guess but I think its right) per airframe in active Sqns

this looks like a great addition - says a guy who fields regular complaints of overly complex scenarios already ...[:'(]

Thank you

[musurca]

Just put out v0.1.1, which includes some small fixes and adds the following Lua API for scenario designers:

-- disable/enable SANDMAN and all fatigue tracking
Sandman_Disable()
Sandman_Enable()

-- set the sleep deficit for a particular unit to a random value between min_hrs and max_hrs
Sandman_SetRandomSleepDeficit(guid, min_hrs, max_hrs)

-- get the current effectiveness for a particular unit as a number [0-1]
Sandman_GetEffectiveness(guid)

-- get the current hourly risk level for a particular unit as a number [0-1]
Sandman_GetCrashRisk(guid)
Sandman_GetMicroNapRisk(guid)

-Is this meant for the scenario designer to setup as default in the scenario or for a player to add if desired?

Both. When you run that script in the editor, it creates all of the necessary Events and Actions, then injects its own code into the scenario so that it remains persistent after save. So if you’re a designer, you just install it once toward the end of the scenario creation process and that’s it. (In other words, no need to run the script every time on load. IKE users will be familiar with this process already.)

If you’re a player, you can just do the same thing with any existing scenario; install the script, save the scenario, and you’ll be good to go.

Note that the framework only knows about the planes that were in the scenario when it was first run, so if you add or remove planes, you'll want to install it again.

-If for the designer, a Special Action might be nice so a player can turn the feature on/off at the start of the game

That's a good idea. In this release, I added a Lua function for use by scenario designers to disable SANDMAN (see above, Sandman_Disable() ), so that you can create your own Special Action to give players the option to do so.

-Can a designer set a fatigue level for the scenario to start at? Is this customizable? I.e. Sqn XXX has been in the fight for 3 days and is bagged but Sqn YYY is just arriving and is fresh.

Yes and yes, although it's not yet as customizable as I'd like. When you install the script, it runs a wizard asking you if you want to customize the default values. You can do so, and set both the MINIMUM and MAXIMUM number of hours that pilots have been awake when the scenario starts.

Right now these values are applied to all pilots from all sides. In the future, the wizard will ask you to customize this by side. I added another Lua function for scenario designers so that you can manually set starting effectiveness levels for individual pilots (see above, Sandman_SetRandomSleepDeficit(guid, min_hrs, max_hrs) ), if you want to make one squadron fresher than another.

-Can/are spare pilots factored in? I know that many nations have about 1.2 to 1.4 pilots (bit of a guess but I think its right) per airframe in active Sqns

Not yet, but I'd like to add this—perhaps by creating a pool of reserve pilots that can be swapped in to replace exhausted pilots once they're on the ground, via a Special Action. If you or anyone else has more information about how this is done, which types of units tend to have spare pilots, etc., it would be very useful.

says a guy who fields regular complaints of overly complex scenarios already

Ha -- yep, this framework is definitely meant for a very particular sort of person!

[Gunner98]

OK this looks great, I need to give it a run this weekend.

B

[AndrewJ]

This looks like a great addition to long multi-day scenarios.

The circadian rhythm alertness variation sounds interesting.

Do the high and low points of alertness, in Zulu time, vary depending on the base location of the aircraft? For example, an aircraft based in one hemisphere being alert when one on the opposite hemisphere is not?

If an aircraft re-bases a significant distance, such as intercontinental squadron deployments, will the planes (gradually?) adopt the alertness times of their new base?

[musurca]

Do the high and low points of alertness, in Zulu time, vary depending on the base location of the aircraft? For example, an aircraft based in one hemisphere being alert when one on the opposite hemisphere is not?

If an aircraft re-bases a significant distance, such as intercontinental squadron deployments, will the planes (gradually?) adopt the alertness times of their new base?

Not yet. Right now, when installing SANDMAN, the user/scenario designer is asked to calibrate the local time zone by positioning the camera. That local time is then used for the circadian rhythms of all pilots. For a scenario in which you're using a spread of time zones, you could pick one in the middle. You could also pick a timezone on the other side of the world if you're dealing with rapidly redeployed pilots who haven't adjusted yet.

In the next release, I'm planning to do what you're suggesting: get the timezone by base location, track the pilot's starting internal circadian clock, and slowly shift it over time. Keep in mind that you probably won't see significant shifts in a pilot's circadian rhythm over the course of a CMO scenario, even if it's on the longer side. With regard to shifting circadian rhythms, "a phase advance (eastward time change) takes about 1.5 d per hour of shift" (Hursh)—which explains why jetlag is so difficult to beat.

[AndrewJ]

With regard to shifting circadian rhythms, "a phase advance (eastward time change) takes about 1.5 d per hour of shift" (Hursh)—which explains why jetlag is so difficult to beat.

That slow? My (limited) experience with five hour trans-Atlantic time shifts was that, provided I stayed active in the day, got as much sunlight as possible, and slept once it was dark, I could reset and operate normally within a couple of days. (Although maybe my body temp cycle hadn't truly caught up, but honestly, I wasn't checking that. [:D])

Regardless, a very interesting addition, which would reduce the temptation to send just one more tanker to those picket fighters...



Edit: Would you actually need to know the specific time zones? Or would it be more biologically representative to pick up the effective "solar time" from the lat/long of the base?

[musurca]

That slow? My (limited) experience with five hour trans-Atlantic time shifts was that, provided I stayed active in the day, got as much sunlight as possible, and slept once it was dark, I could reset and operate normally within a couple of days.

I envy you! That figure in the literature is prefaced with "in general," so there is an amount of individual variation; also, people like me—for whom jetlag is like, as William Gibson put it, "leaving the soul behind"—likely drag down the average. Beyond that, apparently light exposure, as well as “sleep, activity, and social cues” accelerate the process of adjustment, so you’re doing all the right things. The model likely accounts for a general case where the subject is not so conscientious about the adjustment.

Edit: Would you actually need to know the specific time zones? Or would it be more biologically representative to pick up the effective "solar time" from the lat/long of the base?

This is an interesting question, and from the literature, I gather that there’s no simple answer. Ultimately, the “the peak of the circadian rhythm has a reliable relationship to the timing of the period of wakefulness” (Hursh), which would tend to converge on time zone (sometimes called the "social clock") and/or shift-work schedule as opposed to the "sun clock". However, phase shifts are “induced primarily by light exposure rather than by the displacement of sleep or activity” (Czeisler). Switching to a night shift without light exposure by just forcing yourself to wake up in the dark, for example, "may take 2.6 times as long to complete as a comparable transmeridian shift in phase” (Hursh).

The reality is that I’ll probably have to account for both time zone (for initial setting) and length of day by latitude & time of year (to calculate rate of phase shift) as this model becomes more sophisticated. However, after playing around a bit with the CMO map tonight, I think the existing time zone model as reflected in the "local time" is simpler than I thought it was, and is just a function of longitude, without any of the complicated artificial divisions—which is easy enough to replicate.

[Parel803]

Impresive
regards GJ

[boogabooga]

I messed around with version v0.1.0. A few thoughts:

-Why code on one line?

-There was a run through where every single plane that I sent back to base crashed, regardless of the airborne time. Perhaps you have a flag somewhere that needs reset?

-I'm fairly confided that usually, air units have more flight crew than airframes. Therefore, I don't think that you need to permanently track the fatigue with the airframe. Rather, when a plane returns to base, I would assume that the flight crew is replaced by a "fresh" one with the same fatigue level as the other parked A/C.

-I recommend that you take into account the aircraft size and number of crew. Very large aircraft who are going on long-range or long-loiter missions may even have a backup crew with sleeping quarters. Or, they might have the pilot and copilot take turns on rest cycles. But, it will have a lot more resources available for fatigue mitigation than say, a lone single-seat fighter.

-Perhaps the penalties on single seat fighters that have been airborne for say, 14+ hours should be even worse somehow. A "seriously, no human can function this long" level of fatigue.

-Are you being careful to exclude UAVs?

[musurca]

-Why code on one line?

This code is "compiled" (i.e. minified) both to reduce the scenario file size, and to discourage trying to modify it in place, as this will not work due to the way the code is duplicated and converted to an escaped string for re-injection back into the scenario via the wizard.

The best way to play around with the source is to clone the GitHub repository (where the source is nicely laid out in separate files with comments) and set up a build environment as laid out in the README.

(And that way, if you fix a bug or contribute to the model, you can easily submit a "pull request" and I can merge your changes back into the repo for everyone. It would be great for this to be a community-authored fatigue model!)

-There was a run through where every single plane that I sent back to base crashed, regardless of the airborne time. Perhaps you have a flag somewhere that needs reset?

I think this is fixed in v0.1.1 but let me know if you see it happen again. Did you see a special message about it (to the effect of "crashed while attempting to land") or did they just run out of fuel?

The chance of a crash, even with extremely exhausted pilots, should be so minuscule that you should almost never see it happen. Under certain conditions the built-in math.random() may return wacky results, and in the PBEM mod I actually replaced it with a Lua reimplementation to make it perform more reliably. I was trying to avoid cluttering this module with such things, but will import that code as well if this pops up again.

-I'm fairly confided that usually, air units have more flight crew than airframes. Therefore, I don't think that you need to permanently track the fatigue with the airframe. Rather, when a plane returns to base, I would assume that the flight crew is replaced by a "fresh" one with the same fatigue level as the other parked A/C.

It depends on the situation, I think. Sometimes this is true and sometimes not. As Gunner wrote above, "many nations have about 1.2 to 1.4 pilots (bit of a guess but I think its right) per airframe in active Sqns." The main thing is that you almost never have as much manpower as you'd really like.

In a future release, I'm working on a way for the scenario designer to specify the numbers of reserve pilots, and either swap them out automatically upon return or else manually choose which pilots should be refreshed and which shouldn't. Any data you can find on this would be extremely helpful!

-I recommend that you take into account the aircraft size and number of crew. Very large aircraft who are going on long-range or long-loiter missions may even have a backup crew with sleeping quarters. Or, they might have the pilot and copilot take turns on rest cycles. But, it will have a lot more resources available for fatigue mitigation than say, a lone single-seat fighter.

True. Unfortunately as far as I know, ScenEdit_QueryDB() doesn't yet support the 'Aircraft' part of the DB, so there's no easy way to automatically pull the number of crew members for an aircraft type out of CMO. I can think of some awkward ways to pull this out of the DB files and maintain them in the code, but it would have to be done again for every DB update. (Possibly data for only the most common 'large aircraft', which would be resistant to DB updates, could be stored.) For the moment the best way to do this is probably to give the scenario designer more functions to flag multicrewed units.

You're right that it makes sense that a multicrew airframe should be more stable in terms of circadian effectiveness swings, and somewhat more resistant to micronaps. For large aircraft with a backup aircrew on board, it might even make sense to simulate it with a shift-based model (see below). I'm not entirely sure, however, that any rest those crews would be getting in the air would be significantly restorative. The literature seems to suggest that the quality of sleep really matters in this regard. But again, any data that can be scraped here is helpful.

-Perhaps the penalties on single seat fighters that have been airborne for say, 14+ hours should be even worse somehow. A "seriously, no human can function this long" level of fatigue.

This is a good point, but since humans CAN stay in the air this long (and have—see records of solo flights, or the Black Buck bombing runs), I think the way to handle this is to find some data on the additional effectiveness degradation due to considerations of 'discomfort' of being trapped in the same position for hours—the awkwardness of using the bathroom, cramps, etc.—and see if it's significant in comparison to the effects of fatigue.

-Are you being careful to exclude UAVs?

No, as even "unmanned" craft are still manned by the people who monitor them, who are subject to the effects of fatigue. Again, I think this depends on the situation—and someone please jump in here if you have some data on this—but UAVs tend to be manned in shifts with multi-person crews. In ideal circumstances, where manning levels run close to 100%, these shifts would be short and crews would be able to rest—but the literature I've read on this suggests that manning levels rarely reach ideal levels, so fatigue would become a significant factor in a crisis.

The solution here is probably to develop an additional shift-based fatigue system that can be enabled for particular aircraft. I was already thinking of moving in this direction for future updates, by accounting for watches if fatigue were to be extended to naval units.

[RockPaperScissors]

Just another thought for further enrichment; in SIMEX 2018, Apache85 modelled fatigue by progessivly lowering the crew's proficiency with prolonged airborne time.

Excerpt from the game notes: "In C:MANO with air-to-air refuelling it is possible for an aircraft to stay aloft indefinitely. In real life it is uncommon and unsafe for the crew of an aircraft to remain aloft for extended periods. In SIMEX 2018 this is modelled by the Aircrew Fatigue mechanic, which will reduce the proficiency of aircraft that remain aloft for extended periods. If the aircraft remains aloft for an extreme period, there is a chance the pilot may crash due to fatigue."

KR, RPS

[musurca]

Just another thought for further enrichment; in SIMEX 2018, Apache85 modelled fatigue by progessivly lowering the crew's proficiency with prolonged airborne time.

Excerpt from the game notes: "In C:MANO with air-to-air refuelling it is possible for an aircraft to stay aloft indefinitely. In real life it is uncommon and unsafe for the crew of an aircraft to remain aloft for extended periods. In SIMEX 2018 this is modelled by the Aircrew Fatigue mechanic, which will reduce the proficiency of aircraft that remain aloft for extended periods. If the aircraft remains aloft for an extreme period, there is a chance the pilot may crash due to fatigue."

Thank you for the recommendation! This is essentially what SANDMAN does, but I poked into the code for SIMEX 2018 to look at Apache's approach. The difference is that his exhaustion metric is based on airborne time for a particular aircraft, which is then compared an 'exhaustion threshold' that's hand-tuned per class (14hrs for an EA-18G vs. 20hrs for a Poseidon, for example). If SIMEX is modeling an exercise, it makes sense that these numbers should be hand-tuned if they represent not a human limitation, but rather an upper regulatory limit based on managing liability during training. (In that case, a ‘crash,’ rare in the real world, is intended to punish you for breaking a rule.) It also assumes that tired pilots are replaced by fresh ones throughout the entire exercise every time a plane lands. This is all a good approach if you just want to keep the player from running 60-hr CAPs.

What SANDMAN considers, in contrast, are more basic human limitations, for modeling a crisis in non-ideal circumstances where regulatory limits cannot be maintained. It also considers pilots as persistent entities over time, who accumulate sleep debt during extended periods of activity without sufficient rest, as well as human beings with circadian clocks that are sticky, shift slowly, and affect their performance depending on when they’re deployed. It also features a tool for monitoring exactly how fatigued your pilots are.

For a ultra-long scenario like SIMEX (8 days), it would be really interesting to swap out the existing fatigue model with SANDMAN, as you would see crews on both sides start to seriously degrade within a few days without the player actively attempting to stand them down.

It would also be interesting to combine these two approaches: use SANDMAN to apply physical effects of fatigue, but at the same time compare actual flying time to regulatory limits and apply penalties accordingly (by subtracting points or otherwise).

[KnightHawk75]

so there's no easy way to automatically pull the number of crew members for an aircraft type out of CMO.

While you're right in general about QueryDB(), in build ~1147.3x+ (I forget which, november'ish) there is UnitWrapper.crew which should now provide this number, I've not used it much but it seemed to work without issues when I've tried it.

ps. Nice work as usual.

[boogabooga]

I'm not wanting to install all the prerequisites to build from source. I've just downloaded the source as a zip to look at in notepad++. I hope that is okay .

BTW, where can one find SIMEX 2018? I'm not finding it...

musurca, could you please elaborate on what "effectiveness" is actually? How is- say- 86% different than 100%? What value corresponds to a drop in proficiency level?

[musurca]

While you're right in general about QueryDB(), in build ~1147.3x+ (I forget which, november'ish) there is UnitWrapper.crew which should now provide this number, I've not used it much but it seemed to work without issues when I've tried it.

KnightHawk to the rescue as usual -- thank you! I wonder if that number is 0 for UAVs? Will have to do some experimentation.

I'm not wanting to install all the prerequisites to build from source. I've just downloaded the source as a zip to look at in notepad++. I hope that is okay .

BTW, where can one find SIMEX 2018? I'm not finding it...

musurca, could you please elaborate on what "effectiveness" is actually? How is- say- 86% different than 100%? What value corresponds to a drop in proficiency level?

Ha, of course. I've been meaning to rewrite the build script in PowerShell to eliminate the requirement for installing the WSL, but haven't gotten around to it in the extra time I have for "fun" tasks (and I really don't enjoy PS). If anyone wanted to do that I'd happily merge that into the repo.

Here's where I found SIMEX: https://www.matrixgames.com/forums/tm.asp?m=5120464

"Effectiveness" is a metric derived from cognitive capacity and performance in sleep studies, and is related to how people perform a complex task from a baseline measurement. In some studies the subjects were given a difficult task to perform on a computer; in others crewmen attempted to deliver artillery rounds on target.

Currently SANDMAN quantizes that effectiveness percentage and degrades proficiency from the pilot's starting baseline, with a 25% penalty to slightly narrow the window at which the pilot is at absolute tip-top performance. (I may remove the penalty in the future.) While we are working within the limited room of CMO's tiered proficiency system, it's also not so crazy, as the graphs from sleep studies indicate that measurable performance does tend to degrade in tiered stages rather than as a linear function.

Here's what it looks like for an Ace pilot—and I do recommend that you start your pilots at Ace or Veteran, to maximize the levels for quantization:
[center]
[/center]
Beyond that, the raw effectiveness percentage is also the input to the functions that determine current risk of micronaps and crashing. The numbers there are speculative (but, I think, plausible and reasonable) until I find more data.

EDIT: And here's a graph showing how "effectiveness" degrades over time using the model (assuming no sleep whatsoever, and a start at peak awareness of a local time of 1800hrs), which is why I think the tiered approach is actually pretty decent:
[center][/center]

[boogabooga]

How many "micronaps" and what is the crash probability over that time period?