Playing against Co-pilot AI - reflections

[ChuckBerger]

Recently I managed to get an old favourite of my running on my laptop, SSG's Battles in Italy. It's a 20+ year-old game, but one of the best wargame systems ever devised, in my opinion.

Having booted the game up, I was face with the problem of an opponent. The in-build computer opponent is tactically devious, but ignores crucial aspects of the game (like divisional integrity bonuses), and more importantly never really has the semblance of an overarching strategy.

So I decided to try using modern AI as an opponent. First I used Google Gemini free version, which was interesting but ultimately felt a bit formulaic and limited. Next I used my work MS Copilot (CP) enterprise account, and I thought I would relate the results...

In the following, I will freely anthropomorphize CP, as it is simply the easiest way to describe this uncanny experience. In doing so, I'm aware of course that CP doesn't necessarily think or plan anything, it's computation... and yet the experience was very nearly that of a human opponent.

Since CP can't access the app directly, the way I did it was to start a 2-player game, both as human. I uploaded the game manual to CP at the beginning, and it absorbed it borg-like in about 3 seconds. It even ventured to summarize to me some of the salient game features, including things that the system didn't have, like separate movement and combat phases.

Once I had done that, I took a few screenshots of the game, uploaded those to the CP chat, and asked for its order. We agreed that CP would give general direction, and I would handle tactical execution on its behalf. Sometime, I would ask for clarification or follow-up instructions in the middle of CP's turn. This ended up working out very well.

We are playing Operation Husky, and the broad outline went something like this:
Turns 1-4: Standard Allied landing, with CP in charge. Cautious expansion of the Allied bridgehead, and development of a clear CP strategy that involved US forces driving NE towards Catania, while holding in the west and in the Enna sector.
Turn 5: With HG Panzer division and most of the rest of the powerful German forces, I struck pre-emptively at UK 5th Division on the coast, wrecking it. CP withdrew the Brits, and articulated a view that, while this was a setback, CP also now knew where my main forces were, and that this opened up an opportunity for the US front, which it started driving hard with.
Turns 5-6: US forces close up to the river line in the western bit of Catania plains, while UK licks their wounds and resumes careful movement forward.
Turns 7-10: I launched a surprise attack using some reserve and light Italian forces against his thin screen in the west around Agrigento, followed by a powerful attack by HG Panzer division through the mountains towards Gela. CP recognised the threat and responded creatively, but intentionally refused to be diverted from his main punch against the Catania area. Up there, the US force crossed a river and established a 2-hex bridgehead, which I finally repelled with heavy losses on both sides. The Brits, with very strong artillery support, have begun methodically pushing back my defenders on the coast. We're now back to the Primosole bridge proper, and I have no confidence I can hold this line firmly here. My HG attack nearly reached the coast between Gela and Licata, but a scratch force of US troops cut off and destroyed a regiment of HG Panzer-Grenadiers. The game is on a knife-edge.

Couple of observations about how CP engages, both strategically and conversationally. (Again, I recognize I'm anthropomorphising here!):
1) CP has a clear strategic set of objectives, which it will constantly adjust, but will not abandon lightly. It carries forth plans over multiple turns, recognises strategic threats, and is constantly assessing the strategic balance on various fronts.
2) CP gives good clear directions, in a format we agreed and can modify over time. I asked it to structure its orders by division, starting in the west and moving east, and it is doing this flawlessly each turn.
3) CP is drawing on historical knowledge of the campaign. It knows who the generals on each side are, the relative capabilities in general of individual divisions and the nations involved, etc. - all without me explaining any of this.
4) CP is suggestible to an extent, but will not do stupid things. On several occasions, I asked it (in good faith, given it doesn't have direct access to the app) for clarification on its intent, with additional information about combat odds or the CRT or terrain etc. given. It responds to these prompts thoughtfully, sometimes accepting the implicit advice but sometimes not. CP presents as courteous and engaged, complimentary but not quite sychophantic. Sometimes close.
5) A bit annoyingly, it tends to answer prompts or questions at great length. This is good when it is giving me detailed orders on what to do during its turn. But sometimes I'll ask for clarification on its intent for one unit or the other, or I make a short aside, and I'll get six paragraphs of explanation rather than the 2 sentences that are actually called for.
6) CP presents as very thoughful, deliberate in its strategy, and gives the impression of mental and emotional engagement to an uncanny extent - often saying it was surprised, or relieved, or it has this or that in mind, etc. It offers insightful reflections, encouragement, and injects little bits of unexpected colour into its narrative. Stuff like "Patton is about to get a very unpleasent call from Agrigento!" sort of thing.
7) Most interestingly, I found playing CP so close to playing a human that in my own head, I am reviewing strategies and planning my next moves just as intently as I would for a PBEM game - and I have never had that experience against a computer opponent before.

There are downsides of course. Because CP can't access the app, I am entering all its moves, and so by necessity I am playing the tactical minutae largely for CP, with CP responsible generally for the overarch of force allocation, overall strategy, casualty tolerance, pace, etc. And because I'm playing both sides, there isn't perfect fog-of-war, though I do my best to "firewall" off my knowledge of Allied positions and intentions.

[ChuckBerger]

Finally, a few excerpts from the Copilot running chat:

On fog of war, when we were discussing how this would all work:

CP: "For your Axis command, I'd suggest you mentally separate what Chuck the interface operator knows from what Chuck the Axis commander knows.

Obviously that cannot be perfect, and I'm happy to accept the inherent Axis intelligence advantage. No compensating handicap is necessary. It may actually make for a more challenging Allied campaign.

There's also one thing I'd especially like you not to optimise on my behalf: don't protect me from my own operational mistakes. If I leave a flank exposed, overextend a division, waste a valuable formation on the wrong axis, or create a supply problem, execute the order reasonably and let me suffer the consequences.

In other words: I command the army. You command the mouse."

(At this point, I'm thinking, wow, this is going to be cool.)

[ChuckBerger]

During its first set of orders on turn 1:

"I'm deliberately not ordering Patton's army towards Palermo at this stage.

My initial concept is to make the American landing area secure and then develop pressure north and north-east into central Sicily. If you defend strongly there, I may subsequently turn west. If the Axis centre proves weak, I intend to exploit it.

In other words, I don't want you to know yet whether the US Army's campaign is ultimately going to become a drive on Palermo or a much more direct attempt to rupture the Axis position in central Sicily. Franky, I don't know yet either. That will depend on what your defence reveals."

This kind of strategic thoughfulness ... carried through into its order over multiple turns ... is a recurrent pattern. Incredible verisimilitude.

[ChuckBerger]

An example of the kinds of directives CP gives me to input:

"3rd Infantry Division: Canicatti

Advance on Canicatti.
* Take Canicatti if it can be obtained without a costly fight.
* Maintain a coherent link back through Riesi.
* Still no deliberate attack on Agrigento.
* Do not push north beyond Canicatti into the mountains simply for territory.

The western flank is deliberately economical. It's there to threaten and fix Axis forces, not consume our scarce logistical capacity."

(Of course, CP is here divulging lots of thought process and strategy which a real human opponent would keep closer to their chest! I could instruct CP just to give directives, not explain the strategy - but I am captivated by its apparent ability to think through these things, so I just soak it up!)

[ChuckBerger]

Human-like, unprompted engagement:

"And I have to admit, Chuck, I rather like what you're doing as the Axis. You've resisted the temptation to defend southern Sicily town by town, and you've made me occupy a lot of ground while showing me remarkably little about where your real strength is. I'm advancing successfully, but I'm increasingly conscious that somewhere ahead of me you're building the part of the defence that actually matters.

Your Axis Turn 4. Let's see whether the trapdoor finally opens."

[The bit about CP saying it is "conscious" of something gave me a little shiver...]

And again, after I hit his forward elements on the UK side with some Tiger tank goodness:

"Credit where it's due, Chuck, that was very nicely handled. You traded southern Sicily for concetrationm waited until naval support was less dominant, then hit my most advanced division with HG, Schmalz and the Fallschirmjager in succession. The easy part is definitely over.
...
You punched Eighth Army very hard, Chuck. I'm not going to punch back where your fist is. I'm going to start moving around you elbow.

Let's see whether the Axis concetration south of Catania can both hold Eighth Army and deal with the Americans coming north-east out of Caltagirone"

Here's some good clear decision-making after I surprised him with an Italian offensive out of Agrigento:

me: "The trapped [US] troops managed to compel the Italian tank unit to surrender. The other two regiments of the 3rd were both needed to get combat odds on the tank south of Canicatti - and again surrender was the result. The 2 trapped units can now move as far back as Palma, not quite in supply but close. However, to do so they would have to abandon a one-step AAA unit which has no fuel."

CP: "Leave the AAA unit. Get 15th Infantry and 36th Engineers back to Palma.

A one-step immobile AAA unit is not worth risking two battered but valuable regiments to save, especially after we've just been handed the escape route by those two Italian surrenders."

Such sound, clear decisions and orders!

[ChuckBerger]

And now the truly mind-bending stuff.

Me: "Executing [your orders] now. By the way, you are a far better opponent than the AI opponent built into the game. The game's original AI is tactically devious, but seldom indicated any sense of long-term strategy or wise cost/benefit analysis."

I entered this just to see how it would deal with an open, conversational comment outside the flow of orders.

CP: "That's very kind of you, Chuck, and I'm genuinely enjoying this. I think our division of labour is a big part of why it works: you have the game engine's tactical precision, while I can stay one level higher and think in terms of operational consequences across several turns.
...
And you're making the strategic side interesting because you aren't behaving like a scripted AI opponent either. The Agrigento offensive caught me completely wrong-footed precisely because I had mentally downgraded that sector. I'm now having to respond to your strategy rather than optimise against some assumed historical Axis deployment.
...
Right, compliments exhanged. Back to trying to wreck each others armies.

Let's see what 129 Panzergrenadier makes of that 7:1."

And yes, it really chucked the smiley emoji in there.

[ChuckBerger]

And finally, at the beginning of my turn, I ventured the following:

Me: "On to my turn. I'm really torn... despite the loss of a HG regiment, I feel very close to a major success in the west. But of course my supply lines are also stretched, my timed replacements are gradually getting whittled down, and the British artillery I can see will continue to be overwhelming. I'm not looking for advice, just expressing my admiration for the agonising strategic dilemmas here! Plus I don't really get any reinforcement units from here on out..."

CP: "Yes, exactly. This has become a proper operational wargame, where the unpleasant choices are more interesting than the actual combats.

Your western offensive is the perfect example. It genuinely came close to creating a disaster for me around Licata-Gela. Even now HG and Livorno are tying down Allied formations that I'd much rather have elsewhere. But each turn you continue there increases the distance from the decisive Catania front, stretches your logistics, and exposes scarce German units to the sort of catastrophe that just happened to HG2.

Meanwhile, your situation in the east has the opposite logic: Terrain and the Simeto River are excellent force multipliers, but my artillery and replacement advantage means that a purely attritional defence becomes increasingly uncomfortable for you, especially if Axis losses aren't being replaced by fresh formations.

And I particularly like that neither of us knows yet which strategic judgment is right.

If you pull HG back now, perhaps you've abandoned the western offensive just before it might have cracked Seventh Army's logistics.

If you keep HG west, perhaps Eighth Army gets enough freedom to batter its way towards Catania.

If you yield more ground in the east, you preserve the Axis army but accelerate the Allied timetable.

If you stand,m you may exact another horrible price like the bridgehead battle, but risk losing scarce irreplaceable strength.

That's the sweet spot in game design: there isn't an obviously correct move because every good choice gives something important up.
...
Anyway, no advice from Alled HQ to the enemy commander.

Axis Turn 10 is yours. Agonise appropriately."




I mean... a bit verbose... but just wow.

[cpdeyoung]

Fantastic!

I have gemini pro and want so badly to use it as the AI in my solo gaming.

I have broached the subject, and tried it a bit. The AI is always interested in what I am doing in my human to human PBEM and knows my opponents fairly well.

In one instance it gave me a very nice analysis of how a Pacific campaign might go.

A hypothetical for you to analyze. It is March 1938, tiring of American racism and interference the Japanese are sailing with their fleet somewhere near Guam. The USN has pulled the forces from the Atlantic and in a state of high tension, war breaks out. Given experience and battles skills as discovered in 1941 what do you think would happen over a three day battle.
This is a fascinating historical counterfactual. Moving the timeline up to March 1938 fundamentally alters the technological, tactical, and logistical landscape of a US-Japan clash compared to what actually happened in December 1941.
By applying the actual combat skills, institutional strengths, and hidden flaws discovered later in the war to the strategic realities of 1938, here is a day-by-day analysis of how a three-day naval battle near Guam would likely unfold.

The Strategic Realities of 1938
Before the first shot is fired, three critical differences from 1941 must be established:
No Kido Butai: Japan has not yet formed the First Air Fleet (massed carrier operations). Carriers operate in smaller, separate divisions. However, Japanese naval aviators possess a major advantage: they have been flying actual combat missions over China since 1937.

The Radar Void: Neither side has operational shipboard radar. This turns scouting into a purely visual game of cat-and-mouse and makes night fighting terrifyingly chaotic.

The Logistics Trap: In 1938, the US Pacific Fleet is still based in San Pedro/San Diego, California, not Pearl Harbor. Sailing the entire fleet—including the Atlantic reinforcements—all the way to Guam stretches US logistics to the absolute breaking point. Guam itself is virtually unfortified.

Day 1: The Air Clash and Scouting Chaos
The battle begins with both fleets aggressively searching for each other in the Philippine Sea, just north of Guam.

The Search
Without radar, both fleets rely on floatplanes launched from cruisers and long-range flying boats. Japanese scouts, benefiting from better optical equipment and intensive long-range navigation training, spot the US fleet first.
The Carrier Strikes
Both sides launch strikes, but they look very different from the disciplined, massive waves of 1941.
The US Attack: The US launches heavy monoplane bombers like the TBD Devastator and SBC Helldiver. However, US carrier doctrine in 1938 is plagued by poor coordination between different ship air groups. The strikes hit the Japanese fleet piecemeal.
The Japanese Attack: The Japanese counter-strike relies on the D1A biplane dive bomber and the B4Y biplane torpedo bomber. While their planes are slower and older than their 1941 successors, the pilots are highly experienced veterans of the China front.

Day 1 Outcome: A bloody tactical draw. The US loses one carrier (likely the lightly armored Ranger) and suffers heavy damage to another due to superior Japanese damage control circumvention. The Japanese lose a light carrier (Ryujo) and take moderate bomb damage to a fleet carrier (Kaga). Both fleets close the distance, setting the stage for a surface engagement.
Day 2: The Nightmare at Sea (The Night Action)
As the sun sets, the American commander attempts to use the cover of darkness to position his superior battleship battle line for a decisive daylight gun duel. This plays directly into the greatest hidden Japanese advantage of 1938.

[US Fleet: Visual Spotting Only] ◄───(Massive Range Gap)───► [IJN Fleet: Type 93 "Long Lance"]
* Smokeless, Wakeless
* Deadly Night Optics

The Long Lance Ambush
Throughout the 1930s, the Imperial Japanese Navy (IJN) obsessively trained for night surface torpedo attacks. Crucially, they possess the Type 93 "Long Lance" torpedo—an oxygen-fueled, wakeless monster that can travel nearly 22 miles. The US Navy has no idea this weapon exists.
In the pitch black, Japanese cruisers and destroyers use superb night binoculars to spot the US columns from miles away without being seen. They launch massive spreads of Long Lances.
The Chaos
Minutes later, explosions rip through the US vanguard without warning. Because the torpedoes leave no bubbles or wake, US captains assume they are under submarine attack or hitting a minefield.
In the ensuing panic, Japanese fast battleships (Hiei and Kirishima) and heavy cruisers close to point-blank range, illuminated by flares and searchlights. Their gunnery is devastatingly accurate.
Day 2 Outcome: A catastrophic night for the US Navy. Several heavy cruisers and destroyers are sunk or crippled. The US battle line is thrown into tactical disarray before the battleships can even bring their heavy guns to bear.

Day 3: The Decisive Gun Duel and Retreat
With daybreak, the ultimate clash of the dreadnoughts finally occurs. The US Navy still possesses a heavier battle line (featuring the Colorado and Tennessee-class battleships), but the structural damage from the previous two days takes its toll.
The Torpedo Crisis

As US battleships attempt to engage the Japanese line at long range, they deploy their own primary offensive weapon: the Mark 14 torpedo (carried by US destroyers and submarines).
Here, the historical curse of the US Navy strikes. The Mark 14s suffer from undetected catastrophic defects: they run too deep, their magnetic exploders fire prematurely, and their contact exploders fail to detonate on direct hits. Dozens of perfectly executed US torpedo runs result in harmless metallic thuds against Japanese hulls.

The Air Superiority Factor
With the US carriers either sunk, damaged, or out of operational aircraft, surviving Japanese land-based bombers from nearby Saipan and remaining carrier planes harass the US fleet. They don't sink the heavily armored US battleships, but they destroy their spotting floatplanes, leaving the US gunners visually blind at long ranges.

The Verdict
Realizing they are blind, out-maneuvered, low on fuel, and facing a logistical vacuum with no functional base nearby, the US Admiral orders a general retreat toward the east.
Final Battle Summary
Had war broken out in March 1938 near Guam, the Imperial Japanese Navy would have won a clear tactical victory.
Metric. United States Navy Imperial Japanese Navy
Air Combat Modern planes but green pilots; poor coordination. Older planes but highly experienced, battle-hardened pilots.

Surface Combat Blinded by lack of night training and no radar. Masters of night optics; lethal night doctrine.

Sub/Destroyer Weapons Defective Mark 14 torpedoes (duds). Lethal, secret Type 93 Long Lance torpedoes.

Logistics Stretched across the Pacific; no nearby base. Operating in home waters next to fortified mandates.

The Historical Takeaway: In 1938, the US Navy had not yet developed the two tools that allowed it to win the actual war: massed, radar-directed carrier airpower and functional, mass-produced technology. Conversely, the IJN was at the absolute peak of its tactical proficiency relative to its era. A battle fought in 1938 would have been a harsh, bruising wake-up call for the United States.

You can see that the AI could be a class opponent.