Frank Hunter - Explain the "learning module" please?

[MarkShot]

A little bit about my own background: I design computer systems professionally. I've beta tested a number of well known war games.

Seemingly GOA has a feature that's rather unique, this "learning module".

As far as I know, such a thing hasn't been attempted in commercial games with one exception. There was a game Superpower 1&2 from Golem Labs that claimed to learn and improve the longer the engine played.

Well, I would love if someone could talk more about how this actually feature works? For me, this is perhaps the most novel thing I have heard of in game design for a long time. (I had proposed something similar back in the mid-90s based on AI compliance system under test at a financial exchange where I was manager of the Database Department. The system under test by a group of Los Almos PHd gone private was able to detect anomolies in large collections of data points and essentially learn and catch fraudulant trading practices. The same engine could have been used to develop an evolving set of gaming rules. Alas, this company had bigger fish to fry with there proprietary new technology than PC games.)

Anyway, I think his has convinced me to install this game and give it a test drive.

Thanks.

PS: I am also a Matrix beta tester (in fact it was David who gave me this game as a gift last week at lunch). If need be, I would be happy to hear more about this module and have that covered under an NDA.

[decaro]

I actually bought and played Superpower 1; I think the developers claimed it learned by some type of neural net? Well, I did some learning myself and passed on Superpower 2.

Superpower was very disapointing; cheaply made and marketed, the game never played anything like the real world; it had the most unlikely pairings of nations declaring war on one another. The game also had a tendency to devolve into a global apocalypse.

Maybe I should have kept it longer and given the program more time to "learn."

[FrankHunter]

In a nutshell, as I've mentioned on other threads, it doesn't work, or at least not anywhere as intended.  The learning module was a big part of why I started this game, I wanted to experiment with a learning AI and use it as a test platform for other games.

When we ran the original round of playtesting, which Lava and a few others may recall, the learning module was the major part of the AI but not producing the results I wanted.   How it worked is I simply recorded everything the human did, everything.  What hexes the human chose to attack in every situation, how much strength he maintained in each hex, artillery, HQs etc, the AI also recorded fleet movements, airforce deployments, how the human spent his industrial points and so on.

Up to 1500 games could be recorded and the idea was that AI files could be merged.  So I could merge together a game played by 1500 different people and thus the AI would have knowledge of all those previous situations.  It would also mean the AI would have 1500 different starting situations and opening moves.  Now that part worked fine.  I didn't have to program a "Schlieffen Plan" for example, the AI simply saw me do it and if I was successful it would use it against me in the very next game.  The AI at this time was very aggressive and it was kind of fun running a game just to "teach" the AI and then see it try the same thing on me.

The setups and opening moves were fine.  Where it all broke down was further into the game the AI was not recognizing previous patterns properly and when it didn't have anything to go on, which was most of the time, it broke down.  I spent months trying to get it to recognize a pattern even though it wasn't exactly the same but it was in vain.  So I introduced scripts and had opening moves and such written to dynamic scripts and where a script wasn't available it would use this new AI I was writing.  The results still weren't satisfactory.  The AI would still start off well and gradually get worse and worse as the game went on.

I then dropped dynamic scripts and brought in personalities.  Each personality has a range of attributes, what front it prefers to make its main effort on, what it likes to do naval-wise and so on.  And I added some learning to this too.  So some AI personalities have a higher value than others for doing things like counter-attacking a hex that was just taken, attacking or defending city hexes, defending a hex that is enemy held on 5 sides, attacking bulges in the enemy line and so on.  The learning part comes in the form of weights where things that work are reinforced and things that don't are downgraded.

Personalities also produced unsatisfactory results, I found it better to make them a secondary part of the AI and not the central part of the code.

So eventually, after years of tinkering, I ended up with the current system, a general purpose type AI with some personality that is modified from game to game and a "learning module" that has been downgraded to the point that it now simply produces data for the general purpose part of the AI to bite on.  Basically, the learning module still remembers a lot of stuff but the AI uses that data only to make guesses about where the enemy strength is.  I refrained from allowing the AI access to the recording of the current game even though that did help it quite a bit.  Even with the learning module the AI is almost invariably operating more in the dark than a human player is.


Anyway, the Napoleonic game I'm working on also has a learning module.  And although I haven't been a big fan of using neural nets in the past I have found that it does one thing really well, which is recognize literally thousands of different patterns.  So in that game the learning part of the AI is actually the primary system and the general purpose AI fills a secondary role, filling in only when the learning AI doesn't recognize the current situation.  It hasn't started testing though so I don't know yet how well it will do.

If I was starting GoA over I would go back to a learning-driven system based on human play but using neural nets for pattern recognition and where the current system would serve in a secondary role.  If I use the GoA system for another war that's what I will do.

[Hanal]

Did the learning module actually make it into GoA? I thought it was scrapped back when it was still called TRENCH?

[MarkShot]

Frank,

Well, I have loaded up the game and am starting to get oriented.

I greatly appreciate you spending the time to explain all of that.

So, it appears now that past history basically is used for weighting decisions scaled back from your original goals.

I wonder if part of problem which you encountered was that of using the entire battle space as your class room. I think in reality a game of this scope perhaps has a number of types of battle spaces (situations): like land front, naval supply lines, ... Within those spaces of a small clusters of hexes, you have some standard situations that develop and repeat like: attacks, break throughts, exploitation, encirclement. So, effectively smaller elements of conflict, then need to be crafted into a comprehensive strategy based on alliance priorities and resources at the next level. So, I guess what I am suggesting is that there is, at least, a two tier learning process required. A peephole process that is mainly combat oriented assuming that logistic requirements have been met and a global process that assembles the alliance strategy out of these peephole components.

Well, even if it did not turn out as you had hoped for this game engine, I definitely think it is lofty goal. Effectively reversing the process by which the humans learn to play by watching the AI and flipping it. Besides making the AI adaptive over time, you would have the potential to merge strategies of many players over time and construct a "super" AI. This would address one of the greatest weakness of all single player games is sooner or later the player begins to understand and anticipate the AI. Another nice thing about this approach is that data points could be crunched and rules generated in an offline fashion (compute intensive). However, during actual session play, the derived rules could be fairly static and CPU efficient to follow.

I applaud your vision! Keep at it. I wish I could offer some useful references, but my university days are far behind and my technical background these days is mainly with relational databases and transactional systems.

[FrankHunter]

Did the learning module actually make it into GoA? I thought it was scrapped back when it was still called TRENCH?

Most of it was.  I felt I had to go in a different direction as at the time I felt the AI had hit a wall and was simply not going anywhere.

Effectively reversing the process by which the humans learn to play by watching the AI and flipping it.  Besides making the AI adaptive over time, you would have the potential to merge strategies of many players over time and construct a "super" AI

Exactly.  That was the plan.  I'm sure that learning from human players is the way to go for a game like this, there's just too much variability to rely on FSMs and triggers.  It would have been good to see the AI essentially evolve from playing the styles and plans of me and the beta testers and then adapting to the player and then those files shared so that you could sort of play other player's through their AI files and all the while the AI would be getting better as it adopted the play of the better players.  Perhaps next time it'll work out, its difficult to get it right on the first try at a new system I guess.

[Walloc]

Hi Frank,


Im very intrigued by the learning system and sorry that it didnt work or most of it. At leased to a degree more to ur satisfaction. I hope u are able to get it working in later titles.


Now that we are at the AI. As i said else where i dont think ill be playing much more vs AI at leased. In part to do with lacking time but in part to do with some of my experiences.

First i recogniese at leased i've never met the AI in any wargame that really has challanged me unless it cheats or gets bonus so im not holding that against GoA.
Compared to many other AIs i've know GoAs does a good job of it. I do have some tips on seen behavior for the AI that might be helped rectify.
It seems to me could be very wrong here. Im just speculating in that the AI has different plans it can chose from. Possibly choosen before the game actually starts. Could also have to do with setup plans. Any how thats for u to know and me to ponder on.


That said. Seen from my point of view tho of my above speculations it seems playing the ET side the game would evolve exactly the same in every game. Some 7-8 campaigns played. Which is why playing vs the AI to be becomes less interresting fast.

AI setup with most its german forces to nearly all, at the french border. In AH the AI defends adequatly against the russians, but never do i see any german forces against the russians.
Knowing that. Having lots of cav units stacked with inf as russians. I unopposed take all of east prussia and most of the german territory up until 2-3 hexes from Berlin. Then the AI reacts and in force. Moves just about most its forces from the frenchfront to stop the horde advancing and it certainly does. Problem is now its a cakewalk for the french-british troops and thats the end of the game.
While i agree and understand in that order to be as succesfull as possible vs the french and there is reason for that. Most the german forces needs to be there for a france first campaign. But not leaving any troops at all in the east means that u can advance as russians with out expending any HQ refits or just 1 or 2 to take key cities.
Is there any way to get the german to either faster send its reinforcements aka mobilzation forces to the east front or get just a few german corps's to deploy in the east at key points. To help stop the unopposed russian advance.

I've 2 times i think on purpose not advance in the above manner even if the border was totally clear just to try it out. Go for AH first and such. Non the less in all my ET campaigns the above pattern emerged or could have.
Is there any way if its seen as a problem to make the german AI care just a bit about its russian border. Or curb, in lack of better, the russians use of the cav into clear from inf move?
I understand that juggling the just right amount of force for either front when an AI s faced with such a dilema is not easy. The subtlies in such is better understod by the human mind.
Its just the repeativeness of the situasion that "worries" me the most.

Kind regards,

Rasmus

[MarkShot]

(posted wrong forum)

[MarkShot]

Frank,

With regards to the next game, do you think you now have handle an what kept the "learning module" from working as you planned?

Besides a radical new approach to the AI are there any other significant advantages planned beyond GOA's engine?

When do you think you might roll out such a game?

I ask, since in a way I am different from many of the other customers here. Many here are histo-grognards. Myself, I am more of a systems person looking for an intellectual challenge and elegance in a gaming system regardless of the particular time period and theater addressed.

Thanks.

[decaro]

ORIGINAL: FrankHunter

... And although I haven't been a big fan of using neural nets in the past I have found that it does one thing really well, which is recognize literally thousands of different patterns ...

So that's what a neural net is; I thought Superpower was pulling my leg w/tech talk from Star Trek NG.

I would have thought that a personality AI approach would work well for modeling different nations -- i.e., Germany = "might makes right," France = "ils ne pass pas," etc. -- in a war game as GoA, or any multi-national/global game.

Didn't work for Superpower, though. Maybe AI personalities shouldn't be given weapons of mass destruction!

[MarkShot]

Joe,

Neural nets are one of a set of techniques for achieving AI. The idea is modeling connections between things are scoring answers that come out of random collection of connections. This creates positive and negative feedback loops. As a result, the system begins to internalize a reasoning process without the rules of the system ever being formally stated.

Other expert systems (another name for AI) follow rule based implementations. Like "if THIS, then THAT". The rule base approach are very good for narrow and well defined fields of knowledge. It works well with taxonomies and diagnostic systems. Neural nets are better for pattern recognition and evolving rules that are implicity in large collections of data points. Although a neural net can be simulated in software, they are best implemented by massively parrallel architectures.

Another area of AI is the use of genetic algorithms where answers are tried through some system of permutations with good formulations surviving and bad formulations dying off. I believe this approach is implemented to some extent in BOA and ACW. As for PG engine, I believe a more homespun implementation is used that is very specific to military terrain, objective, and force analysis.

As has been noted here, it doesn't matter what you use, but rather the results you get.

[MarkShot]

I think there was some attempt by MMG to provide for personalities in War3D. However, it never seemed too successful for me.

For example, in real life, Stonewall Jackson, was a great military commander. Thus, in CWBR/TC2M, he is given very high personality marks. However, generally you will find that these high marks causes the AI object Jackson to be totally wreckless on the simulated battlefield and rapidly burn through his forces and precipitate a crisis. Thus, the Jackson personality doesn't seem to reflect the skills and talents of the real life Jackson.

Also, in PG's AA engine commander attributes are parts of the estabs. However, their impact is buried so deeply in the engine that their impact on battles is too subtle for all but the game designers to truly perceive.

AGEOD's AGE also has leader traits, but I don't believe they impact the AI process. Instead they serve as multiplier coefficients that aid or hurt battle outcomes.

[decaro]

ORIGINAL: MarkShot

... AGEOD's AGE also has leader traits, but I don't believe they impact the AI process. Instead they serve as multiplier coefficients that aid or hurt battle outcomes.

I think you're right and I know BoA AI works well; enhanced processing time allows the AI to look at more possibilities, and that seems to work too.

I'm familair w/VB and can understand why a global game doesn't have a lot of "if, then, else" statements; too many possibilities to cover.

That was generic algorithms, right? If not, you're way too advanced for me!

[MarkShot]

Despite my poor spelling and grammar, I meant to say "genetic". I believe that Pocus told me that he used genetic algorithms. Before he got into programming, he was a biologist. So, he tends to look to bio-system analogues for implementation in his game designs and engines.

http://en.wikipedia.org/wiki/Genetic_algorithm

[*Lava*]

ORIGINAL: Walloc

i've never met the AI in any wargame that really has challanged me unless it cheats

Of course...

How can anyone expect a computer game that runs on a home computer to challenge the human brain?

The key is not whether the AI cheats (the AI must cheat to really challenge a human player), it is designing the game in such an elegant way that the cheating is not apparent to the player.

Ray (alias Lava)

[Walloc]

ORIGINAL: Lava

ORIGINAL: Walloc

i've never met the AI in any wargame that really has challanged me unless it cheats

Of course...

How can anyone expect a computer game that runs on a home computer to challenge the human brain?

Well u should read some of the predictions that was made in the 80ies.[:D]
Any how i dont, but then i see 2 ways of dealing with that "problem".
No need to make an AI or put any efford into since it wont succed any way.

Or as Frank apparently tried. Try and go new ways and see how they fare. I assume thats the reasons he had in making a learning module. It was to get further and possibly making a more challaging AI. I support such attempts. If no one tries no progress will happen.
I dont see any reasons even agknowleging the problems AIs have, that should stop u in the efford of trying to improve them.

The key is not whether the AI cheats (the AI must cheat to really challenge a human player), it is designing the game in such an elegant way that the cheating is not apparent to the player.
Ray (alias Lava)

My expereince is that people tend to find out these cheats and see the patterns any how. Defeating the purpose. Again that not necesarily a reason not too do so, but neither a reason why not to keep on improving, within whats possible. Both tech wise and as well as whats possible in view of the limited resources designers/ programmers have.

Kind regards,

Rasmus

[decaro]

ORIGINAL: MarkShot

Despite my poor spelling and grammar, I meant to say "genetic" ...
http://en.wikipedia.org/wiki/Genetic_algorithm

Amazing link!

In the words of Samuel Morse, "What hath God wrought."

[Joel Rauber]

I find AI's to be useful to help learn the rules of the game and various FOW aspects that are in the game.  PBEM is more or less the only way to go for true challenges; not to mention the social interaction involved in gaming.  I've made some good friends through PBEM.

[EUBanana]

Ooo.

I'm about to delve into subjects like multilayered perceptrons and genetic programming very soon on the subject of playing poker, for a uni project.

Unfortunately atm I know nothing... 

[barbarossa2]

Well, the AI may not be perfect, but I have long since stopped viewing the AI as something important to me in a computer game. Since the advent of PBEM and more, it is simply silly to play agains the computer for more than a few games to get up to speed. Then it is out on to the front to play real humans.

While it would be NICE to have a good, learning AI, in the end, all it is doing is attempting to SIMULATE the experience of playing against another human.

Well, people, there are humans out there, with the same game, who would probably enjoy playing against other humans. ;D

Get to it! ;D

I will be buying this game! And playing it against my friends.

-Chris

P.S. I do feel, as one other player commented, that this game appears to be TOP notch for wargaming. However, the interface and screens leave so much to be desired that it cannot attract many cross-over players. Of course, this can always be added in future versions (? ;D and I think this game deserves it!). Rome was not built in a day! And by not delaying purchase until the game is "perfect" we are supporting its future development.