AI Acceptance Criteria

[WIF_Killzone]

Assuming that "all known bugs are fixed" is one condition that must be met in order to release World in Flames, what conditions must be met for the AI to be deemed ready for release?

For example, would it simply be that the AI functions as designed, with no bugs or are there other play-related conditions that must be met and how could you specify these?

[WIF_Killzone]

Forgive me if I am naval gazing, my mind is racing, not a good thing.

The question i think I am asking is how to develop a list of performance indicators or measurement criteria so that Matrix has the right type of information to gage the quality of the AI. i.e. how to determine if its ready for release. This could be be done via a percentage of test cases successful (e.g. We will release the product if 98% of strategic test cases function as designed, etc)

How then will the test cases be developed to test the AI?. In normal software development there would be "UAT" (user acceptance testing) that follows system testing (developer and test team testing). A series of test scenarios comprised of a number of test cases would be created.

An example scenario may be "AI playing Germany attempts to repel invasion of France". A series of pre-conditions would be specified (i.e. France is under german control, Vichy France has been established, germany is at war with Russia, etc, as well as expected results (i.e., Germany does abc, xyz, etc...attempts to repel invasion...deosn't sacrifice russia war to0 much?.. these would all be quite "strategically"broad based scenarios. The test cases then would be more detailed, as well as the pre-conditions, and 'expected results'. Quite a lot of work as you can see. Not to frighten, but I have seen testing budgets as large and larger then requirements, design, and development budgets (for example in banking or insurance where a mistake could prove fatal or cost the company a lot of money) I digress.

Conversely, would test cases simply be created to test and verify the AI's strategic, tactical and operational level "thinking" is working as designed? What would these look like?

I am curious on what the approach will be for testing the AI. Should test planning, scenarios and test case development be started now (if that is the approach that is used), and how could the forum support this work, will additional beta testers be required, has the approach been given any thought yet?

Hmmmmm.

[coregames]

ORIGINAL: WIF_Killzone

Forgive me if I am naval gazing, my mind is racing, not a good thing.

The question i think I am asking is how to develop a list of performance indicators or measurement criteria so that Matrix has the right type of information to gage the quality of the AI. i.e. how to determine if its ready for release. This could be be done via a percentage of test cases successful (e.g. We will release the product if 98% of strategic test cases function as designed, etc)

How then will the test cases be developed to test the AI?. In normal software development there would be "UAT" (user acceptance testing) that follows system testing (developer and test team testing). A series of test scenarios comprised of a number of test cases would be created.

An example scenario may be "AI playing Germany attempts to repel invasion of France". A series of pre-conditions would be specified (i.e. France is under german control, Vichy France has been established, germany is at war with Russia, etc, as well as expected results (i.e., Germany does abc, xyz, etc...attempts to repel invasion...deosn't sacrifice russia war to0 much?.. these would all be quite "strategically"broad based scenarios. The test cases then would be more detailed, as well as the pre-conditions, and 'expected results'. Quite a lot of work as you can see. Not to frighten, but I have seen testing budgets as large and larger then requirements, design, and development budgets (for example in banking or insurance where a mistake could prove fatal or cost the company a lot of money) I digress.

Conversely, would test cases simply be created to test and verify the AI's strategic, tactical and operational level "thinking" is working as designed? What would these look like?

I am curious on what the approach will be for testing the AI. Should test planning, scenarios and test case development be started now (if that is the approach that is used), and how could the forum support this work, will additional beta testers be required, has the approach been given any thought yet?

Hmmmmm.

If Steve does adopt some form of rubric for 'grading' the performance of the AIO in various situations, in my opinion the grade should be based on a personal assessment rather than some sort of strict formalized criteria. The game is just too big to lend itself to easy quantification, and players (especially experienced WiFers) will have the ability to think at a higher level in order to judge the AIO's performance, which may not be reflected in the outcome of a scenario, depending on luck among other factors.