Test Plan outlook...

[Greyshaft]

Beta Testers
At this point I'm not making any decisions about who is in or out and it wouldn't matter if I did since Steve makes all those decisions anyway. Stay tuned.


Test Traceability Matrix
What I can share with you is some information on how we are going to test this sucker. For those of you who have never used a Test Traceability Matrix (TTM) I will explain how you build one of these beasts.

STEP 1 - Create a specification detailing what you want the application to do. OK... lets start with the WIF:FE ruleset.

STEP 2 - Divide ruleset into modules. I'd already started doing that before I was distracted by this d@mn fool idea of writing descriptions for the air units. See the attached file - its actually a MS Excel renamed with a .txt extension to fool the Matrix cybercops, so download the file then change the .txt to .xls and open the spreadsheet. Each page in the Excel file is a seperate module. Go to page '4.100-Force pool changes'.

STEP 3 - Identify individual behaviours and outcomes you expect from each module thats the blue text in the left hand column eg 'Having force pools for each of your unit types lets you select the type of unit you want to build.'

STEP 4 - Document a test that proves/disproves that the behaviour works as expected within the application. - Thats the 'Test Cases' underneath the behaviours listed in that module. There's nothing high-tech about it - you just read the behaviours and ask yourself how you would prove that MWiF does or doesn't exhibit those behaviours. If it passes the Test then write 'Pass'. If it fails the test then write 'Fail' with a comment eg 'my fighters abort as expected but if I abort a bomber then I get a system crash'. Lots of work still to be done here.

STEP 5 - Provide a method of summarising the status of the current Test programme. - As a Tester reports that their modules are tested I can tick off those modules on the 'Rules Index' sheet. There are three possible categories "Not yet Tested", "Test passed", "Test failed"

STEP 6 - Provide a mechanism for changing the specification. - Given that Steve has already made some changes to WiF:FE to squeeze it onto a PC then I need to start changing my TTM already. Steve will need to review my changes to be sure I am reflecting his reality and then Testers can read the TTM to ensure they know what MWiF should be behaving.

STEP 7 - Rinse and repeat for every release of the software.

Easy, huh ?

It is important to document the test for the following reasons:
* it creates a record of what you did to identify a bug.
* it allows ongoing refinement of what steps you perform to do the test. If you rely only on memory you might not set up the test the same way each time.
* it allows other testers to examine your testing style and perhaps learn a trick or two (and vice versa).
* someone else can pick up the work if the original tester is unavailable.
* it allows the Test Co-ordinator an effective method of determining whether the application has been fully tested after each release[/i]

A hidden bonus in using a TTM is that at the end of the Testing Phase you have already written 90% of the rulebook.

In between all that work the Testers should play the game to their Hearts Content and beyond.

There will be a phased release of functionality so don't expect to play the Grand Campaign straight away[:-]

Update on Air Commentaries while I'm here...
The Air Comments data is transitioning from Alpha to Beta. I have adequate descriptions for all but a dozen of the c.530 air units and those that are 'missing' are either incremental upgrades (eg engine power went up by 100 hp but no other changes) or unknown aircraft (Z.1024 or 'Mixed' counter) with maybe three or four genuine bona-fide head scratchers where the counter description doesn't match reality. The minimum data I am aiming for with the descriptions are engine horsepower, top speed and details of weapons carried plus an interesting anecdote or piece of trivia. My current work involves reviewing the entries and fixing typos and other inconsistencies and I'm hoping to get that done over the Christmas break. After that I'll post the list for public feedback and I'll get seriously into organising the beta test workload.

[Shannon V. OKeets]

To Greyshaft's fine post I would like to add some bits that I will need.

The specifications of the system you are running to play test the program.

CPU - manufacturer, model, and speed
Main memory
Disk memory - total and available/free
Monitor - screen size and resolution
Graphics card - manufacturer, model, and memory
Keyboard - standard or something special?
Mouse - manufacturer and model (# of buttons, scroll wheel)
Operating system

You only have to do this once, when you sign up. That way I'll have it on file so if something very strange happens I can see if maybe it has to do with how the program runs under your configuration. If you have several machines, you can label them A, B, C,... and report any problems as occurring on machine B (or whatever).

Don't sweat the details too much. Ball park estimates are fine. I hope I never need to even look at this information, but if you have a problem that no one else has, then it is likely to be something about the configuration.

And, .... don't do any of this now. Wait until Greyshaft asks for your help play testing. After all, maybe Santa has a new computer system in his bag just for you!

[Froonp]

OK, some words after the CWiF testing experience I had.

STEP 7 - Rinse and repeat for every release of the software.

During the CWiF era, Chris released new versions almost daily during some periods of intense activity.
Your straight & forward testing plan will also need from the programmer to be as straight & forward as the testing plan.
If Steve releases new versions too quickly, there will be problems with testers bored of testing the same things twice the same day for example, and finally they will not test as fully as they should.
Maybe this straight & forward way of testing can be done only for the real "new versions". To be defined, but with Chris, the released version 0.7.1 to 0.7.71 -- for example -- where the same version with lots of subversions, and were following the 0.6.1 to 0.6.x versions. Ful testing could occur at 0.6.1, 0.7.1, but not between 0.7.1 and 0.7.2 because they were only a few hours in between sometimes.

Easy, huh ?
A hidden bonus in using a TTM is that at the end of the Testing Phase you have already written 90% of the rulebook.

Isn't the rulebook already written ? RAW 7 ?

There will be a phased release of functionality so don't expect to play the Grand Campaign straight away

Could you clarify ? I do not understand fully this expression : "phased release of functionality"

Update on Air Commentaries while I'm here...
The Air Comments data is transitioning from Alpha to Beta. I have adequate descriptions for all but a dozen of the c.530 air units and those that are 'missing' are either incremental upgrades (eg engine power went up by 100 hp but no other changes) or unknown aircraft (Z.1024 or 'Mixed' counter) with maybe three or four genuine bona-fide head scratchers where the counter description doesn't match reality. The minimum data I am aiming for with the descriptions are engine horsepower, top speed and details of weapons carried plus an interesting anecdote or piece of trivia. My current work involves reviewing the entries and fixing typos and other inconsistencies and I'm hoping to get that done over the Christmas break. After that I'll post the list for public feedback and I'll get seriously into organising the beta test workload.

Please, remember that a lot of data can be found in the Planes in Flames rulebook (the booklet that is accompanying the PiF kit, but who is not in the WiF FE box). If you need one, I have loads of it and can send you one.

Best Regards

Patrice

[Greyshaft]

ORIGINAL: Froonp
Maybe this straight & forward way of testing can be done only for the real "new versions". To be defined, but with Chris, the released version 0.7.1 to 0.7.71 -- for example -- where the same version with lots of subversions, and were following the 0.6.1 to 0.6.x versions. Full testing could occur at 0.6.1, 0.7.1, but not between 0.7.1 and 0.7.2 because they were only a few hours in between sometimes.

That's exactly how it will occur.

Isn't the rulebook already written ? RAW 7 ?

Alas no. There are already significant differences between RAW 7 and MWiF such as the process of naval interceptions. These must be documented within the Test plan so the Testers have an understanding of how MWiF should function

Could you clarify ? I do not understand fully this expression : "phased release of functionality"

First Test release will be Barbarossa with no naval activity. We test that scenario, fix bugs and then move onto the next major release which will include... [WE INTERRUPT THIS POST TO ADVISE YOU THAT GREYSHAFT HAS BEEN SENT TO A RE-EDUCATION CAMP WHERE HE IS WRITING OUT 'I will not talk about software releases without prior approval from Matrix' ONE HUNDRED TIMES.]

Please, remember that a lot of data can be found in the Planes in Flames rulebook (the booklet that is accompanying the PiF kit, but who is not in the WiF FE box). If you need one, I have loads of it and can send you one.

Thanks. I also have the PiF rulebook but I'm looking for a lighter touch than a whole bunch of technical height/weight/length data. I want the commentaries to produce a 'Wow! I didn't know that! I'm going to look up some more information on that plane' kind of response from the reader. The technical stuff I am including (hp, speed, armament) is really secondary and is only there to let the players make superficial comparisons between the aircraft.

[Shannon V. OKeets]

ORIGINAL: Greyshaft

Isn't the rulebook already written ? RAW 7 ?

Alas no. There are already significant differences between RAW 7 and MWiF such as the process of naval interceptions. These must be documented within the Test plan so the Testers have an understanding of how MWiF should function

I am not so sure about "the process of naval interceptions" being different, but that can be dealt with later. Greyshaft's main point is correct. There are differences between RAW and MWIF. A few simple ones:

Disrupted units do not 'flip' in MWIF.

There is no 'production circle' per se.

Fog-of-war has been added as an option.

Task forces, as defined in RAW, don't exist in MWIF.

There are no off-map boxes.

The movement costs are the same anywhere on the map (supply lines too).

As to how the rules will be presented to the players, I still am toying around with several ideas. A copy of RAW 7 at hand seems obvious. Probably a full list of differences between RAW and MWIF, for quick reference by experienced WiF players. Something very simplistic along the line of a tutorial for players new to WiF and/or new to IGO-UGO computer war games in general. Greyshaft's definitive list of how each section of the rules are intended to work - as an appendix - also seems appropriatte to include.

[Greyshaft]

ORIGINAL: Shannon V. OKeets
As to how the rules will be presented to the players, I still am toying around with several ideas. A copy of RAW 7 at hand seems obvious. Probably a full list of differences between RAW and MWIF, for quick reference by experienced WiF players.

My main concern is that testers are given a clear understanding of what they are testing at any given time. I have bad memories about projects I have inherited where the Test Team have reported that they have completed the testing and then I later found out they were working from an out of date test plan. They had actually reported some code changes as bugs even though the client had requested those changes and the programmers had incorporated those changes correctly into the application.[:@]

I would like us to be working from a Single Source of Truth for the test programme. Here's my vision:

1. Steve makes code changes, plays some golf, makes some more code changes, finally releases some code for testing along with some explanation about what he has released Ground strikes don't work yet folks and just ignore it when the Panzers turn purple... I'll fix that one next time.

2. Some volunteer (probably me) updates the Test Traceability Matrix with that information. If Steve has given me advance warning of the changes then the TTM is available the same day as the code. Otherwise maybe a day or two later.

3. Testers download the updated code and TTM and spend time doing what testers do and the Test Forum is filled with the sound of aircraft and computers crashing. As the days go by the Testers report their assigned modules as PASSED or FAILED (and here's the bug report) and then they go on having fun. If you find a bug in someone elses module then raise the bug yourself and advise the someone else so they can include that bug in their module's Test Plan for the next release. Running a Test module should take maybe 10-15 minutes plus documenting any bugs you might have found.

4. If testers have questions about the rules of WiF - What's the difference between ATA and ATS factors? - then they ask in the Forum or PM their mentor if they have one. If testers have questions about how MWiF implements WiF - Hey! Where did the off-map boxes go? then they can turn to the TTM which will contain the consolidated updated rules and discover that 'off-map boxes' no longer exist.

Make no mistake - this will an extremely difficult games to test. Grognards like Terje and Neilster will have to forget many of their finely honed WiF instincts - The optimum Chinese defensive line starts in this city and curves around that mountain before... Hey! where did all those hexes come from? - while new players have to learn all of the WiF system. Remember the last time you introduced a virgin into your WiF group? How many nights of playing did it take before they could run their own country? Our challenge is to support those players effectively while they are getting up to speed.

Of course, my vision might just be a Sauvignon-induced optical illusion

[Frederyck]

I have worked professionally as a project manager for software testing (telecommunications, not gaming), and I can say that keeping everybody in line and making them test the correct versions at all times can be very problematic. It takes a lot of effort and there will be mistakes made.

That being said, I think that your outline looks very fine, and with enough dedicated gamers/testers this will be a lot less of a bumpy ride than it could be,

[Greyshaft]

ORIGINAL: Frederyck
I have worked professionally as a project manager for software testing (telecommunications, not gaming), and I can say that keeping everybody in line and making them test the correct versions at all times can be very problematic. It takes a lot of effort and there will be mistakes made.

My role is not to keep people in line. My role is to provide the tools to help them be more effective at testing MWiF. Testers will need to provide their own self-discipline. I can't guarantee that there will be no mistakes made. However I can guarantee that without a Test Plan there will be a lot more mistakes made.

[Shannon V. OKeets]

ORIGINAL: Greyshaft

ORIGINAL: Frederyck
I have worked professionally as a project manager for software testing (telecommunications, not gaming), and I can say that keeping everybody in line and making them test the correct versions at all times can be very problematic. It takes a lot of effort and there will be mistakes made.

My role is not to keep people in line. My role is to provide the tools to help them be more effective at testing MWiF. Testers will need to provide their own self-discipline. I can't guarantee that there will be no mistakes made. However I can guarantee that without a Test Plan there will be a lot more mistakes made.

Yeah, the play testers are an all volunteer army, so making someone do something isn't in the cards.

As for mistakes, one of my personal adages is "Perfection is an elusive goal."

[Neilster]

Grognards like Terje and Neilster...

Please, I'm no WIF grognard. I know a reasonable amount about WW2. That's all.

Cheers, Neilster

[Frederyck]

ORIGINAL: Greyshaft
My role is not to keep people in line. My role is to provide the tools to help them be more effective at testing MWiF. Testers will need to provide their own self-discipline. I can't guarantee that there will be no mistakes made. However I can guarantee that without a Test Plan there will be a lot more mistakes made.

Oh, I understand that, but *someone* needs to be "in charge" of the testing, otherwise a lot of things might easily be done needlessly. However, the people testing this software (of which I hope to be one) are doing it because they want to, not because it's something that pays the bills. And that is a big difference!

[Shannon V. OKeets]

ORIGINAL: Frederyck
However, the people testing this software (of which I hope to be one) are doing it because they want to, not because it's something that pays the bills. And that is a big difference!

That's a big bonus in my opinion. The play testers want the game to work certain ways so they can "see what is happening", "get things done", and, most importantly, enjoy playing the game. Their critiques might be as amatuers in terms of 'play testing professionalism' but that is a trivial matter compared to the fact they want to seriously play the game for hundreds of hours. Their comments on the game interface are as important to me as their detection of program bugs.

[Frederyck]

ORIGINAL: Shannon V. OKeets
Their critiques might be as amatuers in terms of 'play testing professionalism' but that is a trivial matter compared to the fact they want to seriously play the game for hundreds of hours. Their comments on the game interface are as important to me as their detection of program bugs.

I agree and understand. Finding bugs will come as a matter of fact when exploring the game for most people - a sort of added bonus in the feedback to the developer(s). [:)]

It is a world of difference when you are testing database accessing (or whatever) and you personally care that the result is good!

/Carl-Niclas

[buckyzoom]

Just my two cents, but I think it might be helpful to the developers if the person reporting the bug can explain how to produce it. It's always been a little amazing to me how differently people can go about doing the same thing. One person might use a menu, another a mouse click, a third the shortcut key and a fourth will drill into it from a separate application. In some cases the bug will not show up if you enter window x, but will show up if you first go to window y then to window x, or only occurs if you have your handy-dandy excel WIF 2d10 probability chart open in the second monitor while playing CWIF, or whatever... I've beat my head against keyboards long hours trying to reproduce relatively critical problems because I didn't have the right sequence.

[buckyzoom]

I'm assuming as a beta tester you're primarily looking at ease of use, accuracy and robustness of the game. Did you want any system performance metrics? CPU utilization, IO wait or memory info? Are you using flat files or a database to persist the data? If a DB which one?

[Shannon V. OKeets]

ORIGINAL: buckyzoom

I'm assuming as a beta tester you're primarily looking at ease of use, accuracy and robustness of the game. Did you want any system performance metrics? CPU utilization, IO wait or memory info? Are you using flat files or a database to persist the data? If a DB which one?

Yes.

Performance metrics would also be nice, but they are really only important if the response rate goes south.

"persist the data"? The noun 'data' as an object of the verb "to persist" is new to me. Could you please rephrase.

[Greyshaft]

ORIGINAL: buckyzoom
Just my two cents, but I think it might be helpful to the developers if the person reporting the bug can explain how to produce it.

Absolutely! Part of my role is to assist testers to use standardised formats in reporting bugs. The benefit is twofold - first, the standard bug report prompts the tester to fill in ALL the data required by the developer and second, the storing of multiple test reports in the same format helps the Developer and the Test Co-Ordinator to better classify bugs and to see patterns in bugs ... oooh !!! look at the pretty Blue Screen of Death you get whenever you call that module

[buckyzoom]

Hmmm, I may have been being a little obscure.

"persist the data"? The noun 'data' as an object of the verb "to persist" is new to me. Could you please rephrase.

I was wondering how you are going to be storing game information (unit statistics, location on the board, game history and so on). From my limited experience with e-games it seems its pretty common to use flat files (eg - a '.dat' file). I didn't know whether MWIF might be shipping with an embedded database like Derby or MySQL or some other commercial product. Just curious.

As for the phrase, "persist the data", I don't know if its how humans speak. I know we (where I work) talk a lot about data persistance on projects. It's our way of avoiding saying, "How we make sure numbers not go way?!?", and when I talk like that it makes my Mom feel better. She has this thing about me having wasted my college education drinking and chasing women of questionable character.

Gotta go...

[Shannon V. OKeets]

ORIGINAL: buckyzoom

Hmmm, I may have been being a little obscure.

"persist the data"? The noun 'data' as an object of the verb "to persist" is new to me. Could you please rephrase.

I was wondering how you are going to be storing game information (unit statistics, location on the board, game history and so on). From my limited experience with e-games it seems its pretty common to use flat files (eg - a '.dat' file). I didn't know whether MWIF might be shipping with an embedded database like Derby or MySQL or some other commercial product. Just curious.

As for the phrase, "persist the data", I don't know if its how humans speak. I know we (where I work) talk a lot about data persistance on projects. It's our way of avoiding saying, "How we make sure numbers not go way?!?", and when I talk like that it makes my Mom feel better. She has this thing about me having wasted my college education drinking and chasing women of questionable character.

Gotta go...

I have a very plebian approach to data storage. Always have and always will. I like to be able to see the data and modify it using a text editor, if need be. Therefore, the saved games will be in comma separated value files (CSV). However, when playing against an opponent where you are not suppose to see his position (fog of war) or standing orders (PBEM), I will encrypt those files and they will only be viewable using the game and knowing the password(s). The game histories I might compress to save on disk space. I'll see how big they get.

The map and unit data will be in CSV files too, so the players can edit those before a game starts, to add new units or change unit values. WIF is trickier than most games in that setting up a game is a very complicated and complex process: who is at war with whom, which countries have been conquered/liberated, which hexes of countries are held by the enemy, where can naval units be set up, ships that have been sunk, ships that have been replaced (upgraded), trade agreement, neutrality pacts, chits that have been picked - how they were used, US entry options selected, US entry actions already rolled for, ...

That means that players are not going to have the ability to create their own scenarios. They can change unit strengths and ranges, et al, even modify the map terrain to a limited extent. But no "let's just put these units here and fight a battle" type stuff.

[buckyzoom]

I have a very plebian approach to data storage

I like flat files too. There's nothing plebian abou them. How's it work when you a move a unit? There's a phase to everything in WIF and during every phase it seems like you go through a cycle of 'what if' type exercises. I'm sure you're familiar with it OK I could move these units here and attack this hex, or I could move these units here and attack this hex. Over a map you end up moving the pieces to a location, and then moving them back to their starting spots and doing this over and over. How will this process in the game?

Will you be able to rollback or 'undo' moves? Will it be a single unit at a time thing so that once you commit a move there's no going back? Will you be able to rollback to a previous phase and remove it? (I'm probably going to follow up on your answer and ask what part of the moves sequence is written to file and when.)

Hey, if this is too many questions, just say you see!

Merry Christmas!