Building the MWiF Test Plan

[pak19652002]

All the talk of bug reporting, spreadsheets, google, etc. here and on the test forum is blowing my mind! Would it be possible for Steve or his delegate to provide THE definitive format/system for the testers to follow? I'm so confused, I don't even know if this has already been done! I know there was a really long format proposed recently so a spreadsheet can be built. Should I use that? But, then there is this google thing.

The grunts just want the bosses to give us a form to fill in and not have to think about it too much...

Pvt. Pete

[Shannon V. OKeets]

ORIGINAL: pak19652002

All the talk of bug reporting, spreadsheets, google, etc. here and on the test forum is blowing my mind! Would it be possible for Steve or his delegate to provide THE definitive format/system for the testers to follow? I'm so confused, I don't even know if this has already been done! I know there was a really long format proposed recently so a spreadsheet can be built. Should I use that? But, then there is this google thing.

The grunts just want the bosses to give us a form to fill in and not have to think about it too much...

Pvt. Pete

For now, the 20 item list would be best to use (some of those fields are filled in by people other than the beta tester). That way the information can later be stored so it matches other reports about the beta (not everything is a bug, so I am using a more genereal term here).

It is quite possible that the format will evolve as we go along, and I feel no presssure to lock ourselves into anything. This is merely a tool, not a goal in and of itself.

[pak19652002]

Roger that. Just glad I have something to use now.

ORIGINAL: Shannon V. OKeets

ORIGINAL: pak19652002

All the talk of bug reporting, spreadsheets, google, etc. here and on the test forum is blowing my mind! Would it be possible for Steve or his delegate to provide THE definitive format/system for the testers to follow? I'm so confused, I don't even know if this has already been done! I know there was a really long format proposed recently so a spreadsheet can be built. Should I use that? But, then there is this google thing.

The grunts just want the bosses to give us a form to fill in and not have to think about it too much...

Pvt. Pete

For now, the 20 item list would be best to use (some of those fields are filled in by people other than the beta tester). That way the information can later be stored so it matches other reports about the beta (not everything is a bug, so I am using a more genereal term here).

It is quite possible that the format will evolve as we go along, and I feel no presssure to lock ourselves into anything. This is merely a tool, not a goal in and of itself.

[jchastain]

For whatever it is worth, I’ll share a few of my opinions on a controlled testing process. First, let’s talk about the development lifecycle for a moment. In discussing professional software development frameworks, it is easy to get bogged down in the details of RUP and use cases, but I don’t believe that level of complexity is necessary for this discussion so I will instead use the less cumbersome V-Model.

As is shown in the diagram below, we might think of the software development lifecycle in terms of the letter V, where we start at the top left and then just trace the letter V (going down the left slant and then back up the right). The left side represents the construction activities. As we progress down the left side, we get to lower and lower levels of detail. So we might start with business level requirements, then progress to specifications and detailed design, and finally get to the coding and development of individual modules. The right side represents the testing and again, as we move up the slant, we progress from lower layers of detail up to greater abstraction. The key to the model is to show that not only do work products progress directly through the path, but also across to the testing functions that verify similar levels of abstraction. The business requirements drive not only the detailed specs, but also the acceptance testing. Nothing Earth shattering I know, but this is just for level setting.

[jchastain]

So, the final testing activity should be focused on confirming that the software meets the objectives set forth in the business requirements. Integration testing, where development modules come together, should focus on enduring that the software performs according to the specifications and design. While the unit test activities confirms that the individual module was developed properly.

With that as the basic framework for the discussion, let me first say that the conversation thus far has focused on the integration testing and confirming that the software performs according to the predefined rule book or specifications. That is not unexpected as the developers are likely performing some level of unit testing themselves before publishing builds out to the beta test group and therefore integration testing is the next required function. Some have discussed the potential for automated testing. In my experience, test automation isn’t effective for this type of integration testing. It is really more appropriate for regression testing as part of the acceptance process for software that is expected to have multiple releases. I do not see this project being near the point where automation is possible.

As others have indicated, it is necessary to “stage” certain conditions in order to permit the required testing. That is one of the key reasons why test scripts are developed. A test script is simply a series of actions that are taken in order to test a progression of functions. So, for example, we might test taking a navy to sea. Then we might test moving it while unloaded. Then we might test loading a land unit onto a Navy unit. Then we might test moving the loaded navy. Then we might test naval bombardment of an enemy unit on shore. Somewhere else, we better test bombarding a neutral unit just to ensure that doesn’t work. Then we might test unloading the naval unit as part of an amphibious assault. Then we might test allowing that unit to not receive resupply to ensure supply values are dropping appropriately. Then we might test to see if the AI properly attacks the invading unit or at least makes some type of defensive adjustment. Then we might test to see if the unit is properly removed from play once destroyed.

Not to belabor the point, but a test script is a series of events that can be tested together that often allow you to test events in the back of the script that require setup through events that are being tested at the start of the script. Here’s the key – to be successful every spec has to be mapped to at least one script. You literally must pour over the specs and ensure that everything that is supposed to happen is scripted into one of the test scripts. The real work then is defining the sequence of events that test the whole set of specs in as few steps as possible. It is a fairly laborious process, but it ensures full coverage.

In order to ensure that everything is tested, you then report bugs with an indication of the script that was run and the step of the script that failed. In that way, hopefully devs will have more success in duplicating bugs. But equally importantly, the planners can then note that the script has not been successfully executed and can ensure it is rerun once that current issue is addressed (which is necessary to ensure all the other items being tested later in that script are then tested). Once all the scripts have been entirely completed at least once, then you run through them all a second time just to confirm that none of the fixes in the later parts of the tests broke anything that was tested earlier.

That is a fairly lengthy description of something that many if not most here already know. So now, let’s get to the real point I am trying to make. The rulebook is only one set of the specs. It forms one set of the functional requirements for the game (though admittedly it is the most important one). But there are entire algorithms of the AI that determine what the reactions to certain events are. How are those to be tested? Is it also to be a systematic test to ensure the computer is evaluating systems properly or is it just tested in a haphazard way and only the game rules are tested systemically? If the AI is to be tested systemically, are there specifications for it or how would the test planners be able to script various possibilities? Then there is sound, game options, and a whole host of other functions that also need to be tested.

Beyond the functional requirements, there are also many non-functional requirements that really should be tested. How are errors to be handled? What about the UI and human factors engineering? Are there performance related requirements such as how long it should take for the AI to complete a turn or for the opening screen to load? What files are supposed to be editable by the player base and does editing them allow the software to still function properly? I absolutely understand that these are of lesser importance than just getting the game working, but it is more effective to do test planning if you understand the scope of your testing prior to constructing the plan.

Of the non-functional items above, UI and human factors testing will likely have the most significant impact. There really should be some focus on this area and planning these types of tests in an art unto itself. One of the most effective things you can do is develop scripts especially targeted at newbies that simply instruct them to perform various basic actions (move a unit, determine who is at war, change sound preferences, etc) and then ask each of your new testers to attempt to do each of those items and report back on whether or not they were successful and how difficult it was for them. Ideally they should do so without reading any manual or quick start guides to help determine whether or not the UI is intuitive. In high end studios, they actually bring people into labs and record HOW their attempts to perform each action. That is a good deal more useful than self reported actions if there is an ability to use friends and family members as guinea pigs.

Anyway, I continue to believe you would have greater participation, higher quality scripts, and better communication were you to give the test planners access to the software, but that is merely my own opinion. Hopefully some will find these musings helpful. Good Luck!

[Greyshaft]

JChastain,

All of what you say is valid. There are other issues which need to be considered.

1. I agree there is a need to ensure that the testing of MWiF is comprehensive and appropriate to a project of this scale. There is also a need to provide direction for our motivated volunteer force. The posts from pak and Griffitz62 are evidence of the importance of this second goal.  By the testers own admission their experience at formal testing vary considerably (thats an observation - not a criticism) so we have an immediate need to provide direction in this area. Experience in software testing tells me that those testers most in need of using a test plan are those least familiar with how it works.

2. Our budget for the test plan is precisely $0. It is all volunteer work and there are absolutely no resources to create stand-alone usability labs or to send the testers on 'Intro to Software Testing 101' type courses. We need to work with the material at hand, and if we aim to create a 600 page test plan and process flow then we will lose the hearts and minds of our testers real quick. Since I also have a wife, child and full-time job I am currently averaging only about 10 hours a week with my involvement with MWiF. In summary, there is no point in planning to build a Saturn V if you only have the skilled technicians and budget for a V2 or perhaps only a Goddard-type 'Nell'. To paraphrase Robert Watson-Watt, 'Let's have second-best tomorrow rather than perfection next year'. (BTW I'd be interested in learning of any other wargame development team which used a formal test plan...)

3. Test plans should be created after the specification but before the coders start work. In this case the specification for the test plan is the WIF:FE ruleset (RAW) which is available for download from www.a-d-g-.com.au so I don't see why the Plan creators need access to the MWiF software. Sure its great to have a copy and I'm sure Steve would look kindly on a request to issue the Test Planners with the code, but I'm looking for people who want to work on the organisational side of things. If we have spare time then lets refine the Test plan some more rather than double-clicking on MWiF.exe. That's what the Testers are there to do.

4. I can agree (or at least strongly sympathise) with all of your comments in the second part of your post. At this point I am more concerned with providing on-going direction for the testers rather than whether our Test Plan complies with IEEE 829. If we get enought resources into the test planning side of things then we might be able to achieve both objectives.

Thanks again for your post. I'm gratified that this topic has received such strong input.

[Shannon V. OKeets]

Yes, good ideas, well presented.

As for letting testers look at code - the Delphi development system pretty much requires using the Delphi envionment (IDE) to examine the code. That's because the layout for forms (103 at last count) have almost all their data stored in binary and presented to the developer to review and modify via the IDE. The Pascal source listings omit 95% of the form data settings - instead those are stored internally (in binary).

Even so, the Pascal listings are over 250,000 source lines and climbing daily. Virtually all of that would be pure gibberish to the testers who are not programmers. Even for testers with extensive programming experience, many accompanying diagrams and other supporting documents would be needed to understand the code listings. To say nothing of Matrix's security concerns.

Unit test as it is usually defined is not possible either. There are simply too many variables that have to be set before doing virtually anything in MWIF. On a positive note, I think of most of the forms as comparable to unit tests. That is because most forms are used for a specific task. The performance of the task can be tested by testing the form. There is still a ton of other stuff, but it is nice to think that 70 or so forms can be 'unit' tested.

Another source of 'specifications' are the optional rule writeups that I did. I started with RAW but from there many changes/clarifications were made in response to feedback from forum members. The final set of text (a 60 page PDF file) goes through each of the 81 optional rules and says precisely what it suppose to do. These writeups are part of the program and accessible during Start New Game. I will someday also make them accessible during game play. Once that is done, the testers will be able to refer to the 'specifications' for the optional rules while they are testing them.

[jchastain]

Grayshaft,

I think we are essentially in agreement, but let me touch on a few of your points.

First, I agree that you don't want to target boiling the ocean. My intended point was not that you have to meticulously script everything, but rather that you will be more effective if you define your scope of testing prior to beginning your planning. Perhaps the rules is the only thing you want to script out. That's fine - that's a valid decision. But I was suggesting that you consider all of the testing needs and ensure you are making that as a decision instead of just ending up there from failure to consider the broader picture. The other items will still need some type of testing and you'll want some type of feedback mechanism to encourage people to be looking at those things at some point in the cycle.

With regard to a "Testers 101" course, part of the reason I chose to detail out the basics of testing in a rather pronounced manner was to allow less experienced testers to follow the discussion as I hoped it might provide them with some additional perspective. As for access to the software, I believe you and I were both referring to the executables rather than the raw code. I agree that it would be crazy to make the code available to volunteer testers (except for snippets posted in response to specific questions). I continue to believe that the test planners should experience the software, its mechanics, and its UI. But as I have indicated that is just a part of the rationale behind the recommendation. Let's face it, volunteers want to be on the "inside" and WANT to experience the game. That's part of the "payoff" to many if not most. In the real world, test planners make more money and being a test planner represents career progression. But in terms of game testing, you are asking people to do something that is more work without offering them the "reward" afforded the "lesser" position of tester. Even if you said that this was a pathway to becoming a tester, I could see the deferred compensation model working. But to say this is something different that requires more work, does not provide the perceived benefit, and never leads to that benefit, just doesn't seem to provide the proper motivational incentives in my mind. I understand why you don't want people "promoted" to being testers as you don't want turnover in these positions, but I really believe that without any payoff you'll end up with more turnover rather than less. And I understand why you do not want to make the planners testers as well for fear they will ignore the planning and be drawn more and more into the "fun" of testing. But I continue to believe that managing that dynamic is easier than managing retention in a role that never gets to the "fun" part. Again, that's just my opinion on the matter and I'll drop it now (as continuing this discussion in public does not aid your cause).

Of interest, you also asked the question of what other games have attempted such a structured testing approach. Over the past 15 years I have been a volunteer tester for more than a dozen projects and I have never seen any of them even attempt this type of professional organized approach. As you suggest, you cannot turn it into "work" so that the testers feel as if they are operating within a straight jacket and not having any fun. But I would expect you to get some real benefit out of this approach if you can get it properly organized and then execute it in a properly relaxed manner. When you think about it, it is not entirely dissimilar to the write-up activities where people are assigned units of work and then perform them at their leisure. And that is why I continue to believe that Mziln had pretty solid insight in his recommendations to Graf regarding eventual beta participation for those who "prove" themselves through the write-ups effort.

Finally, I am afraid you may have misunderstood my points with regards to UI testing. Though I continue to believe that UI testing is important, I was not suggesting that you have a full blown UI lab. Instead, I believe your first opportunity is new testers. Whether now to WIF or not, that will be new to the UI and their initial impressions should be captured. I was suggesting you give them a list of 10 or 20 basic functions to try and perform without reading any manual or notes. Tell them to take no more than 60 seconds attempting each action and then report back that doing so was:

• 1 - Very Easy. I was able to perform this action on my first attempt with little forethought.
• 2 - Relatively Easy. I was able to perform this action fairly quickly but had to think about it a bit.
• 3 - Somewhat difficult. I was able to perform this action but it required several tries and/or UI exploration.
• 4 - I was not able to complete this action within 60 seconds.

While valuable, self reported statistics such as these have 2 drawbacks. First, we humans have ego and often want to ensure we aren't seen as the dumb one who couldn't figure something out. This will sometimes cause you to get a picture that might be more rosy than the reality. But also, you don't get to see what people actually try. If everyone puts down Relatively Easy for a given function, you might think you are in pretty good shape. But by being to watch them, you might find they all tried the same thing the first time, in which case you may want to change the UI to make the software work in the way people are intuitively attempting to use it. You only get that information when you can watch their progress against the tasks. Obviously you do not have the ability to rent out a full-blown UI lab. That is why I suggested friends and family. If you know people locally who play games but are not yet accustomed to this title, be they friends, family, neighbors, unaffiliated matrix people at an event, forum members who happen to live nearby, whatever, then you may want to have them come by and just watch over their shoulder as they attempt to do the actions. It isn't a lab, but it is free and might provide some useful insights.

[NeBert]

Great discussion!
Until now I was more a consumer of the forum but with some experience in testing and creating testplans I would like to bring in some additional points.

1. Patrice mentioned the problem to just jump into a specific situation of the game (I call it Test Setup): This will be one of the most time consuming tasks for testers without (or with only little) output to the programmers!

Possible solution could be
- to have some kind of Testsetup-Editor (yes I know, this means effort) but it will pay back more than this considering some 10 - 20 testers.
- a pool of saved games within the testgroup to have lots of starting points for different tasks (e.g. Naval Landings, ...)

2. Bug-Reporting
My experience is that once I find a bug the description for the programmer takes quite a time. Some Configuration-Export function (Unit-Setup etc.) will help progammers understanding the problem and reduce the non-testing time for testers.

3. Bug-tracing
A tracing tool is essential! Even if it´s just an Excel-Sheet (better would be a small Database).

4. What about different Hardware- and Software-Configurations among the testers? (RAM, Harddisk, Operating System, ....) Is it already in test or is it part of the coming integration tests?

5. Test-Tracing
A database for all testcases and their performance is as important as for bug-tracing (again, Excel could be sufficient with the problem of multi-user access)

Hope I could help a bit

[Greyshaft]

ORIGINAL: jchastain
Grayshaft,

That's Greyshaft... with an 'e'[:)]

Perhaps the rules is the only thing you want to script out.

Scripting the rules is achievable and essential. It's my highest priority.

But I was suggesting that you consider all of the testing needs and ensure you are making that as a decision instead of just ending up there from failure to consider the broader picture. The other items will still need some type of testing and you'll want some type of feedback mechanism to encourage people to be looking at those things at some point in the cycle.

Hence my call for assistance. If I have to do it all myself then I will focus on the WIF rules since:
* I would rather complete one out of five tasks than pretend I was doing all all five tasks but only get them partially done;
* in a very real sense, the rules are the 'specification' of MWiF;
* the gaming community will forgive a problem with loading the game (c'mon the patch!) but if MWiF makes a mistake with the rules then the game will be seen as a failure (IMHO);
* testing specific rules is the most effective way to give direction to the testers. It also helps the testers educate themselves about the rules.

As for access to the software, I believe you and I were both referring to the executables rather than the raw code.

Agreed. Steve lives in a world of binary so I understand his alternative interpretation [:D]

Let's face it, volunteers want to be on the "inside" and WANT to experience the game.

I completely agree with you. It's a very human reaction. What I want to avoid is the scenario of a person with testing experience signing up as a test planner with the deliberate intention of becoming a Tester further down the track. I'll be on the Test Plan side of things till the game ships and I'm looking for people who are willing to join me here. If no-one is available since everyone wants to be a Tester rather than a Test Planner then that's OK - I completely sympathise with their motivation, but they should then apply to be a Tester rather than apply to be a Test Planner. At least this way we know where we all stand.

I understand why you do not want to make the planners testers as well for fear they will ignore the planning and be drawn more and more into the "fun" of testing. ... that's just my opinion on the matter and I'll drop it now (as continuing this discussion in public does not aid your cause).

Au contraire! I believe that by discussing this issue publicly we leave people under no illusions as to what is involved in being a Test Planner and that is a good thing.


We'll discuss the UI test lab in another post down the track


Nebert,
Thanks for your input. I have to go and feed two-year old Greyshaft Junior but we'll talk more later.

[Shannon V. OKeets]

ORIGINAL: NeBert
Great discussion!
Until now I was more a consumer of the forum but with some experience in testing and creating testplans I would like to bring in some additional points.

1. Patrice mentioned the problem to just jump into a specific situation of the game (I call it Test Setup): This will be one of the most time consuming tasks for testers without (or with only little) output to the programmers!

Possible solution could be
- to have some kind of Testsetup-Editor (yes I know, this means effort) but it will pay back more than this considering some 10 - 20 testers.
- a pool of saved games within the testgroup to have lots of starting points for different tasks (e.g. Naval Landings, ...)

2. Bug-Reporting
My experience is that once I find a bug the description for the programmer takes quite a time. Some Configuration-Export function (Unit-Setup etc.) will help progammers understanding the problem and reduce the non-testing time for testers.

3. Bug-tracing
A tracing tool is essential! Even if it´s just an Excel-Sheet (better would be a small Database).

4. What about different Hardware- and Software-Configurations among the testers? (RAM, Harddisk, Operating System, ....) Is it already in test or is it part of the coming integration tests?

5. Test-Tracing
A database for all testcases and their performance is as important as for bug-tracing (again, Excel could be sufficient with the problem of multi-user access)

Hope I could help a bit

Thanks.

1 - There is a debugging function (Place) that lets players place units on the map. It was part of CWIF and I have fixed dozens of bugs replated to it but it still is such a powerful tool that it is easy to crash the game by placing units in unreasonable locations ("Look! There's a Chinese militia in Berlin! - And Germany is in the middle of the Production Phase!"). A pool of saved games corresponding to test cases is an excellent idea.

2 - MadExcept was doing a fine job automatically reporting errors to me via email with the internal program call sequence. We were using it for version 3.xx. But I lost it when I reinstalled my compiler in the spring. I should have it reinstalled for 5.00.

3 - Greyshaft's area.

4 - MadExcept provides that. Also, even without MadExcept, most beta testers use one machine so the information only needs to be gathered once.

5 - Greyshaft's area.

[NeBert]

Nebert,
Thanks for your input. I have to go and feed two-year old Greyshaft Junior but we'll talk more later.

[;)] Seems we have some similar "constraints" for spare time activities like testing complex software. We just sent our two little Generals to bed...

I fully agree to you in dividing testers and test planners - different people have differnt ideas and my experience at work is that 1+1 is normally more than 2. I would expect better output, most probably even quicker that acting only with lone fighters. Avoiding double work will certainly increase testers morale as well.

So, due to my time-constraints (family, job is demanding in addition) I didn´t apply for a tester in the recent call, but with some additional informations about the expectations (or in other words: a clear outcome out of the discussion about the testing-philosophy) and if a rough timeframe of about 5 hours a week (peak workload might be higher) is enough I could try to help pushing the development forward.

Regards

[Greyshaft]

ORIGINAL: NeBert
So, due to my time-constraints (family, job is demanding in addition) I didn´t apply for a tester in the recent call, but with some additional informations about the expectations (or in other words: a clear outcome out of the discussion about the testing-philosophy) and if a rough timeframe of about 5 hours a week (peak workload might be higher) is enough I could try to help pushing the development forward.

Constraints noted.
Resume noted.
You're in.
Now raise your right hand and repeat after me... "I, NeBert, do hearby solemnly swear etc etc"

Welcome to the Test Planner Team. Remember that this does not give you access to the MWiF software or the Development Forums. If either of us is killed or captured then Matrix will disavow all knowledge of our actions.

* Please visit the download page of www.a-d-g.com.au and download the WiF rules.
* Please send me your email by PM

[siRkid]

I do software testing for a living and have been a beta tester on a lot of games, I was even the coordinator for War in the Pacific. I am not applying for a position because I know nothing of the game but I would like to offer one suggestion. Take the game manual and turn all the statements into questions and build a checklist. For example lets say the Rules state that a unit can only move one hex when it is out of supply. You check list would have a question that reads, "Can you only move a unit that is out of supply one hex?" Then you could divide the list up and assign them to the testers. Just my two cents.

[Froonp]

ORIGINAL: Kid

I do software testing for a living and have been a beta tester on a lot of games, I was even the coordinator for War in the Pacific. I am not applying for a position because I know nothing of the game but I would like to offer one suggestion. Take the game manual and turn all the statements into questions and build a checklist. For example lets say the Rules state that a unit can only move one hex when it is out of supply. You check list would have a question that reads, "Can you only move a unit that is out of supply one hex?" Then you could divide the list up and assign them to the testers. Just my two cents.

Hopefully, something I can understand in this thread !!! Thanks Kid !!! [&o]

So, Greyshaft, understood ? Turn all into questions as Kid said !!! [8D]

[jchastain]

I would actually say turn everything into 2 questions.  You should test the positive and negative of every rule.  So if the rule is "To load a ground unit on amphibeous transport it must be in supply" then you should test:

• Can I load a land unit onto transport when it is in supply?
• Am I prevented from loading a land unit onto transport when it is out of supply?

[Greyshaft]

ORIGINAL: Kid
I do software testing for a living and have been a beta tester on a lot of games, I was even the coordinator for War in the Pacific. I am not applying for a position because I know nothing of the game but I would like to offer one suggestion. Take the game manual and turn all the statements into questions and build a checklist. For example lets say the Rules state that a unit can only move one hex when it is out of supply. You check list would have a question that reads, "Can you only move a unit that is out of supply one hex?" Then you could divide the list up and assign them to the testers. Just my two cents.

Thanks for the input Kid. I have taken discrete topics as defined in the original published WiF rules and copied each of them onto a separate tab of an Excel sheet (see screenshot below - note how the tabs correlate to the index). I will track how MWiF modifies the rules and then have converted the rules into questions (see screenshot in post @17 in this thread on previous page). Sounds like I'm in line with your suggestion but please feel free to offer ideas on fine tuning the model.

[NeBert]

I sent two messages to you. Did they arrive?

Regards

[Shannon V. OKeets]

ORIGINAL: NeBert

I sent two messages to you. Did they arrive?

Regards

Could you please copy me on this stuff? I like to stay informed (though I certainly do not want to micromanage this - far from it![:D]).

[Greyshaft]

ORIGINAL: NeBert

I sent two messages to you. Did they arrive?

Regards

To which 'you' do you refer [;)]
I received and replied to your messages last week via email. I'll send you another mail today to follow up.