Testing 101

[Greyshaft]

I thought I'd post up some information on software testing to give us a common vocabulary. The following information is shamelessly borrowed from the documentation for a commercial software Testing Tool available at Test Complete. I would expect that Matrix will be using this or a similar software testing tool when testing their games so its worth the effort for us wanna-be testers to understand the lingo.

Another purpose of this post is to reiterate the point that Testing is serious work. The difference between testing and simply exploring is that in the case of testing, output is compared against an expected standard. In other words, the basic test sequence is –
* Define expected output.
* Feed corresponding input.
* Gather output.
* Compare to expected output.
* Call for attention if the comparison fails.

Testing occurs under a variety of Test groupings. The major sections are:

* Unit Testing consists of trying out the functions, procedures or methods that a source module makes available to the rest of the application. A "unit" may be anything from a single function to an entire library. The essential point of unit testing is that only a small part of the intended application is tested.

* Functional Testing is the testing of a given application's function, that is, its relation to the user and (especially) to the rest of the system. Traditionally, functional testing is implemented by a team of testers, independent of the developers. This is because functional testing by developers cannot be unbiased. A developer's notion of what the specifications mean may interfere with an objective evaluation. *** NOTE: THIS IS WHERE BETA-TESTERS ARE USED.*** While unit testing tests both what an application segment does (how it meets a part of the application specs) and how it does it (how it behaves towards the rest of the application), functional testing tests only the what. A functional test is not concerned with an application's internal details. Functional testing should begin as soon as there is a function to test, so the application does not "drift off course", and continue through the application's completion and the first customer contact.

* Regression testing means “repeating a test already run successfully, and comparing the new results with the earlier valid results”. This tool is useful when you run a test on your project and then correct the project code. The user gains two things from this process – a test, and a standard for acceptance. Regression testing is based on the idea of reusing a test and acceptance standard, rather than forgetting about them once the test is successful. In true regression testing, all tests of all sizes and their results accumulate, and nothing is thrown away. On each iteration, all existing, validated tests are run, and the new results are compared to the already-achieved standards. And normally, one or more additional tests are run, debugged and rerun until the project successfully passes the test. Regression tests begin as soon as there is anything to test at all. The regression test suite grows as the project moves ahead and acquires new or rewritten code.

There are similarly sized software projects out there such as WiTP, but I don't know of a comparable project which has taken an existing ruleset for a non-computer environment - cardboard WiF - and attempted to make it into a computer game. Hmmm... I wonder how the "A World At War" Team is getting on?


Anyway, I'll pump out some testing templates this week and interested parties can play with those.

[Mziln]

Anyone speak "Australian" [&:]

Kasigi spends his free Japanese intelligence operation on a multiple (hoping to bludge off the German dice rolls).

[Greyshaft]

Welcome to the reality of software development [:D]

The Test Plan is an essential component of the software development process. If there is no test plan then the whole testing process is no more sophisticated than rolling 2D6 a few times and then presuming that there are no sixes on the dice because none showed up on your set of rolls. That's why companies need to release software patches - their testing process did not catch the bugs before the first commercial release of the software.

The good news is that in UAT (User Acceptance Testing) the Testers do not have to actually develop the Test Plan - that's the job of the Test Manager. However in a well operating test environment the testers do commit to running the Test Scripts which they have been given and they also commit to retesting the bugs they report after the next release of the software. Its just sharing the workload. Remember that if the testers don't do it then the developers will be stuck with the job... or the game will be incompletely tested when it is finally released.

Hopefully along the way everyone does get to play with the game and have some fun while they're doing it.

[Greyshaft]

Most of the (IMHO) brilliant ideas I have had for MWiF have already been posted by someone else so I'll spend my time considering Project Management strategies. One of the areas that hasn't yet been covered is the development sandbox i.e. how much of the game will be available at the point of each test release. I'm thinking for:

Test Candidate 01 :-
ACTIVE MAP: Germany v France/England in 1940
UNITS: All available on map but no Naval missions or Naval combat supported
PRODUCTION: Reinforcements available but no new production
POLITICAL: None
AI: No - but hooks should be firmly embedded

Test Candidate 02 :-
ACTIVE MAP: Expand to include Poland and Med
UNITS: All
PRODUCTION: Yes
POLITICAL: Yes to include Poland and Italy
AI: AI strategy devised and AI editor roughly mapped out. Players can create combat scripts for AI-moderated air combats.

Test Candidate 03 :-
ACTIVE MAP: Full map
UNITS: All
PRODUCTION: Full
POLITICAL: Include US entry options
AI: AI manual available for review. Players can write AI modules.


Comments?

[Shannon V. OKeets]

ORIGINAL: Greyshaft
Most of the (IMHO) brilliant ideas I have had for MWiF have already been posted by someone else so I'll spend my time considering Project Management strategies. One of the areas that hasn't yet been covered is the development sandbox i.e. how much of the game will be available at the point of each test release. I'm thinking for:

Test Candidate 01 :-
ACTIVE MAP: Germany v France/England in 1940
UNITS: All available on map but no Naval missions or Naval combat supported
PRODUCTION: Reinforcements available but no new production
POLITICAL: None
AI: No - but hooks should be firmly embedded

Test Candidate 02 :-
ACTIVE MAP: Expand to include Poland and Med
UNITS: All
PRODUCTION: Yes
POLITICAL: Yes to include Poland and Italy
AI: AI strategy devised and AI editor roughly mapped out. Players can create combat scripts for AI-moderated air combats.

Test Candidate 03 :-
ACTIVE MAP: Full map
UNITS: All
PRODUCTION: Full
POLITICAL: Include US entry options
AI: AI manual available for review. Players can write AI modules.
Comments?

I have vastly different test release versions planned. To start with I expect to use the CWIF map and units so I don't need to do a gradual expansion of available map. I also have already done things like US Entry options. Of course, all of these should be tested rigorously using your methods.

What I have as releases for beta testing are:
(1) new start of game, new saved game, new multiplayer system, functionality missing from CWIF (e.g., surrender)
(2) remaining scenarios (up from 3 to 11)
(3) new game interface
(4) animations and sounds
(5) PBEM system (with AI as assistant)
(6) historical detail, online help
(7) AI as opponent

These will overlap in the sense that the testing on the first will probably still be in progress when the second is available. I can coordinate the releases with you (Grayshaft) so they occur whenever you think is best.

[Greyshaft]

Hi Steve,
I have no problem with those additional features being already enabled in the early Test Candidates and non-core issues like the game interface, on-line help and saving/loading files can be added any time you have them ready and will doubtless be tweaked right up until the final code freeze. I'm just considering the house you are building and trying to determine what are the foundations which must be poured and set before the walls are raised and the roof added. Us testers can get very focussed and can sometimes pick a graphic to death while perhaps missing the deeper issues (I know I have!). The Test Candidates system will keep us all aware of which Functional issues need to be tested to completion at that particular time. Additional bugs should always be reported as soon as they are found and many will doubtless be reported before you are ready to fix the relevent piece of code, but the issues nominated for each TC need final signoff before the development moves to the next stage.

So I am expecting that all of these current Forum discussions will be distilled into a Functional Specification which will become the basis of the Test Plan and eventually morph into the player manual released with the game. Were you considering a particular software tool or document format for consolidating all of this information? I need to be on the same page as you're working from for all of this.

[Shannon V. OKeets]

ORIGINAL: Greyshaft

Hi Steve,
I have no problem with those additional features being already enabled in the early Test Candidates and non-core issues like the game interface, on-line help and saving/loading files can be added any time you have them ready and will doubtless be tweaked right up until the final code freeze. I'm just considering the house you are building and trying to determine what are the foundations which must be poured and set before the walls are raised and the roof added. Us testers can get very focussed and can sometimes pick a graphic to death while perhaps missing the deeper issues (I know I have!). The Test Candidates system will keep us all aware of which Functional issues need to be tested to completion at that particular time. Additional bugs should always be reported as soon as they are found and many will doubtless be reported before you are ready to fix the relevent piece of code, but the issues nominated for each TC need final signoff before the development moves to the next stage.

Fine. I am quite happy to have you in charge of this. If you need more decision making by me, let me know. Otherwise I will not intrude on the testing process.

So I am expecting that all of these current Forum discussions will be distilled into a Functional Specification which will become the basis of the Test Plan and eventually morph into the player manual released with the game. Were you considering a particular software tool or document format for consolidating all of this information? I need to be on the same page as you're working from for all of this.

Yes, from the forums I am heading to writing design specifications (the term Functional Specification brings back bad memories). In the past I have used design specifications which were either loosely defined, or rigorous defined and quickly out of date/irrelevant. Often the official document formats required a lot of tedious and valueless repetition of the obvious with all the important details crammed into one or two subsections. Perhaps they have improved in recent years (I seriously doubt it). So, I propose to simply write up specifications that tell me how to program the beast. They will probably include a lot of lists in a hierarchy structure and/or some other structure that denotes branching logic.

I do most of my writing in WordPerfect (my own little protest against MS Word) and then translate documents over to MS Word for people who need that (e.g., Matrix). The player manual will be a PDF file towards which I will contribute but for which Matrix will be the final author/publisher.

[Greyshaft]

I'm no fan of Word or MS but I don't have WordPerfect so we can talk in text or Word docs if you can. I use the term Functional Spec at work (we're an ITIL and Prince2 shop) but I'll try to remember your preference [:)] I'm no fan of superfluous documentation so I'm happy to work from bullet point lists or even the back of an envelope if you've got one. I see the existing WiF RAW as an enormous aid in providing a fixed object around which the initial design spec and Test plan will revolve e.g."The supply rules will be like Wif RAW except as follows... blah blah"

[Mziln]

Working my posterior off at work and converting RaW to Html when I get home

Work, work, and more work

[Shannon V. OKeets]

ORIGINAL: Mziln
... converting RaW to Html when I get home

Interesting idea. What do you think the advantages might be if we worked in HTML? WordPerfect has no problem doing so.

[Mziln]

Size (a 30k microsoft office doc shrinks to 3k), Graphics, Ease of use (you can use hiperlinks to step through the rules), Portability(as long as all the Html files are in the same directory the hyperlinks work), Rules would be easier to organize.

Is there a way to upload you a copy? Send me a PM and I'll send you a rough demonstration.