Testing 101
Posted: Sun Jun 05, 2005 10:24 pm
I thought I'd post up some information on software testing to give us a common vocabulary. The following information is shamelessly borrowed from the documentation for a commercial software Testing Tool available at Test Complete. I would expect that Matrix will be using this or a similar software testing tool when testing their games so its worth the effort for us wanna-be testers to understand the lingo.
Another purpose of this post is to reiterate the point that Testing is serious work. The difference between testing and simply exploring is that in the case of testing, output is compared against an expected standard. In other words, the basic test sequence is –
* Define expected output.
* Feed corresponding input.
* Gather output.
* Compare to expected output.
* Call for attention if the comparison fails.
Testing occurs under a variety of Test groupings. The major sections are:
* Unit Testing consists of trying out the functions, procedures or methods that a source module makes available to the rest of the application. A "unit" may be anything from a single function to an entire library. The essential point of unit testing is that only a small part of the intended application is tested.
* Functional Testing is the testing of a given application's function, that is, its relation to the user and (especially) to the rest of the system. Traditionally, functional testing is implemented by a team of testers, independent of the developers. This is because functional testing by developers cannot be unbiased. A developer's notion of what the specifications mean may interfere with an objective evaluation. *** NOTE: THIS IS WHERE BETA-TESTERS ARE USED.*** While unit testing tests both what an application segment does (how it meets a part of the application specs) and how it does it (how it behaves towards the rest of the application), functional testing tests only the what. A functional test is not concerned with an application's internal details. Functional testing should begin as soon as there is a function to test, so the application does not "drift off course", and continue through the application's completion and the first customer contact.
* Regression testing means “repeating a test already run successfully, and comparing the new results with the earlier valid results”. This tool is useful when you run a test on your project and then correct the project code. The user gains two things from this process – a test, and a standard for acceptance. Regression testing is based on the idea of reusing a test and acceptance standard, rather than forgetting about them once the test is successful. In true regression testing, all tests of all sizes and their results accumulate, and nothing is thrown away. On each iteration, all existing, validated tests are run, and the new results are compared to the already-achieved standards. And normally, one or more additional tests are run, debugged and rerun until the project successfully passes the test. Regression tests begin as soon as there is anything to test at all. The regression test suite grows as the project moves ahead and acquires new or rewritten code.
There are similarly sized software projects out there such as WiTP, but I don't know of a comparable project which has taken an existing ruleset for a non-computer environment - cardboard WiF - and attempted to make it into a computer game. Hmmm... I wonder how the "A World At War" Team is getting on?
Anyway, I'll pump out some testing templates this week and interested parties can play with those.
Another purpose of this post is to reiterate the point that Testing is serious work. The difference between testing and simply exploring is that in the case of testing, output is compared against an expected standard. In other words, the basic test sequence is –
* Define expected output.
* Feed corresponding input.
* Gather output.
* Compare to expected output.
* Call for attention if the comparison fails.
Testing occurs under a variety of Test groupings. The major sections are:
* Unit Testing consists of trying out the functions, procedures or methods that a source module makes available to the rest of the application. A "unit" may be anything from a single function to an entire library. The essential point of unit testing is that only a small part of the intended application is tested.
* Functional Testing is the testing of a given application's function, that is, its relation to the user and (especially) to the rest of the system. Traditionally, functional testing is implemented by a team of testers, independent of the developers. This is because functional testing by developers cannot be unbiased. A developer's notion of what the specifications mean may interfere with an objective evaluation. *** NOTE: THIS IS WHERE BETA-TESTERS ARE USED.*** While unit testing tests both what an application segment does (how it meets a part of the application specs) and how it does it (how it behaves towards the rest of the application), functional testing tests only the what. A functional test is not concerned with an application's internal details. Functional testing should begin as soon as there is a function to test, so the application does not "drift off course", and continue through the application's completion and the first customer contact.
* Regression testing means “repeating a test already run successfully, and comparing the new results with the earlier valid results”. This tool is useful when you run a test on your project and then correct the project code. The user gains two things from this process – a test, and a standard for acceptance. Regression testing is based on the idea of reusing a test and acceptance standard, rather than forgetting about them once the test is successful. In true regression testing, all tests of all sizes and their results accumulate, and nothing is thrown away. On each iteration, all existing, validated tests are run, and the new results are compared to the already-achieved standards. And normally, one or more additional tests are run, debugged and rerun until the project successfully passes the test. Regression tests begin as soon as there is anything to test at all. The regression test suite grows as the project moves ahead and acquires new or rewritten code.
There are similarly sized software projects out there such as WiTP, but I don't know of a comparable project which has taken an existing ruleset for a non-computer environment - cardboard WiF - and attempted to make it into a computer game. Hmmm... I wonder how the "A World At War" Team is getting on?
Anyway, I'll pump out some testing templates this week and interested parties can play with those.
Working my posterior off at work and converting RaW to Html when I get home 
Work, work, and more work 