Playtesters?

[MarkShot]

My profession isn't actually computers in the sense of programming. Of course, I did some of that starting out many years ago. But what I really do is design systems.

One of the basics of good system design is the assumption that the system will have both execution defects (things causing crashing and other undesirable actions) and logic defects when the system just fails to function as specified. Most programming environments provide various tools for debugging programs. However, a system usually should be designed with built-in testing tools to help the developers track down problems. These test tools will be more oriented towards the purpose and functionality of the system than simply displaying the values of variables and whether a particular code path is taken.

A game is nothing more than a relatively simple system architecture. As complex as customers think they are, they tend to be much simpler than a trading floor or most commerical systems. In any case, the game should include facilities for logging critical data to simplify the problem of tracking down bugs and behavioral problems. The game should have facilities to exactly replicate the same sequences of events over and over again (and remove random behavior; good for playing, but bad for debugging). The game should have facilities to capture key stroke/mouse logs so that user inputs can be replayed to the engine. Logging and data capture tools (with replay capability) become even more important for multi-player testing, since the concurrency adds all types of timing related problems such that traditional debugging would completely screw up the sequencing of events due to the long delays it imposes during to programmer interactions.

The bottom line: A good design anticipates problems and builds tools into the design for reproducing, investigating, and resolving problems. Poor designs rely on nothing more than traditional debugging tools and brute force. In the end, any money saved by reducing the amount of development effort coding custom debugging tools is lost during the increased time needed for defect resolution.

[GoodGuy]

ORIGINAL: MarkShot

The game should have facilities to exactly replicate the same sequences of events over and over again (and remove random behavior; good for playing, but bad for debugging). The game should have facilities to capture key stroke/mouse logs so that user inputs can be replayed to the engine.......[].

Right, that's how it should be. That makes me curious though, the AA engine doesn't have this, right (i.e. built-in crash-log feature containing key/mouse logs)?

The bottom line: A good design anticipates problems and builds tools into the design for reproducing, investigating, and resolving problems. Poor designs rely on nothing more than traditional debugging tools and brute force. In the end, any money saved by reducing the amount of development effort coding custom debugging tools is lost during the increased time needed for defect resolution.

The question is if programmers have the time / ressources / knowledge to incorporate sophisticated tools (maybe even with additional AIvsAI checks) and if they use/like this method/approach. My guess is that many small devs are using basic/traditional debugging tools, for various reasons.

[MarkShot]

AA Engine:

The debugging builds do produce UI interaction logs which can be used to reproduce problems spotted by beta testers with all randomness (the seed is fixed) removed. However, the dreaded "out of sync" bug has at time compromised this capability. This code is removed in production builds (gold release and patches), since it adds a performance overhead that the customers would not really appreciate. Additionally, the beta builds contain a large collection of sanity check assert statements to catch conditions which would lead to CTDs. Once again, this code is removed for production, and if corrupt data appears, then the customer just simply sees a CTD.

I must say that Paul Scobell who did most of the programming on the engine was an extremely talented and meticulous programmer. While he was working full-time for PG (RDOA and HTTR), the quality of the beta builds often exceed the quality of many commercial production products. He was from the old school of programming; the same one I was raised in. To have anyone else catch a bug in your work is a personal disgrace. You don't need an SQA department, since it is your responsability to make sure it is defect free.

My guess is that most don't:

The primary reasons being a lack of experience in designing systems "in the large". There is a big difference between simply writing programs and designing entire systems for testability. Such systems design experience is extremely valuable for games development, since like commercial systems, most games continue to evolve the same code base year after year. So, the quality of structuring that code base for long term development often determines how feasible it is to bring out each successive game.

AGE Studio is a good example of a very well structured code base. AGE Studio unlike many developers is able to back port most new features being added to the engine for A-ACW to the original game, BoA. This is the result of a well thought out modular design.

[GoodGuy]

I hate to quote myself, but I have to specify my question:

ORIGINAL: GoodGuy

the AA engine doesn't have this, right (i.e. built-in crash-log feature containing key/mouse logs)?

I meant a crash-report capability that could provide online feedback from users/with the user's logs. I know that nothing like that comes up, but wondered if it was part of an unemployed engine feature, besides an offline-version (logs) being employed in the beta builds.

ORIGINAL: MarkShot

My guess is that most don't:

You mean most don't use debugging tools?
In my experience, quite a few niche-devs (wargame-niche isn't the only niche) use debugging features to some extent, be it tools accessible to devs only, or tools accessible to testers. Some licensed engines even come with a myriad of debugging features and major QA divisions have customized testing routines.
Like I said before, incorporation of sophisticated tools and sophisticated software design (like you said, designing things "in the large") may not be found in various small devs's projects, but may not be found at major devs using engines off the rack, too. Some major devs have to create a "stripped" version after obtaining a license (an engine), so it's like they have to "gut" most of the core code in order to achieve certain customized effects/results, but they can still use the tools.... so it's like many of these engines "support" some level of lazyness. Some of the most prominent examples of such customizations (plagued with bugs and sloppy work) were SWAT 4 and Tribes 3, highly customized, but based on a stripped UT engine.

So you may be right when you're saying that "old school" programmers might have the better approach.

Back to wargaming... that said, I wonder if the devs of COI, for example, will be able to come up with a completely new engine/approach or "just" with a revamped one.