Matrix Games Forums

Hi Guys,
Just thought i'd comment on the question thats cropped up a couple of times now regarding our testers.
We recruit testers based on many factors, and at any one time try to have a well rounded group including some that like to just play and have fun, and some that love to get right into the engine and get the most out of every feature. We tend to get the best results this way. We try to be fairly hands off with our testing, and dont pester of nag people to try certain features - we make mention of them in the patch notes and leave the rest to them.
We like to think our games are fairly polished on release. This is generally backed up from our customers feedback, and the fact that our games dont tend to require and endless stream of patching over the next 12 months after their release. They are also generally free of 'showstopping' bugs that require immediate patching (ie releasing a patch withing a day or two of the problem being found), though obviously everyone's opinion of what constitutes a showstopping bug is different.
We think that a major reason our games are polished is due to the excellent testers we have. Sure, not every bug gets picked up in the testing process, but we certainly think that the majority do, and are happy with the process we have in place.
I know it can seem incredible that some of these bugs that seem obvious arent picked up, but dont forget that we have been testing CAW for over 6 months now - and whilst the game is fun, we (and many of our testers) take it upon themselves really play the game in every spare minute they have and therefore after 6 months of playing, sometimes the forest can't be seen for the tree's (ie obvious bugs can stare us in the face).

ORIGINAL: Prince of Eckmühl

ORIGINAL: Scott_WAR

Honestly, between this major issue, and the major issue with multiplayer, one wonders............... Does SSG need better testers, or did they not listen to their testers? Either way somebody really dropped the ball.

Now, if its a design choice, and NOT a bug, then thats just sad. Why would you purposely make the newer version inferior to the old version?

Don't blame the testers. I don't believe that they got their hands on the game that we've been playing before it was released. Way up the thread list, I suggested that the major focus of what testing did take place was stability testing for multi-player. While not intending to put words in anyone's mouth, I believe that Chris Merchant has confirmed as much.

As owners of the game, our immediate problem is trying to sort out bugs from features, trying to read the mind of the developer, as it's nature is so completely unknown to us. The holes in our understanding of what's going on with its different processes are just that substantial. Ouija board, anyone?

And it doesn't help when reps from the developer (Gregor) keep(s) showing up and pointing out to us that he (SSG) sold a bazillion of these things "back in the day," and that we should trust them about this or that element of a design that is so demonstrably untested and undocumented.

PoE (aka ivanmoe)

No game that we, or anyone else for that matter, has ever done has ever made all users happy, and nor can it. We know this and we take all feedback seriously. So this is what is going to happen.

Bugs, that is where the game isn't working as intended, will be fixed.

Improvements, i.e changes which don't affect the results of the game, but which improve the experience for users and which can reasonably be implemented, will be made. An example of this is not showing the results of strkes in the combat screen until the animations have played out. We didn't anticpate the need for this, but its a change we're happy to make.

Game design changes are harder to categorise. Every game must choose what it feels are the appropriate levels of detail for the various elements and inevitably some people will disagree with those choices. It is here that I feel we are entitled to cite the prior history of the game, and its success, as evidence that lots of people have, in general, agreed with our choices. I see nothing sinister or disrepectful in that. That doesn't mean that we will refuse to entertain suggestions for change, quite the opposite is true. We read every post and discuss all questions raised. Some ideas might not be immediately possible but might inspire future changes.

I'm sorry that I don't have the time, at the moment to go into more detail, but if I might make one last appeal to history, our record shows that we do conscientously support our games, that we do listen to our customers and I can assure you all that this will not change.

Gregor

Fair enough. Many games take a patch or two to realize their full potential. As it is there are few issues compared to many games, and those shouldnt be too much to fix. A few new scenarios designed and balanced for multiplayer will take care of two birds with one stone... the multiplayer issue AND the low number of scenarios.

I have no doubt a patch is forthcoming, and SSG has a strong rep for responding to gamers' feedback. I'm going to switch back to lurker mode for the forseeable future, mainly because the EGO displayed by some folks on these forums....constantly stating they're being ignored and questioning the integrity and work ethic of people they've never met while assuming that what they're saying HAS to be true and that they have the answers, if only someone would listen.....gives me a nosebleed.

ORIGINAL: themattcurtis

I have no doubt a patch is forthcoming, and SSG has a strong rep for responding to gamers' feedback. I'm going to switch back to lurker mode for the forseeable future, mainly because the EGO displayed by some folks on these forums....constantly stating they're being ignored and questioning the integrity and work ethic of people they've never met while assuming that what they're saying HAS to be true and that they have the answers, if only someone would listen.....gives me a nosebleed.

Those of who have purchased the game are in something of a quandry in that we are as yet unable to discern a bug from a feature. Examples would include the CAP routine, in which a/c that are assigned to it, land, and then fail to resume the activity, and the Forming Up routine in which a/c enter into the state, and then fail to progress to the next state, On Mission.

Seriously, we don't know if these are bugs or features. If they are features, no one can blame SSG or the playtesters, because they are obviously working correctly. The same can't be said if they are bugs, because they are obviously broken. One way or the other, the onus is on the developer to document these details, not the end-user, your generosity with our time and money duly noted.

Shame about your nose,

PoE (aka ivanmoe)

...Improvements, i.e changes which don't affect the results of the game, but which improve the experience for users and which can reasonably be implemented, will be made. An example of this is not showing the results of strkes in the combat screen until the animations have played out. We didn't anticpate the need for this, but its a change we're happy to make ...

This should be simple enough; a boolean expression that assumes all bomb/torpedo icons are invisible (false); then if a hit is scored during the attack, the state for that icon changes to True = Visible.

ORIGINAL: Mart

I'm sorry to see all this CAW bashing going on. I haven't played it, as it is on my "buy next" list, but that doesn't mean it's not possible to make some reasonable comment. The first is that CAW has a long and distinguished history and people have bemoaned it's demise for years, to the point that it practically had legendary status.

Well, the original game is still played. I have the original CAW, and I had CCAW at some point, before losing it in a spring cleaning (I still sob at the thought [:@] ) The original CAW is on my HD now, and it is still great - no "stuff of legends" at all, but a solid, concrete, good game about carrier warfare in WWII.

ORIGINAL: Mart
Now, when it comes back, programmed by the same team, with all the advantages that the original CAW had, plus some extra facilities, suddenly the AI is no good. "It's not like the good old days" doesn't really work when your arguing with the original programmer.

Time passes, and you expect something more. I would have expected, for example, more scenarios, like CAW I and II combined, and some sort of random scenario alterator/generator out of the box.

ORIGINAL: Mart
It's a game. It's pretty cheap. It's highly rated, has the same AI as the original which everybody worships, and there's no point telling Gregor that he doesn't know what he's talking about, unless you're suggesting he did the new programming whilst sucking on LSD tabs.

You can buy CCAW, today, for about $30 (I do not know, however, how legal it is - the internet site selling it smells of snake oil...) That package gives you the original game with all the scenarios, the editor, the AI editor and extra theaters like the Med and the North Sea/Norway (all with correct OOBs for ships and planes employed there by UK, Germany and Italy) - all wrapped up in a great game, running under XP (at least, my CAW runs well under DosBox) and still perfectly playable today.

I agree that, even given the new interface and the polishing up, for 2007 and $49.99 ($59.99 for physical shipment), and with the basic groundwork of game design (not programming of course, since it was redone from scratch) already done (and very solid) one could have expected something more.

I bought CCAW from that site and am playing it now.

I posted here before purchasing and no one (SSG or Matrix) advised me that the vendor was selling pirate software. In fact, I think the vendor is the authorized vendor of this product in North America. Of course, I suppose SSG and Matrix would rather be moving CAW (which I also have) than CCAW for $32.50. Of course, that's not hard thing to do. CAW is still is alive and not a legacy game. CAW simply needs some patching, a little more content, and some more extensive documentation.

Aside from reporting bugs, game play/balance issues, and some areas for enhancement, there is not much more that CAW customers can do to advance the future of CAW. The game's future is really in the hands of the developer.

I am having fun with CCAW since I got it a week or so ago. There is no reason why CAW should not have a bright future as well.

ORIGINAL: alexs

We recruit testers based on many factors... ............ [] we make mention of them in the patch notes and leave the rest to them.

I don't think ppl are/were objecting the quality of the testers regarding bug-testing, and I don't think that ppl claimed the testers had been lazy or anything...
But, to be honest, "leaving the rest to them" doesn't sound very good, imho.

While a developer relies on the testers to find unusual or unexpected bugs along the way (while they're playing it "free-for-all"-style), the developer should mark the "claim" where the testers are supposed to dig first, once a game hits the Beta phase.

But testers should also be encouraged to behave like what we call "mündige Bürger" by the dev, sorry for the German - this might translate to politically mature/responsible citizens, which means that testers should be encouraged to express thoughts about general approaches, game depth, amount of scenarios, content in general, frankly. So the testers should be proactive and describe opinions about (acceptance of) a given game, too, besides them being used for pure stress-testing or bug-reporting.
In my experience, it's a good idea to involve testers on an Alpha level, so that specific details or suggestions can be taken into account before putting out the Beta. While this can be frustrating for the testers (since Alpha versions are anything than stable and often have fragments of the game working only), they may come up with some valuable input.

From what I've read/observed in this forum, and from my experience playing the game, ppl were suspecting that the testers were rather hmm bound to hunt the bugs (which worked well) than ready to test the balance for MP or to analyze a situation where overall-content was subject to what ppl might call "lean production".
That's where ppl (customers) started to ask what the testers were doing, since customers are .... guess what .... customers, and they don't like to be used as testers.

While the dev has to give a general direction (to stress-test finished scenarios + play-balance, for example), testers should have the guts to say if something (regarding concept or content) is wrong or if there are any shortcomings ... the group of testers are the spotting planes for the developer's "carrier", in all aspects, but the dev is the SKIPPER. [:D]

I know it can seem incredible that some of these bugs that seem obvious arent picked up, but dont forget that we have been testing CAW for over 6 months now ... [] sometimes the forest can't be seen for the tree's (ie obvious bugs can stare us in the face).

Nah, again, that wasn't the problem. I work in the localization buiz and I know that such "forest"-situations happen once in a while. Let me tell you, that the 1 or 2 disturbing bugs are still kinda peanuts .... I don't play the game anymore, and it's not because of these few bugs.
After looking at all the screenshots MarkShot posted, I was tempted to buy the old game for 3 reasons: Depth, variety, transparency.
Example for Transparency: Excellent debriefing screens - screens that inform the player about what happened to lost planes, means infos about what branch/unit of the enemy forces had ripped apart the player's squadrons.

That's neat. While it adds to the immersion, it also gives the virtual Admiral the chance to change the a/c - setup, like changing the amount of escort aircraft, if the enemy units are still spilling fighter planes, for example. Sending escort fighters can really make a difference.

CAW's GUI is excellent, so I won't buy the "ancient" CCAW, but I really hope that things are going to be changed/fixed.
If things are planned to be "pimped" in a sequel only, I'll opt out, I guess.