Playtesters?

[Prince of Eckmühl]

The game credits cite the members of the playtest crew. Are any of you fellas out there?

PoE (aka ivanmoe)

[Rob Gjessing]

Most of them hang out at the Run5/SSG forums rather then here.

[RedMike]

Yeah...why?

[JSS]

I see a healthy portion of us here too.

[Prince of Eckmühl]

First, hello, and thanks for responding.

I inquired because there's a lot of uncertainty as to how long it takes to perform certain tasks, arming a/c for instance. Do you guys know of any documentation that we (players) can get our hands on that might clear some of this up? Again, the rest of us have no way of knowing what's going on with the internals of the game as so many of these particulars failed to find their way into the game manual. Even if no such documentation exists, I'll just about guarantee you that new players would welcome your insights into the game.

Thanks again,

PoE (aka ivanmoe)

[e_barkmann]

I'm not aware of any available documentation about internal routines so it's probably best to ask those kind of questions directly to SSG.

cheers

[Richard III]

They also seem to be playing a different, bug free, well documented version of CAW and those of us here.......[8|]

[Oleg Mastruko]

And they played each scenario like once, vs AI [8|]
 
(No need to answer.)

[Prince of Eckmühl]

ORIGINAL: Oleg Mastruko

And they played each scenario like once, vs AI [8|]

(No need to answer.)

Hey, don't go hatin' on the playtest brothers! [;)]

Seriously, there's no telling when last it was that any of them touched the game. And everyone would do good to recall that the forum was an unhappy place for months before the game was released because so many members insisted that THE GAME WAS TAKING TOO LONG TO COMPLETE!

If anyone's to blame for the state of the game that I purchased for $59.95+s/h

http://www.digitalriver.com/dr/v2/ec_Ma ... CACHE_ID=0

It's likely NOT the play-testers.

PoE (aka ivanmoe)

[Oleg Mastruko]

ORIGINAL: Prince of Eckmühl

ORIGINAL: Oleg Mastruko

And they played each scenario like once, vs AI [8|]

(No need to answer.)

Hey, don't go hatin' on the playtest brothers! [;)]

LOL [:D] You're right, lets all be friends



Playtesting can be very ungrateful process, I know, but honestly I cannot imagine a wargame that is EASIER to test than CAW. I can literally play thru ALL scenarios in about two hours, and that's without cutting any corners, real serious play. Compare testing CAW to testing, say, WITP, where you need months of real time to finish half the campaign.

(I played several AI vs AI tests in WITP, on my office PC during the nights and weekends - it took around 6-7 days for PC working 24/7 to finish a game!)

CAW is a good game. It oughta have been tested more thoroughly in MP environment, and in all honesty, should have been priced lower for what meager content it offers. Some improvements over the old CAW would be welcome as well (see my threads Speed and Bombardment) just so we see it's not a carbon copy design. Other than that, no complaints from me (even though those are pretty substantial complaints).

[Prince of Eckmühl]

ORIGINAL: Oleg Mastruko

Playtesting can be very ungrateful process, I know, but honestly I cannot imagine a wargame that is EASIER to test than CAW.

Oleg, I understand your sarcasm, but I don't think that you get it. YES, many of the broken parts of the game do sorta JUMP off the screen at you, but that doesn't mean that the play-testers were in any position to catch the stuff. They may not have touched the game for months or years before it arrived on our desktops. It also might be that there was a patch in the works before the game even shipped, one that wasn't applied because it would have postponed payday for the developer. Did you ever consider those possibilities? [:)]

PoE (aka ivanmoe)

[Rob Gjessing]

Couple of comments and responses here:

- The testers mostly hang out at the SSG/Run5 website because thats the official SSG website and we are SSG testers. But as JSS mentioned, some do also read and post on these forums.

- If there was any internal documenation then we would not be allowed to release it without the support of SSG; so as Chris points out you are best to direct those questions to SSG.

- Re the comments about the quality of testing that occured on this or any other game. Testers are simply that - testers. We may make recommendations and provide feedback but we dont have any power of veto nor are responsible for design decisions, so dont blame any testers for any failings you may or may not think that there are with this or any other games.

[Prince of Eckmühl]

ORIGINAL: Rob Gjessing

Couple of comments and responses here:

- The testers mostly hang out at the SSG/Run5 website because thats the official SSG website and we are SSG testers. But as JSS mentioned, some do also read and post on these forums.

- If there was any internal documenation then we would now be allowed to release it without the support of SSG; so as Chris points out you are best to direct those questions to SSG.

- Re the comments about the quality of testing that occured on this or any other game. Testers are simply that - testers. We may make recommendations and provide feedback but we dont have any power of veto nor are responsible for design decisions, so dont blame any testers for any failings you may or may not think that there are with this or any other games.

Well, Oleg, its a fine mess you've made of things this time.

First you kill Hamachi and then you disconnect us from any meaningful interpretation of CaW's otherwise inscrutable game routines.

Gloom, despair and agony on you!

PoE (aka ivanmoe)

[MarkShot]

Beta testing ... now there is a topic that I know something about and I also happen to work in the software industry:

(1) Arrangements between developers/publishers/testers vary incredibly between one situation to the next. In some cases, testers are strictly that and are largely ignored by the developers. In other cases, testers a very much involved in the design and evolution of the product. Their responsibilities extend to everything short of writing code such as: research, scenario design, tutorials, writing manuals, etc...

(2) The approach to recruiting testers can be very rigid and formal involving NCA, NDA, handing out task lists, due dates, assignments, and hours required. In fact, I was connected with one developer where it seemed like I just accepted employment (without pay) for a job that was a mistake. And other developers, just give you access to a private area with no paperwork to sign and say "here play it and have fun; let us know what you think and if there are any problems". Some developer lose 50% of their team with each project due to burn out and others have members with them 5 years or more, since they understand the special needs of managing volunteers.

(3) You have the testers themselves. Some are incredibly motivated and work 20 hours/week without pay, because they simply love the game engine or the subject matter. Others barely do more than install the game and it was just an ego trip to get early look at the game or see their name in the credits. Some bring incredible skills of amateur or professional historians to a team, others bring years of gaming experience, others bring familiarity with software projects and product development, and other bring experience writing and developing guides and training materials. The smart developer/publisher learns to assemble a team of varied talent and create a culture where individual talents emerge and are fully utilized; and that turn over is reduced or minimized.

(4) The manner in how is game is tested is equally varied. What do testers test and who do they choose it? 90% of the time, the test team are the ones who know the engine best and are your hardcore grog players. It is no wonder that easy and basic options for games receive very poor testing coverage. Additionally, the testers often fail to do obviously stupid and counter productive things that noobs do, because they know the game. So, CTDs which commonly occurs when the game go gold may have never been seen before.

(5) If a game engine/scenario creator allows for a lot of randomness, then it is very hard to compare that changes from introduced in the game play from one build to the next. How much is normal randomness and how much is new code? How good has the developer designed their product for testing anyway? Are their tools just for beta testers to use? Tools that lift the FOW or provide deeper insights (execution logs) into the AI behavior than the customer will see? Or are beta testers getting now more access to the internals than the customer and just being depended on to make better inferences due to their greater experience? Does the developer of automated testing facilities/harnesses for the game and test suite to rapidly test any new builds?

---

I could go on. Software development is one of the most complex engineering endeavors undertaken by the human race. Independent game development is often done on shoe string budgets with volunteers who like the customers have spouses, children, parents, jobs, and other committments. Most of the game developers I have personally known are doing it more for simply the love and passion of the artform than for any intention to get rich or build a highly profitable business. For if most developers who are talented enough to build complex wargames really wanted to be financial successes, they would well choose another line of work. For me, this is why I have only beta tested. I am much too focused on my personal financial security to throw all caution to the wind.

So, our hobby is filled with too many crappy and poorly designed games. It is also filled with too many games which are atrociously buggy. It is filled with too many developers/publishers who never really finish/fix the product that they release. This is all true. But given the chances of making any money at this, we enjoy playing these games should be thankful that anyone is foolish enough to try this line of work at all. And that anyone who enjoys playing such games would give up their personal free time to step through CTD after CTD and document the exact conditions to reproduce rather than having a fun time to do something else.

---

Well, I hope that gives a little insight to what goes on behind the scenes with those "inept" developers and "bumbling" beta testers. Thanks for reading.

Disclosure: I am not a beta tester for SSG and anything which I said above that bears and resemblance to SSG or any other company is purely accidental.

[Prince of Eckmühl]

ORIGINAL: MarkShot

90% of the time, the test team are the ones who know the engine best and are your hardcore grog players.

I just wanted to comment on this point from MarkShot's excellent post, above.

Developing a full-featured game is such an enormous task that the essential personnel in the project, artists, designers and programmer, often have NO time to actually play the game. Because of this, the package can get away from those central figures. They are so focused on the minutiae of bringing all the parts together, that they simply lose track of what's going on with the "big-picture," and, in particular, with how the game plays. IF THEY TRUST THEIR TESTERS, they'll make in-course corrections as they proceed. If not, the end-user can wind up with hash on his hands.

Again, nice post MarkShot,

PoE (aka ivanmoe)

[JD Walter]

Thanks for the insight, MarkShot.

I negotiate contracts for a major corporation, and am surprised at the similarities in drawing up a fair, balanced, well-written (i.e., not subject to two different, equally valid interpretations) corpus of terms and conditions, with what you describe above.

From time to time I think about re-entering the game biz ( I was an initial investor & playtester in ICE), but realize I have limited experience in the computer-side of it, and am probably best as a player, not a tester.


(But I am good at documentation!) [;)]

[GoodGuy]

ORIGINAL: Prince of Eckmühl

one that wasn't applied because it would have postponed payday for the developer. Did you ever consider those possibilities? [:)]

Well, I think I just spotted your first hmm less bright statement. I, as a customer, have no sympathy for that kinda situation. That might be cruel, but can be explained easily: If you go to a store and let's say you buy milk, you don't want to find out that the product you just bought turns out to be adulterated milk once you open it at home, although the "best before" seal made you think it was fresh milk, right?

You trust your fav store that it stocked the milk correctly, and you trust the transport company that they turn on the cooling devices on their trucks during the transport. Also, you would not accept an excuse from the producer, like "hey, but we did test the milk before it left our dairy facility, so we're not part of the deal anymore".

Product liability laws in Europe use to grant the right to return products that are missing advertized features or which are featuring malfunctions. Funny thing is, this is valid for fridges, radios, mp3 players and what not, but (here in Germany for example) once you've opened a sealed game/software box, stores won't take it back to give a refund. If you get loud (so that other customers would hear it) and if you quote a few law paragraphs some stores might give in, but they would just give you a voucher to purchase another game.
Some day a customer will go to court for some "lousy" (in lawyers + courts' views) 50 bucks, and fight for the right to get bug-free software. I am sure.

I accept your point where you said ppl were trying to force the release in the forums, but I wouldn't accept the "payday" deal as a criterion for not applying vital fixes prior to release. In fact, I couldn't care less about a dev's payday (even if developers are tiny companies or even one or two-man-shows only, like in this niche-market), as I've got the right to get a thoroughly tested / solid programmed piece of software for my money, in my books.

Oh, last but not least, humans (programmers in this case) aren't perfect, they do make mistakes, sure.... understandable.
But if the fridge I just purchased has a malfunction, I get a new one within 24 hrs, or the store who sells it has the right to repair it (up to 3 times in Europe), before the seller has to grant a refund. Well, that's still faster than months with a series of patches for many games these days. Oh yes, software is more complex than a simple fridge, but if a software company (like let's say EA) with fully fledged Q/A divisions isn't able to test/verify things, quite frankly, it should produce/sell fridges.....

Any other company (that's not in the software business) has to deal with that pressure on the market. Software companies still get away with their "there are so many possible hardware combinations, we can't test it on all possible combos"-excuses. It's about sloppy programming, rushed out games and about bad testing procedures (lack of time to test a given product, too) in most cases.
The majority of console games are relatively bug-free, as there are less chances to patch things. The amounts of bugs in such games is one of the major factors which determine success or failure on the consoles. And yes, I'm well aware of the fact that console games contain more and more bugs these days.

[e_barkmann]

well for my part I concentrated on testing multiplayer stability/functionality late in the dev cycle, and there was certainly a lot of attention paid to this by SSG.  Happy to answer questions on this aspect [:)]

cheers 

[Prince of Eckmühl]

Hi GoodGuy,

I'm perfectly willing to accept the notion that some of my comments are "less than bright," but the fact of the matter remains that the developer and publisher are responsible for the quality of the product that we purchased, not the play-testers. If I strike you as overly sympathetic to their plight, it's because they are as grossly overworked as they are woefully under-compensated. As for my suggestion that a patch might have been in the works prior to the game's release, it is but one of myriad possibilities which might explain the state of CaW as I have known it, one that takes a little tar off of the testers.

Thanks for your comments, [:)]

PoE (aka ivanmoe)

[Prince of Eckmühl]

ORIGINAL: Chris Merchant

well for my part I concentrated on testing multiplayer stability/functionality late in the dev cycle, and there was certainly a lot of attention paid to this by SSG. Happy to answer questions on this aspect [:)]

cheers

Hi Chris,

When playing online, are both players file-sets used to set the parameters for the game, or just the host?

PoE