A general post on PC Games and bugs...

[Erik Rutins]

Just felt the need to pontificate a bit today... nothing in specific brought this on, it's just been percolating for a while and some thoughts accumulating based on many posts I've read here and there. So here is an extended ramble on the subject which I hope will be illuminating to some and at least won't put too many of you to sleep or irritate you. This is basically stream of consciousness and not intended as a whine, but rather to inform as a number of the posts that made me want to write this were from customers who may not know what goes on "behind the curtain".

Developing software for the PC is a lot different in one sense from developing many other products. Your "platform" is a moving target with nearly infinite permutations. If your company makes kitchen scissors, you can reasonably assume that you will not have a customer who has hooves instead of hands or who wraps his food in sheet metal that your scissors can't cut, instead of aluminum foil.

Similarly, in a much closer example to PC games, if you develop games for the XBox, you can expect that your customers will all have the same XBox manufactured by Microsoft with the same hardware. Once the hardware changes, it's a whole new platform (i.e. XBox 360 or Playstation).

What I'm getting at is the perhaps now old yarn that developing for PCs is different. Not only is every PC virtually unique in its own combination of hardware and some of them are home built by folks with just enough knowledge to be dangerous, you have on top of that the drivers (which are in fact the cause of the majority of our technical complaints) as not only do folks not keep them up to date (many don't even know how) but even some of the up to date drivers have bugs or perform differently from the spec they are supposed to math.

Then you've got the operating system and its components, which come in a variety of flavors, with a whole new layer of complexity based on which version of Windows, which Service Pack, what Hotfixes may or may not be applied, what version other components like DirectX and .NET are at, etc. Finally, you have the third party software. Over 90% of our help desk complaints that read like "I can't get the game to install or run." are because someone has their computer's anti-virus/security software setup so that it won't even allow any programs to read or write to any part of the registry.

If we could convince the OS and Hardware manufactures to somehow automatically keep people's drivers up to date and we could convince the third party software makers to have their default settings be a bit less aggressive, I think our Help Desk volume would easily drop by half or even more.

Ok, so with all this said - right, we know that, we got in to the business knowing it and as professionals we do our best to make sure we work around it. Nevertheless, you have to keep it in mind because the reality is that we simply can't test on every single PC and we can't play every game exactly the way every single customer might play it, so things will slip through.

Of course, then there are the real bugs that we created. They exist and I don't think I've ever seen a piece of software that didn't have one, the real question is how many there are and how frequent or severe they are. Even in non-complex games, our pre-release testing often finds many, many bugs and sometimes fixing one bug creates another. We don't release a game until we are happy that as far as our own testing can determine, there are no bugs to be found. Unfortunately, this is as much an art as a science and while I'd say we are much better at it than we were when we first started doing this, there is still an aspect of it that comes down to a judgement call. We are more successful with that judgement call on some games than on others, so we've had some releases where it turned out we really had found 99.99% of the bugs pre-release and fixed them, but others where we only found say 90% and a lot needed doing post-release. So, as I said you release and then the universe re-confirms for you the difference between say 20 testers and developers and a few thousand very motivated customers is - it's big. Especially when you mix in the hardware/software/driver/OS stuff I mentioned above. And yet, even if the customers find some bugs, there's no guarantee that if we'd kept it in testing for another month or even a year, that with our mix of systems and tester styles we would have found the same bug.

This is the part where I think post-release support is simply necessary and a fact of life. We recognize that it is simply not realistic to expect that because a game seems "bug free" to us that it actually will prove to be bug free once customers start using it. We strive for that, but we try to provide the best possible support to quickly fix issues after release and make sure that if something slipped through, it's addressed quickly. I feel that this way customers know that even if they hit a bug, the game will be well-supported and that bug will not survive long or affect their enjoyment of the game long-term. This to us is the only way to truly make a game effectively "bug free" and its part of realizing that releasing a PC game "into the wild" is much like going into battle. Whatever your plan was before, whatever you thought you knew, you will have to adapt and change if you are to succeed.

On top of this, you have the "bugs that are not bugs", areas where a customer either disagrees with the design of the game or where he had different expectations. I think of these as "issues" and we try to address as many of these as we can, to make customers happy while keeping to the vision of the game.

For example, on our most recent release of WITP Admiral's Edition, we have not had a single confirmed crash issue yet. However, we had some interface lag issues come up (already partially addressed with a hotfix) due to how the game interacts with some kinds of dual core/quad core chipsets. Despite the fact that we have dual core system and even had a quad core or two on the testing team, this did not come up, as it was not an issue on those particular systems. There are some other data typo issues and balance tweaks we are working on, but overall I'm confident that we would not have found 99% of the reported issues so far by keeping it in testing any longer. The first release is an enjoyable, stable game that a lot of customers are getting a lot of entertainment value out of. Now our job is to improve the game post-release as quickly as we can.

Nevertheless, a customer could reasonably say "well look, they released another game with bugs". And that same customer could look at his XBox game (note that they do have bugs too, but generally are more bug free for the reasons noted above) and say "why can't these guys make a game without bugs like this one"?

I guess my point is that if making games for the PC were just like making games for the tabletop or for some other platforms, or making some other products, then we wouldn't have to deal with bugs either, or at least far fewer bugs. But the nature of PC game development and especially niche "independent" development/publishing where the development and testing teams are by necessity smaller due to smaller budgets, is such that it seems there will always be some bugs, even in a well tested, well-programmed game.

Sitting where I sit, I also know how many issues we do catch and address before release and I know that we never "pull the trigger" on a release until we're as sure as we can be that it's really ready for the customer to dive into. I can't promise our customers that every game will be bug free, I truly think that's impossible and I also know that we have made some mistakes in the past where a game was not tested thoroughly enough and we made a release decision based on incomplete information (we have learned from those mistakes, fortunately). I can say that we will do our homework before release to allow as few bugs as possible to slip through and we'll work with each developer post-release to provide the best possible support so that any issues are fixed as quickly as possible and your long-term enjoyment of the release is assured.

I will also say that, as many of you also do, I buy "mainstream" games. Some of them just interest me, others I buy to keep up with the "state of the art" and to see what the AAA publishers and developers are doing. Over the last few years, with a few exceptions, every such release that I've bought has in my experience been more buggy at release than our games are and often less well-supported. I have sympathy for the AAA publishers and developers, in the sense that they are pushing the cutting edge hardware/software wise much more than we are and thus are dealing with many more situations where drivers or other components just can't keep up. They also have to deal with retail deadlines and unfortunately retail in this day is a very unforgiving environment for PC games and the retail deadlines and policies probably cause more "buggy" releases than anything else. This is one of the reasons why we send fairly few games into retail these days.

Anyway, my point on that last general comparison is just that I think we stack up pretty well as far as delivering quality games at release and supporting them well, but we will continue to try to improve and find ways to be better at what we do with the resources we have.

If you read this far, thanks for listening.

Regards,

- Erik

[pvthudson01]

I have to say that in the 6+ years I have bought Matrix games I haven't had any issues with buggy games that crash or anything, but I have had issues with some games I thought should not have been released when they were. Mainly due to interface issues or things like the AI or such. However, that can be fixed. I know you guys took a lot of flak for World War One, but I am not sure about the entire story and what went down with the separate developers.

Also some of the games you release are rehashes of older titles that have been spruced up, so those I do not really count (unless you charge a lot of money for them and they offer very little new to the consumer...that I cannot stand).

The only thing I would say to watch out for is the older games that you try to fix up that may not run on Vista. That and the lack of the developers to show up here and respond to people and things like that. I think that gets on people's nerves more than anything honestly.

EDIT: And no, I am not the type of player that will notice things like a +1 modifier being applied to some unit that should not have it because he is 1 hex over from a swamp. I am talking cosmetically and how well a game runs. Can I click things, does the text overlap the chat window, are the units easy to see, does the tutorial match the text, etc etc. I will admit I am horrible at finding little itty bitty details that some people may complain about so as far as that aspect I am quite a newbie even to this day

[Greybriar]

Thanks for taking the time to write your rant, Erik. I appreciate what you wrote.

[JudgeDredd]

Read it all and as a developer, I absolutely agree 100%. I hope some of the people who believe in the myth of bug free software read it and digest.

[Lützow]

One thing which flows through my mind for some time. Is it really harder to make bug free games nowadays than 10-15 years back, or even in the early nineties? I mean back to the days there did not exist something like directX, dev's had to consider different architectures, as well as more graphic chipsets. When I started my gaming career with adventures, those games ran on XT/AT/386 and they had to work out of the box, as there was no way to distribute patches.

[JudgeDredd]

Oh there were patches alright - you just couldn't download them. I remember many months of waiting and subscribing to magazines to get patches on CDs (if I'm truthful, I remember some patches soming on floppy disk!).

I think, as Erik alluded to, that ALOT of it nowadays is down to the mix of 3rd party and OS stuff. Look at the .Net stuff you have to have on your PC for somethings to run! Anti virus software - firewalls...in particular "security" software - they are designed to "lock down" your system. This was not the case in the early nineties.

[Greybriar]

You're truthful, JudgeDredd. I remember receiving patches on floppy disks back in the '90s from game companies.

As I recall, Windows was touted because it was supposed to do away with the conflicts due to different system configurations. Remember Plug & Pray? [:D]

[EwingNJ]

Well Eric, what you wrote is all well and good. But you really borked up AE. How? Perusing my list of Allied pilots it seems about 10% have the same last name. Yep, you guessed it, Rutins. [X(] Everyone knows that no one from the Granite State knows how to either fly or shoot straight. [:@] Thanks for nothing.


Oh, also, I cannot think of another company that is as responsive to its customers as Matrix. You have my appreciation. Keep fighting the good fight.

[IronManBeta]

Speaking as a developer who has made a game for Matrix and is working on another I can cheerfully say that I agree 100% with Erik.  It take ferocious and unremitting attention to quality to keep the bug count down and heaven help you if you ever let your guard down.  No code change, no matter how small, will ever fail to bite you one or more times before the game ships.

I've been programming for going on 30 years now and the technology has improved beyond all recognition but the daily grind is remarkably similar.  Imagine the code change, plan the change, make the change, test the change, document the change, release the change, explain the change to those who don't read the dox, then get buried by complaints, gripes and genuine bugs large and small that you just didn't see or imagine at the time.  Add the worthy ones to the to-do list.  Repeat until you grow old and die. 

My interest is in modelling the battlefield and in coming up with interesting AI mechanisms built on that understanding.  I probably spend less than 1/2 of 1% of my time actually doing that.  The bulk of my time is spent doing the loop described above for UI (in a general sense) code, and running the game to see where it is going to deviate from expectations next so that I can trace down and rewrite some obscure routine for the umpteenth time.  It is a painful but necessary tax to pay in order to do the fun stuff.

Software development has come a long way over the years but it sure has a long way to go.  I'm really, really proud of the low bug count in Flashpoint Germany and I worked super hard to make it that way but I don't flatter myself that it is perfect.  I've read that with a page or less of code it is possible to program something that would take longer than the universe has existed to debug completely and I believe it.  As Erik says, in the end it becomes a matter of truthfully being able to say 'is the game good enough in it's present state to ship?'  The huge advantage that Matrix enjoys is that David Heath laid down that no game ships until the developer feels it is ready.  With that protection we really can just put our heads down and work on it until it is done to our satisfaction and not be stampeded into a premature release.

Just my $0.02 worth, cheers all, Rob

[knilli]

there are those 2 departments, PC and software on one side and the game on the other side.
though...as a customer i have a slightly different angle of looking at it. i am forced to believe what the publisher put in its advertisements and screenshots, i have to assume that if the publisher releases a game it is playable. said so...i have/had games where the patches downloads exceeded the size of the game, only because the publishers did not merge the patches (i bought this particular game very late, so all the patches where already out)...hey...i do not have unlimited download!

as a customer i differentiate a few "game / software scenarios":

1. i bought a game and like it, works fine, just some minor patches have to be applied
2. i bought a game and discover i do not like it, advertisement and my expection are not the same
those were the easy ones

3. i bought games with a multiplayer part announced and advertised ....i still wait for it (anno xxxx)....but was impossible to give the game back. never ever bought a game from this company again...in my eyes dishonest advertisement
4. games where the download volume of the patches exceeds the size of the game to make it somehow playable (borderline, see above)
5. games so buggy that you could not play it and no support
those where the "once in a life time" mistakes

6. games with a lot of potential and good support, but released too early (aegod ww1 for example). very dedicated team and very hard working to fix it + willing to pay money back ...have no problem with this...."sh.t happens" and as long as it is not an everytime case....no influence on me
7. games where after the release problems occur, very often because there are probably as many PC configurations as there are sand grains on the beach...no problem with that either...as long as you see the team behind willing to fix it
those are the "annoying" but not really a problem ones

8. games with DRM, phone/online activation: i do not buy games like this. i do not like this particular business model (see below)
this category is a no go for me

9. more general:
as i am playing since C64 times...i have seen a lot, liked a lot and was frustrated a lot. what i like with matrix, and that is the reason why i stick around and finally bothered to register to the forum, you do not feel abandoned (and no i do not want to slime in). there is always someone (staff, users, devs) to give you help/answers. another reason why i am here is, i like matrix's  businessmodel, i hade DRM and internet/phone activations. i do not spend my hard earned money on games/programs where i need  to do that. i am not a criminal and i am not a baby. you sell, i buy! i see the problem (as i see the problems the big music producers have, hey i work for a community radio station, so i know the problems) but to tread me like a "maybe criminal"....sorry no money from me.
And, that's where my subjective judgement comes in, DRM/activation in a game equals in "no good game" = i do not buy.   though, i think that devs have to life with the fact that, from a consumers perspective, the way their product is sold, affects the reputation of their game too!
i rarely was disappointed when i bought a matrix game (actually it was one game which turned out to be different from my expectations, but hey, that is very subjective), never was disappointed with the quality of the product.

[AttuWatcher]

ORIGINAL: Erik Rutins

Anyway, my point on that last general comparison is just that I think we stack up pretty well as far as delivering quality games at release and supporting them well, but we will continue to try to improve and find ways to be better at what we do with the resources we have.

If you read this far, thanks for listening.

Regards,

- Erik

I think Matrix stacks up better than well. Don't get me started talking about "mainstream" [:'(]

I'm a casual collector of retro arcade game PCB's (circuit boards). These are games that flash on the screen "insert coin", there is no cover charge, they have to earn their own money, they have to work again and again to have any hope. Even some of these games, games hardly more complex than Galaga, made in the late 80s to mid 90s that are completely self contained other than an interface cable connected to an arcade cabinet STILL have obvious bugs.

So anyway, bugs are a fact of life, developer/publisher pre-release QA and post-release support/patches are not. Be thankful when you get them.

[Prince of Eckmühl]

ORIGINAL: FAsea

I think Matrix stacks up better than well. [:'(]

IMHO, this is very much developer-specific. Matrix publishes games from some developers which obviously take great pride in their work. Among those, I'd make note of Panther Games and Koios Works. Game from those guys are essentially bug-free out of the box. Post-release patches tend to focus on new features and content, rather than a panoply of bug-fixes. I won't make note of less diligent developers as it would only cause a row. But, I think that it hurts Matrix when they don't hold all their developers to as high a standard as that embodied by the good ones. [:)]

PoE (aka ivanmoe)

[E]

As I recall, Pong was bug-free! (NO! do NOT deflate my happy memories of throwing quarter after quarter at those damn pixel paddles!)

[JudgeDredd]

Actually Pong wasn't - some t**t released it with a square ball...I MEAN!!! Tennis with a square ball? Now that's an oversight of enormous magnitude!! [:D]

[Joram]

Hi Erik, it's always refreshing to hear the publisher/developer viewpoint.  I agree with you 100% regarding crashing/lag type issues.  I know the developers behind most of matrix games are quite small and lack the resources as you say to thoroughly test the products.  You can knock out 90% of the bugs within a few months, but a few years may still only net you 95% with the resources available.  For that reason I am pretty forgiving for Matrix games developers (and other 'small' companies).. 
 
With that said, I am very unforgiving to bigger developers as since they can sell tens if not hundreds of thousands, then they can certainly afford to put in some more testing into it.  I won't name any names as that will just get us off topic but some releases by big name companies since the beginning of the year are a case in point.   Those companies harm the industry as much as they help it.
 
I'm not talking flash over substance, that's always a matter of customer preference.  I'm talking about basic gameplay mechanics not working, even as designed.  For these larger developers, that's unexcusable.   For Matrix developers and other smaller companies, I am willing to be a bit more flexible.  Do keep up the good work!

[Adam Parker]

Designing a PC war game is like managing the Red Sox.

(Twiddles thumbs)

Rant over.

[Adam Parker]

I feel better now too!

Actually what I really wanted to write is that if we leave the AI out, isn't there less chance of things going wrong?

Shouldn't we try it?

[Arckon]

ORIGINAL: Adam Parker

I feel better now too!

Actually what I really wanted to write is that if we leave the AI out, isn't there less chance of things going wrong?

Shouldn't we try it?

I don't believe we should as the only way for AI to improve is if it is being programmed. Granted it will be many, many years before significant progress is made, but if it isn't being programmed it will never get there at all.

[JudgeDredd]

ORIGINAL: Adam Parker

I feel better now too!

Actually what I really wanted to write is that if we leave the AI out, isn't there less chance of things going wrong?

Shouldn't we try it?

I think those are different kind of bugs (though bugs all the same). It would reduce it, sure...but not kill them off. Bugs are there to stay. Like it or not. The difference between companies is how many are there on release, is a game released with game breakers and how quickly they fix those found.

I know several mainstream companies who completely abandoned their game on release - some have issued one patch when the games quite clearly needed more.

I think as a niche market, the wargaming community has alot to sing for. That why when I see (sometimes senseless, irresponsible) slagging of any wargamer devs it irritates me.

And just as a big thumbs up, I have several wargaming sites I buy from - like I said before - I don't care who the dev or publisher is - if it's a game I like the sound of and I have the cash - the buy will be made.

Matrix is my fav publisher - not because of the games you release, as I have a couple of other wargames out there that are far better than anything I have from Matrix (apart from COTA and Forge of Freedom....and Kharkov DotD....and Panzer Command:Kharkov...and Uncommon Valour.....and GG:WbtS!!) - but because of the support and customer interaction you have.

[jackx]

ORIGINAL: Joram

With that said, I am very unforgiving to bigger developers as since they can sell tens if not hundreds of thousands, then they can certainly afford to put in some more testing into it.  I won't name any names as that will just get us off topic but some releases by big name companies since the beginning of the year are a case in point.   Those companies harm the industry as much as they help it.

Maybe, but they're probably still acting rationally. They're big, and so they have high running costs, and only part of their personnel can be used for bug-hunting/fixing, and if you combine that with a mainstream audience, which is far less demanding with regard to long-term support, releasing unfinished products in rapid succession seems to be an appropriate business model.

Not sure how that's going to change, or how one could actively change it, but customers are probably a large part of the equation - and since we're talking strictly non-essential products that nobody needs to buy, so long as people are buying, those who are selling are doing at least something right. ;x

That's why these days, I usually give multi-platform releases a wide berth, even if they look interesting. The PC version is probably the most buggy, but also the least-selling, and so patches/support are very unlikely - much rather, you'll get DL content for the console versions, as that is more profitable.

[/stream of consciousness vaguely related to the subject (sounds better than "rant", doesn't it?)]