Or, why bugs in new products are regrettable but almost inevitable and why developers shouldn't use that as an excuse for poor design choices

Foreword:

Flight simulator developers and their customers share a lot of common interests. Although you would never know it to read some of the posts that appear in the online message forums. In particular there seems to be an alarming trend that accompanies the release of each new sim product lately: loud and violent protests at developers from customers angered by perceived bugs or "mis-features." This is lamentable and is symptomatic of behaviors on both sides that will over the long-term serve to weaken the industry that supports this hobby if we are not careful. Rather than assign blame, a better approach would be to find ways to improve customer satisfaction and developer rewards. Both constituencies can learn a thing or two here and we can all be better off.

You will undoubtedly have seen the posts that denounce developers for releasing products that are chock full of bugs. Some of these posts get amazingly vitriolic and occasionally even ascribe fraudulent or other malicious intent to developers. Such posts usually assert that the product cannot possibly have been tested properly or perhaps that the beta test crew is entirely incompetent and would not know a bug if it bit them on the nose. You've seen the sort of thing I'm sure.

Consider a fundamental truth for this hobby: the developers are in business to make money. They do this by creating products that sim enthusiasts will buy. In this context, it's almost too obvious to suggest that customers shouldn't buy (or at least should be prepared to return) products that don't appeal, for whatever reason. Market forces will soon dispatch developers who aren't able to match customer demand. Of course it's not quite as simple as all that and the question becomes what can developers and customers alike can do to augment their cash-driven interaction at the unsympathetic retail point of sale.

"Customer flight: reality check, over…"

If you are occasionally tempted to fire off a post like this, there are a few things that you might want to consider before you hit the send button.

Click the image for a larger shot..

Is the object of your ire really a bug or is it rather a feature that you don't like? If you've worked as a software developer, you've undoubtedly seen the cute cartoon that shows an insect dressed up in a coat and tie with the caption "That's not a bug, it's a feature!" Which is to say, sometimes it's hard to tell the difference in a world where one person's bug is another's favorite feature and black and white answers may not be obvious. However, here's a rule of thumb that might be useful: if it works in the way described by the documentation, then it's probably a feature; if not then you've got a bug.

[ Footnote: Now you run into a small problem if the documentation is silent on the problem you're looking at. This should tell you some of why I view the documentation as such a vital part of any sim product. Arguably, undocumented features or bugs are first and foremost bugs in the documentation! ]

Assume you find a genuine bug. In this case you have a legitimate reason to gripe to the developer and what's more they should listen (of which more below). To borrow an example though: there's a big difference between saying "your view system is broken and that sucks!" and "when I go from an outside view back to internal view the mission ends abruptly, even though I didn't get shot down and the only key press I made was to command forward view." This may show the extremes of the spectrum but the point here is simple. Unless you explain what is broken, it's rather hard for the developer to fix it for you. Back up your complaint with enough explanation that the developer can replicate the problem and your chances of getting a fix increase dramatically.

Before writing off a new product on the grounds that it's "buggy" however, you need to consider whether the bug(s) really reduce the value of what you have to nothing. If a sim is so badly flawed that you can't complete even a single mission or whole hosts of features just plain don't work, you've got a dud and it deserves to be returned or not purchased in the first place. Slow sales and or a flood of returns should signal that initial product quality is important to this community.

Beyond that though expecting a new product to be perfect out of the gate (however desirable that might be) is just not realistic when sim products are composed of the large number of lines of code typical in current releases. Nevertheless expect developers to fix bugs with patches. Let them know that you expect this as well. Favor developers who make a common practice of releasing patches and sustaining the life of their products in the market with successive patches. More importantly bypass those who don't or do so only in extremis as a one-shot effort.

Features that you don't like are a harder proposition all around. In these cases, you still need to provide objective data to explain your issue to the developer; again without this you stand little chance of getting a fix. However, you also need to be realistic about expectations.

Simulation products are very complex software "machines" with thousands upon thousands of lines of code, all of which are delicately bound together as a whole. To change any one part could cause entire subsystems to come unraveled. The bigger the feature the more likely this becomes. One favorite example much under fire lately is the topic of padlock views. While it may be obvious that tying the ability to padlock bogeys to first having a radar lock is a bad design choice, actually changing the way the sim keeps track of potential padlock targets in visual range may require extensive changes. This situation is worse still if there was no padlock included in the first place but you think it should be there.

What of the testers and their apparent blindness? It may surprise you to learn that developers take feedback from beta testers very seriously; well most of the time. In most cases though, all a beta tester can hope to contribute is helping to identify genuine bugs in a near-complete product. For the most part, design decisions such as the padlock example above are long ago cast in stone. This can result in a situation where a tester tells a developer that some aspect of the design is poor enough that the product should not be released until a fix is implemented while the developers cannot hope to respond even though they might agree.

The reason this can occur is that the window of opportunity for changes has already closed by the time a beta version is ready. These products are developed as a business proposition. By the time beta versions are ready, the majority of the R&D budget for most sims is already spent. There just isn't any more time and money to develop fixes for features testers don't like unless the product can first generate some revenue (and in some cases, maybe not even then unless the initial release sells well).

Beyond this it's not realistic to expect some features to be added to an existing product. For example, don't expect a sim to suddenly sprout a dynamic campaign in a follow up patch. Unless originally promised anyway, expect to pay more for such major additions that will only come along as next generation or add-on products.

Speaking of paying for sim products, it's also common to see complaints about high initial purchase costs or having to pay for add-ons or upgrades. Recall, for example, the recent hue and cry when it was suggested (erroneously it seems) by a USENET poster that Falcon 4 might cost well over US $100 when (and if?) it's finally released.

Click for larger image.

[ Footnote: hue and cry -- the literal reference to the "pursuit of criminals" seems ironically apposite here ]

Why should this prospect cause such alarm? We are conditioned perhaps to expect an average sales price of approaching US $50 for new releases. While that may be true, it doesn't of necessity mean that some other price might not be reasonable. Contrast the cost of a flight sim that you may spend tens or even hundreds of hours with against the cost of a ticket to see a two-hour first-run movie. Consider also what you expect to get back from investment in a new sim. Is it not worth a little extra to you to see a few more corners knocked off that rough diamond to expose the brilliance of a new sim or are you content to see yet more products miss their full potential? It seems that at current price points we often see wasted potential but true diamonds are few and far between.

[ Footnote: Would be critics will be quick to point out the example of Back to Baghdad as giving lie to the argument here. In this case it's hard to argue that higher initial cost delivered a commensurate return in terms of a more complete or better-polished product. Ultimately though this counter argument misses the point: a product with a higher initial or follow cost that does deliver more than one of average cost is worth more. Discerning sim customers will likely follow a winner and price point be hanged; equally high-priced but empty products will find their way to the bargain bins quite quickly. ]

"Developer flight: radio check, over…"

If it's fair to say that many flight sim customers could use a little more perspective, it's equally resonable to say that some developers could stand to acknowledge and listen to customers more. Sometimes you have to wonder if developers are even on the same wavelength as their customers at all.

It seems to be the nature of the software business in general that releasing bug-free products is the exception rather than the rule. This is lamentable and every developer with any sense of professional pride should be searching for better ways to deliver higher quality products.

Without inside information it's difficult to determine with authority what the issues are that impact software quality but there are some clear observations that can be parlayed into questions about the software quality processes that are in general use in the industry.

For example, at least two products based on the Windows 95 platform have been released only to discover that on large numbers of customers' systems significant performance problems exist. You'll be familiar with posts decrying "stutters" or outright pauses in the games in question. It's a fair bet that this behavior wasn't specified in the original design.

While the very nature of the commodity PC-AT compatible market means that it's impossible for developers to test every possible hardware combination for compatibility, it certainly seems as though the variety system configurations tested is often lacking. Developers would do well to test on a better variety of systems with a wide variety of tests. It seems from observation that test team personnel in house at developers are often countable on one hand. The number of system configurations available for test is also apparently quite low. Adding more of both is clearly impractical in many cases since the additional cost would be prohibitive. Once again, the business model for the product would be compromised.

Click the image for a larger shot...

There is at least one fairly obvious answer to this dilemma. External beta testers are often more than willing to help out and typically will agree to do so for little to no (financial anyway) consideration. Currently the typical external beta team might include 10 or so people. What might help software quality considerably is to change the numbers involved, not by two or three times but by an order of magnitude or two.

The extreme example of this perhaps is the Air Warrior II (AW) open beta approach. Now, the AW business model for return on investment is a little different to the more traditional retail sales channel tack but the principle is there nevertheless: get large numbers of customers to do testing for the developer. If it's a concern, there are plenty of techniques available to protect the software from out-and-out piracy during such a beta period that will deter the casual pirate.

[ Footnote: Sadly, the determined pirate is likely to steal work regardless of whether it's in beta or full release. ]

If the developer has confidence in the product, there should be more upside to increased testing than downside from exposing the product early.

[ Footnote: Now of course the beta may not be well received which could serve to shake a developer's confidence in the sales potential for the product. However, knowing there are problems in the appeal of the product before the launch offers the opportunity to rethink the release. Depending on how the beta team is picked though, this can probably happen regardless of test team size. ]

Even with unlimited testing resources though, bugs can and will make it into released products. In such cases, customers expect fixes, usually in the form of patches. Ready (if not universal) access to the Internet makes distribution of patches more practical and cost-effective than ever before. Today there's really no excuse not to build into the business model for a sim the ability to support it after release with follow up patches to address problems.

If it changes the cost basis for the product to build in post release development support then charge more for the product. This may sound like heresy to some customers but there is considerable value in knowing that a product has life beyond the initial shrink-wrap. Of course there's a little quid-pro-quo here: a clear public statement that additional cost is built in to original purchase price so that follow on development can occur is likely to beget a better reaction than silently hiking the price. And then of course, delivery on the promise must happen. At least one developer has alienated many customers by charging over the going rate for an initial release with the promise of follow on work that never materialized; they'll have a hard time getting repeat business now and rightly so.

Bug identification and fixing is one thing, but design choices that define the essence of a new sim product are quite another. As mentioned above, beta test comes far too late in the development cycle to be of any value in steering the basic design precepts for a product. Such decisions have to be taken early in the life of a product; in some cases perhaps even before the first line of code is written. A distressingly high proportion of new releases comes out with features or design trade-offs that seemingly defy logic.

If beta test is the first opportunity that customers have to get involved and feature decisions are already set by then, it seems there's no way to break the impasse. However, what if customers were involved in the project before the design was locked down?

Microprose has recently demonstrated a limited form of this sort of thinking with the customer focus groups that discussed Falcon 4 (as subsequently reported on the Internet and elsewhere). It seems that in this case the developer has shown a genuine interest in getting customer input before it is too late to do anything with it in design terms. Kudos to Microprose for taking this bold step.

The details of a forthcoming design are perhaps trade secrets or intellectual property every bit as much as the actual source code for a product, especially in advance of the actual release date. However it is not unusual for software industry companies to share early design specifications under non-disclosure with customers to help ensure that the eventual product will hit the mark. Given that the average age of computer game buyers is quoted as being somewhere around 30, it seems reasonable to suppose that there are a few responsible adults in most sim developers' customers bases who could fulfill this function.

In the spirit of an ounce of prevention being worth a pound of cure, many would-be sim customers would probably welcome the chance to help steer a developer away from a poor design choice early in the cycle. It's much easier to fix a line or two of text in a design spec than it is to change hundreds of line of code after the fact.

"Cleared to land…"

The sort of loud denouncements that regularly seem to accompany new releases are hardly constructive and ultimately pointless. As customers we should be striving to help developers understand and correct issues so that products get better rather than simply venting; behavior more likely to have committed developers questioning why they bothered in the first place. And developers for their part should think about deepening their relationship with customers; more than one major enterprise has discovered the effect on sales and stock price that concerted voices expressing discontent over the Internet can have.

You may not agree with any of the ideas suggested above but if you have made it this far, perhaps you may be tempted to re-evaluate some of your own thinking in this area. If that is so then this article has accomplished its purpose handsomely, be you a developer or a customer.

Editor: In real life Mark manages a team focused on application software performance optimization for 3rd party server and workstation applications that run on Intel Architecture. You can send him comments at Mark Doran or send a letter to the Editor.

Ah, yeah, thats me feeling a bit lost in Flying Corps 3d..

Main    Back

© 1997 - 2000 COMBATSIM.COM, Inc. All Rights Reserved.

Last Updated August 30th, 1997