The Big Picture

How should you test your software? There are easy answers: unit tests, or functional tests, or hiring some QA professionals, or even not doing any testing at all. You will hear forcibly argued opinions in favor each of these, and just as forcible arguments against. But while picking one solution and sticking to it is easy, it’s also likely to be the wrong solution for you. Each software project is different; each software project needs, and can afford, different forms of testing.

How then should you decide what and how to test? To do so you need to understand:

  • Whose goals are you addressing?

  • How can testing support these goals?

  • How can testing impede these goals?

Once you can answer these questions for your particular situation you can then proceed to examine the suitability of particular forms of testing.

Whose goals?

The software you produce and the processes revolving around it—​from reporting bugs to deploying new versions—​originate from the goals of three groups.

The organization surrounding the development team determines the high level goals of the product. A company might sell software, or have a free website that makes money from advertising, or create software for marketing purposes. An open source project might aim to build infrastructure to meet the needs of the contributors' own organizations.

Users use the product; they want it to do something. Whether for utility or for entertainment, users require particular features as well as reliability, security, usability.

To create the software product the organization forms a development team; this is where you come in. You and your colleagues design, write, maintain, deploy, upgrade, and distribute the software. We’ll call the combination of all these activities software production. The people and processes involved in software production often have their own internal goals. For example, software developers and designers are often driven by the enjoyment of creating something new. Employees want to keep their job and, if possible, get paid more.

Useful software testing must support the goals of at least one of these groups.

The benefits of testing

Testing can help achieve these goals by providing faster feedback about defects. The earlier you find defects the lower their costs.

There are a variety of software defects:

  • Hard to use user interfaces.

  • Bad business logic.

  • Slowness.

  • Missing features.

  • etc..

Even software that is purportedly defect-free will, with time, grow new defects as its dependencies change (e.g. a new operating system) and user requirements evolve (e.g. a competing product introduces superior functionality).

Defects like a missing or hard to use feature can prevent users from achieving their goals. The economist Albert Hirschman points out that users who are unhappy with a product have two options: voicing their concerns or exiting, i.e. giving up on your product. Outside of academia, it is a rare software project that can meet organizational goals without users. User exit is therefore also an impediment to organizational goals. The alternative of voice can provide useful feedback to the development team, but some forms may also impede organizational goals: defending against lawsuits is expensive.

Another cost of defects is the cost of fixing them. If the development team discovers defects early in the development process they are usually cheap and easy to fix. Fixing defects discovered later in the development process is more expensive, especially if the project already has users:

  • Developers need to figure out what is causing the defect.

  • The changes to fix the defect cannot break other functionality that users rely on. Moreover, the developers need to know they are not breaking other functionality, which can be difficult in itself.

As an extreme example, a well-known software vendor had a three year project attempting to solve scalability issues in their product. The project failed because of architectural choices made years before, choices that could no longer be undone. Many of the developers on the project left to start their own companies inspired by what it could have achieved; implementing these ideas from scratch is much simpler than retrofitting an existing system.

Defects are costly, and testing can reduce these costs:

  • Testing can catch defects in the software before they impact users. By reducing user problems, the organization has less issues with users leaving or complaining.

  • Testing can catch defects earlier in the development process, reducing the cost to the development team (and therefore of the organization) of fixing them.

The problem with testing

While testing has its benefits, it also has its costs. The most obvious cost is time and money: resources that the organization could spend elsewhere. If the development team is spending time on testing, there is less time to develop features that users want. Bad tests can also waste time by increasing the maintenance burden of the development team. The development team may also suffer: testing is less fun than developing new features. Management may penalize developers who spend time on testing, since they are taking longer to produce code than their less careful co-workers.

The problem with testing is that the costs are short-term and immediately visible; the benefits are a longer-term and potentially less visible reduction in costs. Even if the benefits far outweigh the costs, testing can be a hard sell.

Making the case for testing

Whether you’re running the project or just a lowly peon, making a convincing case for testing requires knowing what your options for testing are, both benefits and costs. In the following chapters we will go over the benefits and costs of different forms of testing. In general the earlier you catch a problem the lower the costs of fixing it. We will therefore start with testing you can apply before software development starts and end with the ultimate, unavoidable form of testing: real-world testing by your users.

You will also need to understand your own project, determining whether the costs from defects in your product are something you need to care about. Are you experiencing customer churn, declining sales or rapid growth? If you make a mistake are you likely to get sued, will people get hurt or even die, or will no one be particular upset? Is adding new features to your software easy or difficult? Will you throw the project (or library, or function) away in a month or will you be maintaining it for a decade?

Only when you can compare the costs of defects and the costs of testing will you be able to decide what is best for your particular situation. For example, spending a week on upfront testing before development starts is a fine idea for a project that will take a year, and a bad idea for a project that you need to deliver in a month. A later chapter will discuss this process in more detail.