Your software promises it will do something on behalf of its users, that it will provide certain functionality. A word processor application promises to the peope using it that they can write a letter and then print it. A graphics software component might promise to the software using it that certain API calls will draw certain shapes on a provided buffer. Whether your software is for people or for other software you want to ensure you keep those promises, that the features you provide function now and will continue to function in the future.
The first big decision you need to make when setting up a testing process is determining the environment in which tests will run. Functionality testing requires realism, matching the user environment as closely as possible. Unrealistic environments are less likely to catch bugs that will impact your users. If for example you test your website on a desktop computer with a large monitor but all your users are using a small laptop they might have issues with the user interface that you will never encounter. Unrealistic environments can also cause problems that won’t happen in the real world, wasting your time investigating and fixing irrelevancies. If all your users have large monitors but you test with a small laptop you might spend time fixing user interface issues that won’t impact your users.
In general the more realistic the environment the more costly it is to setup and run, so you may need to reduce realism in order to reduce costs. From most to least realistic potential test environments include:
An environment that is an exact duplicate of the real world environment. If you are going to deploy an application on 10 machines in a particular configuration, you can instead provision 20 machines and setup two environments: the real one and the test one.
A simplified but still complete environment. You can test a web application with only one application server talking to the database, as opposed to the 10 used in the real environment. Unless you’re doing performance testing this simplification may not matter.
An environment where you have removed some of the components or replaced them with fake versions. You can test a mobile application on an emulator running on a computer, rather than on an actual phone. This may suffice to see if your application functions on particular versions of the phone’s operating system.
The second big decision to make is what sort of assertions you want to make, what claims about functionality you wish to validate.
The fundamental divide here is between applications that people will use and software components like libraries or APIs that other software will use.
People can deal with far more ambiguity than software can, which means the assertions for applications can and should be much less specific.
To see why this is the case imagine you are building a web service that sends emails.
Your email sending service has both a user interface for people and an API usable by software.
If the button for sending an email changes its text from "Send Mail" to "Send" most people won’t even notice; they’ll happily keep clicking on the modified button.
In contrast if the API endpoint changes from
/send every piece of software using the API will break.
If you are testing an application you don’t want to make the assertions too specific. When you change a button from "Send Mail" to "Send" you don’t want your test process to consider this a failure; spurious failures waste your time and may hide real failures. If in contrast you are testing a software component you want the assertions to be far more specific since even trivial changes to functionality may break software using the component.
Having chosen and setup a testing environment and determined the level of assertions you want, you can create a testing process that validates your software:
Functions as expected now.
Continues to function as expected over time as your change your software.
What this process can’t help you with is transformational change. If your application’s user interface changes fundamentally the tests you’ve implemented likely won’t apply anymore. Likewise, if you want to take the graphics engine you’ve written for a game and reuse it for another game, the tests for the original game don’t guarantee anything about the graphic engine since they are testing the game, not its underlying software components individually.
Additionally, functionality tests can end up being slow enough to impede your ability to learn quickly from the feedback they give. Once an assertion eventually fails it may also be difficult to tell which underlying component caused the problem.
We’ll discuss how we can deal with these limitations in later chapters. For now let’s move on to focus on application and then software component testing.