Software Components: When to Test

Applications (i.e. software used by humans) rely on underlying software components like libraries, operating system services, and remote services exposed via an API. Software components may also rely on underlying software sub-components: a Python library relies on other Python libraries, and on the Python interpreter, which in turns relies on C libraries, which rely on the standard C library, which relies on the operating system’s kernel, etc.. The requirements for testing software components depends on whose point of view we take. From the point of view of end-users of higher level applications or components, underlying sub-components are only useful insofar as they support the higher-level functionality. So long as we are testing the higher-level functionality we don’t need to worry about the correctness of the sub-components. From the point of view of the software developers who rely on sub-components to develop applications or higher-level components the correctness and stability of the sub-components are important. From that perspective we may well benefit from testing of the sub-components.

Why software components?

Most software projects, whether applications or software components, start out writing code directly for their users' needs; we’ll call this "business logic". As the business logic becomes more complex programmers will notice certain common functionality and move it into software components. For example, a blogging website might include a Post object which has some Comment objects attached to it, and some logic for rendering those objects to HTML. As the blogging website grows the number of posts and comments may become too large to show all at once. This suggests the need for some paging code, to load only a subset of posts or comments at a time, and the logic is similar for both object types. The result is a shared function or class implementing paging in a generic way that is not tied to either Post or Comment.

A real world example of this process is the Twisted networking framework for Python. It originates in a game engine called Twisted Reality, one of whose sub-components was a networking framework. Other projects began relying on the networking sub-component, so Twisted Reality’s developers split the project in two: Twisted the networking framework, and Reality the game. The networking framework is still in use, the game engine has for the most part been forgotten.

End-user perspective

Imagine you are developing an application or software component: do you need to test underlying software sub-components you develop or use along the way? As a first pass the answer is no.

Functionality tests for the product itself ensure it meets the requirements of users. Given limited resources it is therefore more important than testing of the underlying sub-components. If a sub-component breaks then presumably this will break the functionality tests for the product. Functionality tests for the product can suffice, up to a point, to test that the sub-components are correct.

While not strictly necessary, sub-component testing meets a need of the development team that are not addressed by testing the top-level product: transformational change. At some point the product may change in some fundamental way that makes its existing functionality tests unusable, e.g. a new user interface may invalidate existing tests that relied on the old user interface. Or perhaps you may wish to use the sub-component in a different product. If you have tests for your sub-component that don’t rely on the higher-level product then all is well. Lacking such tests you have no way to guarantee that the sub-component code you’ve written actually behaves as you expect and will continue to do so in isolation from the product where it originated.

Sub-component testing is also useful for debugging and ease of testing in larger scale systems. We have previously explained that functionality tests should aim for realism, testing the whole system end-to-end. Once a software system is complex enough these tests will often become slower, more difficult to debug when they fail. Testing small changes in sub-components will become less easy with end-to-end functionality tests for the whole system. In these case testing the sub-components in isolation is useful as a way of catching and diagnosing problems earlier in the development process.

Direct user perspective

If we focus on the software component itself and ignore the higher-level software that relies on it, a different motivation arises. The software component is a product in its own right, providing some functionality for a group of users: software developers who rely on it. From this perspective everything we’ve learned about functionality testing applies to the tests we write for the component. They help us meet current requirements for the component itself, and ensure incremental changes to those requirements don’t break existing functionality.

Once again what is not handled by these tests is transformational change to the component itself. In order to be realistic the functionality tests for a component must run against its public API. Since transformational change to the component will likely require changes to the API, the tests will no longer be valid and will need to be rewritten or at least updated. If we do need to make significant changes to a component and can’t rely on its existing tests, we can always rely on its sub-components…​ assuming they in turn have tests. Tests for the higher-level software that builds on the component can be useful as well: if an application that relies on an old version of the component continues to pass its tests when ported to a new and different version of the component then that gives some assurance the transformed component is still functioning correctly.

When to test

Software components can benefit from testing under different circumstances:

  • When multiple distinct products rely on the component, in particular when these products live in multiple organizations. We need to make sure a component continues to function correctly for all its users, and once running the tests for every such user becomes unfeasible the component likely needs its own tests.

  • When a sub-component that is currently used in only one product is likely to be used by multiple products in the near future.

  • A large product which is complex or difficult to test will benefit from isolated testing of its sub-components as a way of providing fast feedback on changes.

  • A large product where transformational change is likely will benefit from isolated testing of its sub-components in order to minimize the amount of code invalidated by such a change.