Multilayered testing

Automated testing is at the heart of modern development and operations, but it's often segregated between developer-focused unit testing and separate, often semi-automated integration testing. Alex Martelli explains that the best kind of testing deploys in many layers, reusing test-components in various configurations for multiple purposes.

The role of automated testing at the heart of modern development and operations is accepted as a given. However, the traditional approach to testing, which cleaves it drastically into two—developer-focused unit testing and separate (and, alas, often only semi-automated) integration testing—is not optimal in the modern, fluid world of DevOps. Nothing short of full automation is suitable for continuous integration; any “testing” requiring humans has a drastically different place in the continuum of development and deployment and should best be called by a completely different name like quality assurance. Within the realm of fully automated testing, the best approach, just as for other kinds of software, is highly modular. Alex Martelli highlights the proper design of components meant for testing purposes and explains how such a design lets you compose multiple, layered testing suites that span the gamut from unit tests meant to run all the time during development to full-fledged end-to-end tests of whole systems—and, crucially, the often-neglected intermediate layers, bridging the thoroughness of end-to-end tests with unit tests’ ability to pinpoint the location of any problems that emerge, enabling rapid fixes of most such problems. Alex also discusses the use of modular, layered testing components to validate software refactoring (all the way to deep rearchitecting of a system) and (when deployed in a load-testing arrangement) identify and validate software (and architectural) optimizations.