Adventures in accidental human architecture

Are you vigilantly keeping watch on your system in anticipation of altering some setting or system state? When parts of your system fail, do you perform manual cleanup? If so, humans have been injected into your architecture. Michelle Brush explores the factors that lead to systems that run on human intervention and shares best practices for detecting and removing humans from the machine.

In the 1770s, Wolfgang von Kempelen built an amazing machine: the Mechanical Turk, an automaton that played chess and played it well. He and his chess-playing automaton toured the world and defeated heads of state. However, it was an elaborate hoax. The machine cleverly concealed a human, who actually operated the machine. As software architects, we build complex systems, and we draw boxes and arrows to capture the structure that supports the system. We have high expectations for the performance, scalability, and quality of the software that the boxes represent. But sometimes, accidentally, we build humans into those boxes, and human intervention becomes essential to keeping the system running. These software equivalents to Mechanical Turks reveal themselves in production when engineers find themselves constantly hand-tuning the system or having to initiate common workflows through support tickets. The fragility of such a system becomes obvious when the it fails as soon as the human operators step away. Michelle Brush explores the architectural and organizational anti-patterns that lead to systems that require constant human intervention, discusses heuristics for discovering the humans in the machines in existing systems, and shares guiding principles and practices for removing the people from systems through automation, resiliency, observability, and a little usability.