December 22, 2019

377 words 2 mins read

Need for speed: Accelerate automation tests from three hours to three minutes

Need for speed: Accelerate automation tests from three hours to three minutes

The elephant in the room that every developer ignores is that all automated tests (except unit) are too slow for todays fast-paced, first-to-market environment. With slow automated tests, youre just shipping problems faster to production. Emanuil Slavov explains how the team at Komfo made automated tests run 60x faster using containers, hermetic servers, and improved architecture.

Talk Title Need for speed: Accelerate automation tests from three hours to three minutes
Conference Velocity
Conf Tag Build resilient systems at scale
Location Amsterdam, The Netherlands
Date November 7-9, 2016
URL Talk Page
Slides Talk Slides

All automated tests (other than unit) are too slow and unreliable for the fast development pace every company craves, and there are many external dependencies and factors outside of your control, so when these tests fail, it’s hard to pinpoint the exact reason why. Komfo had automated tests running for more than three hours every night. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable due to a feedback loop. At one point, the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. The Komfo team decided to put an end to this madness—after considerable effort and dedication, the same tests currently run for only three minutes. Emanuil Slavov tells the story of how Komfo achieved 60x faster tests. Along the way, Emanuil touches on topics such as test automation framework design, hermetic servers, optimizing test environments for speed, architecture for testability, DevOps collaboration, virtual containers, reliable tests when depending on services outside of your control, and the joys and pitfalls of parallel execution. Having automated tests run that fast completely changes the software development game. There is no need to wait for the nightly tests to complete, and there are no more smoke tests. You can run all tests, get fast feedback, and release instantly without falling behind. In the near future, this will be standard practice, much like infrastructure as code or continuous integration are today. In order for a company to stay competitive, all existing automated tests—static code analysis, unit tests, API, and UI—should complete in less than five minutes after every code, database, or infrastructure change.

comments powered by Disqus