Haste makes waste
If you are testing the asynchronous behavior of a system, you often have to wait before asserting a desired condition in your tests. Not waiting for concurrent activity might cause the tests to fail sporadically, reducing their benefit as developers get used to red tests. And thereby, haste once again makes waste.
At CoreMedia concurrency is all around us. Many distributed servers store, transform and index vast amounts of data, processing hundreds or thousands of requests each second. But still, concurrency rears its ugly head most annoyingly in automated UI tests.
That works, at least after several rounds of test failures showed that the wait period has to be extended and extended and extended until it suffices. But that would inappropriate for tests which are supposed to be run early and often.
Then you could poll frequently until the given condition is reached, giving up a after a predefined timeout. That’s nice, but unless you choose the polling period carefully, you might generate so much load through the assertions alone that the system under test does not respond as quickly as it could. Come on, load from checks of UI state? Yes, that has happened. For example, assertions using XPath to check the presence of a DOM element can consume lots of CPU.
For our testing framework Joala we have settled on the exponential backoff algorithm, which has very nice properties in terms of worst-case performance. In essence, you start with a short polling interval and increase the wait period by a small constant factor after each unsuccessful check.
This means that as the waiting time gets longer and longer, the fraction of time spent in tests eventually approaches zero. Because we let the polling interval grow only by 10% each time, we are sure that the total waiting time will never be much longer than the time until the tested condition becomes true.
For tests that succeed after a short time, we ensure that the wait time between two tests is at least as long as the time taken for the test. In turn this makes sure that at most half of the system load originates from the checks and that the system under test is not slowed too much.
With Joala waiting for a condition has become so cheap for the test developer that as a rule we completely forgo immediate asserts and use waiting asserts throughout our tests.