Skip to content

Haste makes waste

by on November 29, 2012

If you are testing the asynchronous behavior of a system, you often have to wait before asserting a desired condition in your tests. Not waiting for concurrent activity might cause the tests to fail sporadically, reducing their benefit as developers get used to red tests. And thereby, haste once again makes waste.

At CoreMedia concurrency is all around us. Many distributed servers store, transform and index vast amounts of data, processing hundreds or thousands of requests each second. But still, concurrency rears its ugly head most annoyingly in automated UI tests.

Why is that? In most other cases, the system is specifically designed to hide concurrency from API users. Methods wait just long enough until all caches have been invalidated and the client code sees a clean state. Tests must work very carefully to set up particularly complex concurrency that might, but mostly won’t break the system. But in modern browser-based UIs data is often fetched asynchronously by means of AJAX calls and the UI will not wait until all calls are done, gladly reporting the old state to the test. The test has to care about waiting until the desired result manifests in the UI, either the HTML DOM or the underlying JavaScript model in the browser.The easiest approach to tests in an asynchronous environment is to add a constant wait period before an assertion is made.

That works, at least after several rounds of test failures showed that the wait period has to be extended and extended and extended until it suffices. But that would inappropriate for tests which are supposed to be run early and often.

Then you could poll frequently until the given condition is reached, giving up a after a predefined timeout. That’s nice, but unless you choose the polling period carefully, you might generate so much load through the assertions alone that the system under test does not respond as quickly as it could. Come on, load from checks of UI state? Yes, that has happened. For example, assertions using XPath to check the presence of a DOM element can consume lots of CPU.

For our testing framework Joala we have settled on the exponential backoff algorithm, which has very nice properties in terms of worst-case performance. In essence, you start with a short polling interval and increase the wait period by a small constant factor after each unsuccessful check.

This means that as the waiting time gets longer and longer, the fraction of time spent in tests eventually approaches zero. Because we let the polling interval grow only by 10% each time, we are sure that the total waiting time will never be much longer than the time until the tested condition becomes true.

For tests that succeed after a short time, we ensure that the wait time between two tests is at least as long as the time taken for the test. In turn this makes sure that at most half of the system load originates from the checks and that the system under test is not slowed too much.

With Joala waiting for a condition has become so cheap for the test developer that as a rule we completely forgo immediate asserts and use waiting asserts throughout our tests.

From → Testing

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s