The Calm after the Storm
In “Haste Makes Waste” I described how waiting for a condition can improve the quality of test results for UI tests. The focus was placed on avoiding false positives: tests that detect errors because the UI is still updating and has not reached the desired state.
The case of false negatives is even trickier. For a false negative, a test asserts that all is well when in fact the UI is simply too slow to indicate the given error. As an example, a test might assert that no message box appears after an operation. In this case, simply waiting until no message box is present might be dangerous. The message box might appear any time.
Ok, so in this situation we have to wait again. But how long? A fixed time period? As always, this slows down the tests. We should wait until all current activity has died down and not longer. Often this condition is hard to detect, but we can help the test by augmenting the appplication.
If the application maintains a central event queue and monitors pending request to external services, the test would simply have to wait for the central service to report an idle system and the test can proceed. This is definitely an effort, but design for testability is helpful in other cases and should be relatively cheap if done early.
In fact, when developing a web UI, it is always a good idea to maintain a custom event queue. The function
window.setTimeout() provided by the browser behave subtly different on the various browsers and wrapping it would stabilize an application even in the absence of tests.
We cannot expect third-party libraries to use event queue an remoting abstractions. Patching
window.setTimeout() might be possible, but it might also cause the application to behave differently during test runs. This is why waiting for individual conditions is still a good idea and why we cannot always wait for the calm after the storm. But more often than not, it goes long way towards more stable tests.
So should we always wait for the application to calm down before making an abitrary assertion? Probably not if we intend to make multiple assertions and it is surely more efficient to check for idleness once. Should an idle check be made after each click? At the moment, I think it should, but time will tell whether that, too, is overly costly.