Dealing with Broken Integration Tests
Again and again I hear in daily stand-ups (see: Scrum):
Integration Test XYZ is broken since some days. Actually I can’t reproduce the problem manually. What should we do about it?
Of course this might also occur with unit-tests… but most of the time they are much easier to debug and have higher priority as they break the whole build process.
A typical decision is to ignore the test for now and to test it manually on every release. But this raises the risk of the Broken Window Effect:
Once you get used to the red state in your integration tests you won’t even recognize new failed tests.
Some teams decide to keep deltas of failed tests and so one team member gets the task to validate the delta every day to see if we got worse. I don’t need to mention, that this costs much more than just to see if something is green or red, do I?
Other teams decide to add @Ignore to their tests – and typically they find the tests two years later completely broken and no one understands what their actual intent has been. So they get deleted (after another two years).
Recommendation: Assume and Assert
- Create an issue in your bug-tracking system, say BIT-1234
- Replace the failing assertion in your test with an assumption.
- Directly after the assumption add a statement like
fail(“BIT-1234 seems to be fixed”).
So what do we have now?
- The assumption fails and will mark the test as skipped. Thus your tests are green (or in some cases yellow) again.
- If someone fixes the bug (perhaps not even knowing that the fix is related to the former failure) the test will fail – as it assumed to be broken.
- Now you are happy and can just replace the assumption by an assertion again and remove the fail-statement.