Skip to content

Dealing with Broken Integration Tests

by on May 11, 2012
Broken Eggs

Again and again I hear in daily stand-ups (see: Scrum):

Integration Test XYZ is broken since some days. Actually I can’t reproduce the problem manually. What should we do about it?

Of course this might also occur with unit-tests… but most of the time they are much easier to debug and have higher priority as they break the whole build process.

A typical decision is to ignore the test for now and to test it manually on every release. But this raises the risk of the Broken Window Effect:

Once you get used to the red state in your integration tests you won’t even recognize new failed tests.

Some teams decide to keep deltas of failed tests and so one team member gets the task to validate the delta every day to see if we got worse. I don’t need to mention, that this costs much more than just to see if something is green or red, do I?

Other teams decide to add @Ignore to their tests – and typically they find the tests two years later completely broken and no one understands what their actual intent has been. So they get deleted (after another two years).

Recommendation: Assume and Assert

My recommendation is to use JUnit’s pattern of Assume and Assert to get rid of the Broken Window:

  1. Create an issue in your bug-tracking system, say BIT-1234
  2. Replace the failing assertion in your test with an assumption.
  3. Directly after the assumption add a statement like fail(“BIT-1234 seems to be fixed”).

So what do we have now?

  • The assumption fails and will mark the test as skipped. Thus your tests are green (or in some cases yellow) again.
  • If someone fixes the bug (perhaps not even knowing that the fix is related to the former failure) the test will fail – as it assumed to be broken.
  • Now you are happy and can just replace the assumption by an assertion again and remove the fail-statement.

From → Dev

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s