Skip to content

Meetup Softwerkskammer HH: Collected Thoughts on Testing Pyramid

by on March 20, 2013

Yesterday the Softwerkskammer Hamburg (Xing, Meetup), a group dedicated to Software Craftsmanship, had its March-Meetup. I joined a session titled “Test Pyramid” and we had an interesting discussion on the value of different levels of testing. You will find some thoughts collected from that session in this post.

Test Pyramid

The test pyramid got introduced by Mike Cohn (see for example The Forgotten Layer of the Test Automation Pyramid). It basically states that you have different levels of automated test approaches (Unit, Integration, UI) with increasing costs regarding implementation and maintenance of tests. Thus a UI test is hard to write and very often brittle – not necessarily because the UI changes (it is possible to write UI tests quite robust to UI changes) but because of dealing with asynchronous events and infrastructure problems such as UI machines having a resolution which is too low for testing.

Thus the Test Pyramid favors to have a broad set of unit tests: they are fast, easy to write and – that’s the assumption – easy to maintain.

Unit Tests – the Silver Bullet?

This is where the discussion actually started. Are Unit Tests really the Silver Bullet to develop well crafted software? Experiences reported:

  • I had high coverage – but when I refactored it resulted in high costs in also adopting the tests.
  • I spent hours on mocking.

Actually what I once learned about testing:

A test is always also a specification.

What’s the point of this? You should be aware that if you write a test you also fixate the code under test. Thus if you test with high code coverage you actually fixate the actual implementation thus as a loop within your code, or ensuring that you visit every branch of an if-statement.

Is this really what you want to do? The advantage is that you can easily add a measurement to those tests (Code Coverage) so you might get a feeling when your tests are sufficient. The disadvantage: You become less agile.

Lowering TCO of Tests

The main discussion was about how to lower the Total Cost of Ownership (TCO) of tests. The experiences reported in the session has shown one observed effect:

While Unit Tests are most likely to break on refactoring, acceptance tests won’t break as easy.

This is actually what Jakub Holy mentioned in his post Principles for Creating Maintainable and Evolvable Tests (I really recommend reading it!). He focuses on “tests that tell a story" – so actually acceptance tests. The (well-written) story will survive longer than the implementation details.

Acceptance Tests with plain JUnit and Gherkin

One approach I reported was using plain Java tooling to write tests as living documentation, tests which tell a story. While Cucumber and JBehave rely on parsed text documents we started to write the stories and scenarios with plain Java classes. The advantage: We have high IDE support on refactoring and good reporting on for example unused step definitions.

A roughly sketched example would look like this:

class StoryWritingMaintainableTestsTest {
  void scenario_refactor_test_after_many_years() {

The actual challenge is writing the story so that it describes a valuable story. A step which most of the time has no value for a story is when_I_click_button_B(). What if clicking a button is later replaced by speech recognition?


Actually I again would like to quote the Conclusion by Jakub Holy:

Given that most of bigger business software systems live for quite a number of years, it’s essential to write tests in such a way that they enhance and not limit the evolvability of the system. It is not easy but we must make efforts towards that.

To succeed we must structure our code and tests in a particular way and approach our test methods, test classes, and test suites as telling stories about what the code under test does.


From → Dev, Testing

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s