A Better Testing Pyramid

The Automation Triangle (or pyramid) has continually caused a bit of a stir in the testing world. It has been mislabelled many times to be called a “testing” triangle. It has been abused in many companies by being followed as a “test strategy”. It has been butchered in ways that cause further misunderstandings and has caused many a headache to many different people involved in software for years.

Recently, I decided to try and change my stance on the model which was previously to cringe and moan any time I saw the triangle being used in a way that only tells half a story about testing. I decided to intentionally look at the triangle from a perspective of awesomeness.

That didn’t quite work out for me – BUT, I did have an idea on some different ways to draw different testing pyramid models!

 

Here’s one:

Triangle Refactoring 1

From this perspective, the triangle could be useful to see that testing (both investigative testing and assertive testing approaches) are done at all the different layers from the user interface right down to code level.

Lets explore this idea for a second. If you are investigating risks to discover unknown information, you can test at all layers. If you are writing automation scripts to assert any expectations that you have, then you can write the automation at any layer too. The pyramid is still based on value, like Mike Cohn’s original automation pyramid. It’s cheaper and takes less time to test the things at the bottom, then the middle, then the top. So that means we should test for more risks at the bottom if we can, then test at the middle layers if we can, then test through the UI where necessary.

 

Here’s another one:

Triangle Refactoring 2

This triangle represents the amount of tests and test ideas that we can generate across the abstract layers of ignorance. There is information that we know. This is usually very limited compared to the information that we are aware that we don’t know. And beyond that there is even more information that we are unaware of…

Although the triangle isn’t based on value, it can help us to focus on trying to uncover more information from our testing, after all the purpose of testing is to provide information, right?

 

Here’s another one:

Triangle Refactoring 3

With this one, we can think about the different environments and where we should conduct our testing. The majority of the risks that we are investigating should be done on a Dev environment. Ideally while pairing with a developer. This offers the fastest feedback loop and any problems can be resolved there and then when discovered.

A test environment is still useful for testing risks such as integration risks, where you are merging your developed code in with other developed code that might not have been available on your device environment. This is a slower feedback loop, but offers you a safe space to test and build confidence in the quality of the software before you release to prod.

Testing in prod is also an option. Although this offers the slowest possible feedback, and has a higher risk of users becoming pissed off with any bugs you might have released (this risk is especially high if you *only* do testing in production), there are some benefits to testing in the production environment. One common one is if you are releasing to a set of users for them to trial in a beta test/acceptance test sense. Additionally, it’s very hard to mimic the production system with a “like-live” system, so if you need the exact hardware and exact amount of data, then you should investigate whether it’s feasible enough to just use the production environment for testing those risks… But I strongly advise to weigh up the business risks in leaving that testing till the end with the risks of users seeing problems first before you make that decision.

 

Here’s another one:

Triangle Refactoring 4

 

This triangle has a focus on the different approaches to testing.

The idea is that a traditional scripted testing approach (using test cases that list steps and expectations for a person to check) should be reducing with the rise of using an automation approach to be able to automatically assert those expectations through code.

But with us having very limited information regarding expectations compared to the unknowns around all the different product risks, especially in the complex software that we see today, then the majority of our testing will use an exploratory testing approach to investigate these risks. Especially when we think about testing the ideas of new features, and testing the artefacts that we create to describe our expectations, and testing the UX/UI designs, and testing the code design, and testing the code through code reviews, and testing our processes, etc. It becomes clear how much exploratory testing we do (or should do, if you aren’t doing this) all the way through the SDLC.

ALL MODELS ARE FALLIBLE!!!!

So, there are many different ways that we can draw a triangle, or pyramid. Each model has a specific purpose, based on the creator of the model and what they needed the model for at that point in time.

Lumping a different meaning onto a model to try and portray something different to what the model was created for can be dangerous and ignorant. I’d strongly advise just to create a new model. It can be in the same format (as I have demonstrated in my above “triangles”), or it can be in a completely different, unique format – whatever helps you regarding your need for the model.

For example, if you are looking to create a model for a “test strategy”, then check out an example of a model for test strategy that I created previously. Or if you want to create a model for showing where testing fits within Agile, you can look at the Agile Testing Quadrants model for inspiration. (Coincidentally, you will also see on this blog post from Lisa Crispin how easy it is for someone to misunderstand and misuse a model in a wildly dramatically misunderstood way than the original model was intended to be used).

 

Share your models!

I’m keen to see some of your own models. Rather than try and fit other models into your needs (like some people have tried to do by using the Automation Pyramid as a test strategy), have a go at creating your own! Share it with me on twitter or on here in the comments section. Good luck!

Advertisements

7 thoughts on “A Better Testing Pyramid

  1. Hi Dan,

    Nice one. When i see your second picture where you said at the top piece of triangle “knowns to assert”, i still feel we will get unknowns there(at UI level) where even though you have discovered more risks at bottom or middle layer. i think Unknowns will be there at any level. say for example, if you are investigating in integrated environment you still don’t know how your UI is going to behave when integrated to another environments right?

    Liked by 1 person

    1. Hey Srini,

      Thanks for your comment!

      In the second triangle, the top part of the triangle isn’t relating to UI (it sounds like you are interpreting this as an overlay to the first triangle?)… This second triangle is purely relating to knowns, unknowns that we are aware of, and unknowns that we are unaware of. So it’s not an overlay to the initial triangle – it’s a completely separate model.

      Thanks,
      Dan

      Like

  2. I like that you are trying to explore different ways of using the pyramid (or triangle as it should rightly be called). In ‘More Agile Testing’, we showed how different people had adapted it for their purposes, so thanks for pushing the edges even more. Each of these models above made me think a bit differently, and I think I need to sit down and explore them. Perhaps some day when I’m in London 🙂

    Liked by 1 person

  3. I always looked at the test pyramid as a concept. Every client had different failure points. So I had tests at different levels for the different things which could go wrong. I guess I was testing for risk without really thinking about it that way.

    I did always like to use the pyramid concept to prevent bottlenecks I saw many times. What I’m talking about is when the pyramid is inverted and all the testing is done at the end, I inevitably see the QA department ‘stuck’ waiting for development to fix ‘the defect’ but it turns out there are multiple problems resulting from the ONE defect QA has found. Or there are situations where multiple technologies are interacting (with side effects). The end result is the development teams having to work together to figure out what is really going on. In the past (Waterfall days) we would form a Tiger Team to figure out the issue and resolve it. This often resulted in hours if not days.

    So my pyramid required us to try and have one possible failure at each level (sometime more at the unit level). So if an action which added business value could fail for 5 different reasons, I’d try to see how many unit tests could be written to catch as many of those 5 reasons. Maybe we could write 2 uit tests. So now when we get to the next level it might be a contract test which could fail for 3 different reasons. However, because the 2 unit tests passed, we know if the contract test fails then it failed for the 1 remaining reason. At the next level we might have integration tests. There might be 4 reasons an integration test could fail. But 3 of those failures cannot happen because they would have been caught by the unit tests and contract tests. So if the integration test fails we quickly know why it failed. The final level would be UI testing. Because there are 5 reasons the business value could fail and 4 of those reasons are caught at the lower levels on the pyramid, we can know there is only one reason the UI test would fail.

    Now this concept is easier said that done. Additionally, there is always the times when there are actually more possible failure reasons than we had anticipated. This is why exploratory testing is still necessary as well (I draw a little cloud at the top of my pyramid).

    Like

Please leave a comment!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s