I’ve recently attended a number of conferences, some testing ones, some agile ones and some dev ones too. Although some of the talks were painful due to misunderstandings of one thing or another (be it testing, automation, agile, BDD, TDD, etc…), overall, I thought the conferences were pretty good and I met some new people too and got to share some stories with them.
One thing I heard a fair amount was about DevOps. Many people were talking about it. It’s a big topic! But too many people seemed hugely confused about where testing fits in this wonderful world of DevOps. Some suggested that you only need automation in DevOps, but when asked to explain, their arguments fell floppily by the waist-side. Some people blatently refused to guess at how they’d try and implement testing in DevOps.
And one person even said that no testing was required at all, as he pointed to a DevOps model on a slide, which had no mention of testing on it whatsoever.
Here’s my hand-drawn attempt at replicating that model:
It transpired that this model was presented quite a lot throughout the conferences (or extremely similar ones were presented) and you can see why people struggle to understand where testing fits in a model that doesn’t mention it at all.
For me, testing fits at each and every single point in this model. Let me break this down for you in an updated version of that model to show you what I mean:
We can test the plan. If, in this devops world, plan comes to mean design, then we can test the design. Testing the design of the software is exploratory in nature. It can still be risk based, as we use the risks as heuristics to focus our investigation. Our ideas that we explore through questions will help us incover more information. Information that will help us refactor the design (or plan) to make it better!
We can test a branch (not a physical branch on a tree, although I’m sure that would make an interesting testing challenge for someone…). We can test individual development branches, or we could test our branching strategies. See all the conversations that you have about your branching strategy, and you ask questions and investigate ideas and refactor your strategies… That’s testing.
And of course we can investigate our code! We can also check our code too against any expectations that were discussed as part of our planning! Code reviews – exploratory testing to investigate for mistakes and to see if the code is written is a really great way or to investigate if there is a better way… Unit checks – awesome automated checks at code level to ensure that an expectation is met by the code… And if you are a developer writing some code, then you can even pair up with the tester who will share some really cool test ideas about different risks, different interactions, different perspectives, different variables and different combinations of all these things to help you right there while you code the software! (Yey!! How cool!!)
Merges need to be tested and checked too! Watch out for the hidden risks of different integrations (and different levels of integrations), and also watch out for merge clashes too!
We can certainly test the build too! Why not test your build process and enhance it?
On top of that, Why not spin up an env and do some exploratory testing of the product too? ET is the most effective approach to testing and helps us see a perceived level of quality.
When it comes to the release, we can certainly test that process too. And we can test our deployment (on a test env & on production too). Some people like to do “sanity” or “smoke” testing and checking. Explore your heart out! Just make sure that your testing (or your test data) isn’t disruptive to the users.
And once its live, no doubt that your users will see the shiny fields and buttons and will be clicking away to appease their curiosity of solving the “what if I do this…” mysteries that live inside us all.
And finally, here comes the monitoring… But in no way is it least by being last. Monitoring is powerful in supplying us information on what we tell it to look for. And that’s where we can test it! Are we telling it to look at the right things? What should it be looking for to automatically send us those informative little bundles of information? Lets explore these ideas! And lets keep an eye out for that information and investigate what its telling us too.
But what is missing in the diagram…? I can’t help but feel there is something missing…..
Oh yeah! It’s so obvious now! We can’t start planning unless we have the idea, right??
The model doesn’t mention all that work that happens before that planning bit. Here’s an explosive fluffy-clouded gif to help explain my thinking on this:
Testing the idea is something that some people completely miss the boat with… Just like the model did. Testing the idea is vital; investigating to uncover information, refactoring the idea to enhance and solidify it. It allows us to stem lots of different artifacts and we can test those artifacts them to refactor them too. And we can then use those artifacts to stem all of the activities that we conduct, from; designing and planning, development, testing and checking… And those activities produce lots of useful outputs too – Quality software for one!
So hopefully this blog post has been entertaining and has enabled us to take one step further forward in helping to solve that puzzle of where testing fits within the world of DevOps.
Dan Ashby 
Dan, you might have not really liked all the talks, but God you learned one thing or two very good things judging by this post. This is a precious post, thank you so much for sharing it. Having been in a dev ops context for the last 4 years, I can tell you there are some small minor imprecisions here and there but I will let the nitpickers have a go at them, what I see is a great clarity of what DevOps is, fair play to you!
First time I see one of your church actually getting it :-), it is refreshing, go and lead the way, my friend! You know I am a messer and I am sure you won’t take the church reference as offensive, am I right?
This is so beautiful I feel like from today context people and agile lean people can hold each other’s hand and work together! Thank you Dan!
LikeLiked by 1 person
Gus! I love this comment! Thank you!!! 😄
LikeLiked by 1 person
Excellent post Dan! It’s narrow minded to still think of testing as a phase for any chosen delivery method. I think this post sums that up perfectly!
I had to print screen that animated GIF! It was driving me crazy trying to read it, very nicely done!
LikeLiked by 2 people
I’ve literally just attended a meetup on Microservices and Continuous Delivery and got back home feeling testing is being completely missed out and then I read this! Someone actually asked at the meetup about testing and the answers from the Devs there were immediately about unit and integration tests that was all. Generally the vibe was that testers weren’t needed (though it was never explicitly stated).
Thanks for sharing Dan, I’d like to go back to that Microservices meetup and do a talk about how testers can provide value even when code is being immediately committed and released rapidly.
I fully agree that we test everywhere, all of the time – and I think the biggest reason people don’t see it this way when talking about DevOps is because they only think of automated checks, test cases or bugs when thinking about the value of testing.
LikeLiked by 2 people
Hi Dan,
thanks for sharing your observation. In my observation most people underestimate the force to test their ideas and challenge the assumptions.
Luckily in my recent project in the continuous delivery and DevOps environment were aware the you need to perform testing.
Special thanks for the animated GIF.
LikeLiked by 1 person
Hi Dan,
I never leave comments but this is brilliant.
Here’s me 4 years into DevOps and busy building multiple environments (again) and pondering the whole “from idea to live” issue(s). Yes we can build stuff, automate tests yadda yadda but how does it all stitch together and fit in with a business’ needs?
You’ve only gone and nailed it 🙂
What an excellent start to a dull Monday morning.
LikeLiked by 1 person
This approach is pretty symptomatic of the general ignorance of testing in most places. Test has historically been a 3rd class citizen to other teams (development etc…) and as Continuous Delivery is adopted in more companies, testing will be shown as the main weak spot for most. It’s easy to deliver things quickly, but delivering _quality_ things quickly is more difficult, and that’s where testing at every stage and as much as possible is important. Great article calling this out.
LikeLike
Wow. That is so elegant and logical and clearly explained. Keep it up! I follow up your blog for future post.
LikeLike
Reblogged this on TestOps and commented:
Good blog about Testing in DevOps
LikeLike
Excellent
LikeLike
Dan, great post and blog. Wandering if you mind using me using your second image in this post for a talk on DevOps, of course, I would cite you. I’m trying to convey how important testing is at all stages, and your image speaks volumes. Best, Luis
LikeLike
Thanks for the comment Luis! Yes! Feel free to use it and thanks for citing me.
Thanks for asking too – I appreciate that.
Dan
LikeLiked by 1 person
I see this model referenced quite often. I really think you should highlight that if you don’t know what is testing, this model is not relevant. Agile testing, dev and SDET culture has a very shallow understanding of testing. I know that Dan Ashby has a good understanding of testing. Unless you give that disclaimer, this furthers the misunderstanding of testing.
LikeLiked by 1 person
Really interesting perspective Nilanjan!
Is “not relevant” really the case though? Or can blogs like this one not open the door for people to think “the way testing is being spoken about here is different from my own thinking on what testing is…”, In which case, I’d hope people would ask questions and look to learn more about what I’m talking about 😁
I tend to use this model (and a few others) to help me start conversations with people to talk about what testing is too. I think it’s been helpful from that perspective also.
LikeLike
To follow up on my previous comment, here is an example that is worth including when talking about testing: https://testingindevops.org/anticipating-problems-with-infrastructure-as-code-in-devops/ (this is my blog post).
This is on the ‘ops’ side. I will later post a similar example for the ‘dev’ side.
LikeLike
Great one Devops and testing
There is new things must start have it from QA is [Quality Assistance]
LikeLike