Dispelling the misconceptions #5 – “Lets 100% automate everything and get rid of all our testers”

I hear this all the time. Even from very senior people. The latest one was from a development manager, asking why we “still need to have testers testing the software, since we are aiming for 100% automating for everything”…

Automation

Firstly, the truth is very simple: YOU CAN ONLY AUTOMATE WHAT YOU KNOW AND EXPECT.

When we think about the “orders of ignorance”, then it makes it clearer on why it is impossible to automate everything. For those that aren’t familiar with the orders of ignorance, let me explain them…

There are 5 orders of ignorance:

  • The 0th order of ignorance is a lack of ignorance. This is essentially knowledge. Knowledge that you can demonstrate in a tangible form. For example, I am knowledgeable on the fact that it is impossible to automate everything.
  • The 1st order of ignorance is a lack of knowledge. This is where you don’t know something and you are aware that you don’t know that thing. An example of this would be that I don’t know how to use the OWASP Zed Attack Proxy tool to its full potential. I am aware that I don’t know much about this specific tool and I can choose to either learn it or continue using different tools like Fiddler or Burp Suite.
  • The 2nd order of ignorance is a lack of awareness. This is where you don’t know that you don’t know something. So this is where I’m not just ignorant to something, but I’m ignorant to the fact that I’m ignorant regarding that thing. Obviously you can’t give good examples of this since you’re ignorant to it. 🙂
  • The 3rd order of ignorance is a lack of process of discovery. So this is where you are not aware of any way to gain the knowledge that you don’t know that you don’t know something. This means that you can’t learn about the things that you don’t know, you don’t know (i.e. you can’t become aware of the things that are 2nd order ignorance to turn them into 1st order ignorance things).
  • The 4th order of ignorance is a bit cheeky. It’s basically being unaware of the 5 orders of ignorance. 🙂

So, when relating this to Automation… You can only really automate the things that are within the 0th order of ignorance, where you have the knowledge and an expectation of how the feature should work, therefore you script an assert that checks the output and returns a pass or fail as to whether the output meets that initial expectation.

On the flip side, anything that fits within the rest of the orders of ignorance (specifically the 1st and 2nd orders of ignorance), it will be impossible to automate since you don’t have prior knowledge on how the feature or function will work. You don’t have an expectation or aren’t aware of certain scenarios or variables surrounding the feature in order to be able to form an expectation to be able to assert that check.

Lateral Thinking and Exploratory Testing help in turning the “unknowns” into “knowns”. They also help in turning the “unknown unknowns” into “known unknowns”. Although, it’s not possible to do this solely from having conversations up front. Conversations help, but the best way to properly learn about how a system actually works is from learning from the system as we investigate it. So we are continually learning, designing our tests (thinking of our new questions to ask of the software) and doing our testing as we learn more.

A second point I’d like to highlight is that even for the 0th order stuff (the stuff that you do know, not even thinking about the stuff that you don’t know right now), is there really going to be value in striving to automate everything that you have an expectation for? For example, is it worth automating a scenario that we only need to run once?

When thinking about what to automate, we need to think about VALUE… I am a huge advocate of automation. It is very valuable in todays agile environments, with everything moving much faster and fluid, with more agility. It gives us repeatability and extremely fast feedback for our checking tasks, such as regression testing or sanity/smoke testing. It’s also really useful for aiding your sapient testing activities, using it for things like data creation or performing actions to get you deep into the workflow to the part of the system that you need to test… However, we need to be aware of what automation is useful for and what the limitations are so that we don’t fall into this misconception and make some severe mistakes.

So, the next time you hear someone say “Lets automate everything”, then talk to them about why that’s not actually possible and help make the world a better place 🙂

Advertisements

7 thoughts on “Dispelling the misconceptions #5 – “Lets 100% automate everything and get rid of all our testers”

  1. Great post! It is pretty much impossible to bridge the knowledge gap with automation, but it gets far worse than that. You can’t even automate a process to discover what you already expect. You can write code that may or may not represent an explicit, specific, encoded assertion based on one environment, at one time which may or may not correctly report an estimation of what happened. And at the end you still need a person to read it.

    Liked by 1 person

  2. Hi Dan, nice article, I subscribe to your general line of thought with one exception.

    From what you say it looks like the only way a development team can discover what they don’t know is exploratory testing. I agree 100% that exploratory testing is effective for that purpose but it is not the only way.

    I have now worked for many years using specification by example/ATDD and I can tell you that a lot of deliberate discovery can be done before you write a single line of code. Doing specification by example well means also that you test ideas and assumptions using the knowledge of everybody in the team, this allows you to discover some of the things that you don’t know you don’t know. Another way of discovering unknowns before exploring the application that I tried is an approach by which you break down the user stories you need to deliver to a very low level of complexity, it can be difficult at first but once you get good at it, you will see how many things you didn’t know you didn’t know you find out before writing any code!

    Liked by 1 person

  3. I want to have the “5 orders of ignorance” printed on my palm and slap it across the face of anyone who ever mentions the phrase “LETS 100% AUTOMATE EVERYTHING AND GET RID OF ALL OUR TESTERS”. Great post Dan.

    Liked by 1 person

Please leave a comment!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s