I hear this all the time. Even from very senior people. The latest one was from a development manager, asking why we “still need to have testers testing the software, since we are aiming for 100% automating for everything”…
Firstly, the truth is very simple: YOU CAN ONLY AUTOMATE WHAT YOU KNOW AND EXPECT.
When we think about the “orders of ignorance”, then it makes it clearer on why it is impossible to automate everything. For those that aren’t familiar with the orders of ignorance, let me explain them…
There are 5 orders of ignorance:
- The 0th order of ignorance is a lack of ignorance. This is essentially knowledge. Knowledge that you can demonstrate in a tangible form. For example, I am knowledgeable on the fact that it is impossible to automate everything.
- The 1st order of ignorance is a lack of knowledge. This is where you don’t know something and you are aware that you don’t know that thing. An example of this would be that I don’t know how to use the OWASP Zed Attack Proxy tool to its full potential. I am aware that I don’t know much about this specific tool and I can choose to either learn it or continue using different tools like Fiddler or Burp Suite.
- The 2nd order of ignorance is a lack of awareness. This is where you don’t know that you don’t know something. So this is where I’m not just ignorant to something, but I’m ignorant to the fact that I’m ignorant regarding that thing. Obviously you can’t give good examples of this since you’re ignorant to it. 🙂
- The 3rd order of ignorance is a lack of process of discovery. So this is where you are not aware of any way to gain the knowledge that you don’t know that you don’t know something. This means that you can’t learn about the things that you don’t know, you don’t know (i.e. you can’t become aware of the things that are 2nd order ignorance to turn them into 1st order ignorance things).
- The 4th order of ignorance is a bit cheeky. It’s basically being unaware of the 5 orders of ignorance. 🙂
So, when relating this to Automation… You can only really automate the things that are within the 0th order of ignorance, where you have the knowledge and an expectation of how the feature should work, therefore you script an assert that checks the output and returns a pass or fail as to whether the output meets that initial expectation.
On the flip side, anything that fits within the rest of the orders of ignorance (specifically the 1st and 2nd orders of ignorance), it will be impossible to automate since you don’t have prior knowledge on how the feature or function will work. You don’t have an expectation or aren’t aware of certain scenarios or variables surrounding the feature in order to be able to form an expectation to be able to assert that check.
Lateral Thinking and Exploratory Testing help in turning the “unknowns” into “knowns”. They also help in turning the “unknown unknowns” into “known unknowns”. Although, it’s not possible to do this solely from having conversations up front. Conversations help, but the best way to properly learn about how a system actually works is from learning from the system as we investigate it. So we are continually learning, designing our tests (thinking of our new questions to ask of the software) and doing our testing as we learn more.
A second point I’d like to highlight is that even for the 0th order stuff (the stuff that you do know, not even thinking about the stuff that you don’t know right now), is there really going to be value in striving to automate everything that you have an expectation for? For example, is it worth automating a scenario that we only need to run once?
When thinking about what to automate, we need to think about VALUE… I am a huge advocate of automation. It is very valuable in todays agile environments, with everything moving much faster and fluid, with more agility. It gives us repeatability and extremely fast feedback for our checking tasks, such as regression testing or sanity/smoke testing. It’s also really useful for aiding your sapient testing activities, using it for things like data creation or performing actions to get you deep into the workflow to the part of the system that you need to test… However, we need to be aware of what automation is useful for and what the limitations are so that we don’t fall into this misconception and make some severe mistakes.
So, the next time you hear someone say “Lets automate everything”, then talk to them about why that’s not actually possible and help make the world a better place 🙂