There are many misconceptions in the software industry regarding both Manual Testing and Automated Testing. Some people believe that Automated Testing is the bee’s knees and exists as a replacement for Manual Testing (which is simply not true). And others believe that Manual Testing is a simple set of step-by-step tasks that anyone can run through to check an expected output, and that it’s dying out (which is also not true).
In fact, there are many, many more myths and misconceptions about manual testing and automated testing, but the truth is that both are very important and necessary. Both go hand-in-hand and complement each other. In order to produce the highest quality app, you should have a strong manual testing element in place alongside an automated framework.
So let us start by explaining a little bit about Automated Testing…
Automated Testing is a form of testing that utilises scripts to automatically run a set of procedures on the software under test, to check that the steps which are coded in the script work.
For example: if we had a script that logged into the website, then added an item to the basket and placed the order, a basic automated test would check that this path through the system is operational – the “happy path”, (which is an affirmation that the function operates without causing any known validation errors or exceptions and produces an expected output)… It does not check anything that is not written in the script.
The key word here, relating to automation, is “check”. As computers can’t think for themselves, they can only follow a set of commands that we give them, which offer a “yes/no” response. Anything that has a set “expected result” can be classed as a check, where no real sapience is required. (Michael Bolton has a great blog post on “Testing vs Checking” that is well worth a read if you haven’t already).
You simply need to “Observe” > “Compare” > “Report”.
This process occurs with every automated test. “Step 1: Do this – observe – does it work as expected? Yes – inform that this check has passed… Step 2: Do this – observe – does it work as expected? No – inform that this check has failed”.
Image has been adapted from James Bach’s presentation “Testing Without Testing”.
(http://skillsmatter.com/podcast/agile-testing/london-tester-gathering-with-james-bach)
Don’t think of this as being negative though! Automation is great and is essential to include as part of your testing if you want to be Lean and keep a good focus on quality while working in any type of fast paced development environment, such as Agile or RAD.
If you find yourself running through manual test scripts (i.e. for regression or smoke testing), then these are areas that you should be automating to speed up your process and be able to focus your manual testing in other important areas.
Regression testing is where you check that the existing, working parts of the system are still working after a new feature has been implemented. This is to ensure that the change in code, with the new feature being implemented, has not broken any of the existing functionality. As code interacts with other code, it can become very complex and there are many ways in which implementing new code might affect other unchanged parts of the system without us realising. This is why it is important to perform regression testing.
Smoke testing is where you perform a very basic check of the system to touch every part of the system in a cursory way. It is a simple check that is done to discover whether the system is functioning correctly, but not bothering with the finer details within each feature… Wide but shallow.
It is definitely worth automating anything that you have a manual script for, with stepped procedures that have to be run in every release. Automating these scripts helps to reduce the time and effort required to manually check that the existing functionality has not been disturbed.
Manual Testing, on the other hand, is more than just checking. Although “checking” is an important part of manual testing, it is in fact only a small part of manual testing… The most useful and important tool when it comes to manual testing, is your brain. This is something a computer doesn’t have!
There are various things that an automated script cannot do for you. The diagram below (adapted from one of James Bach’s presentations) only shows a subset of factors that are utilised when using a Manual Testing approach!
Image has been adapted from James Bach’s presentation “Testing Without Testing”.
(http://skillsmatter.com/podcast/agile-testing/london-tester-gathering-with-james-bach)
To explain the meaning of a few of these areas (from my own interpretation of them):
- Functional Testing: Automation can’t test what the system CAN and CAN’T do. Automation can only check what you know (and specify) that the system SHOULD and SHOULDN’T do. It cannot deviate from this path.
- Usability Testing: Automation can’t test the layout or “look and feel” of the website. It also can’t test the intuitiveness of the website either.
- Requirements Analysis: It’s not possible for a computer to ask questions of your requirements.
- Tacit Test Procedures: These are the tests that you KNOW you need to perform, but don’t tend to write them down… A few examples might be; “I know that pound symbols can sometimes cause problems in text fields”, or “Invalid dates should cause a suitable error message to be displayed rather than a ‘server error’ message”, or “The Surname Field should allow dashes and single quote symbols as some surnames have these in them”. These tend to be spur-of-the-moment tests that you perform while you are exploring. To add these tests into an automated script has to be planned and maintained.
- Domain Testing (or data testing): This is testing how the system processes the data – looking at both inputs and the outputs, and tracking the data through the entire system.
- Risk Testing: This is identifying and imagining what kind of problems or bugs the function might have and then looking for them. It’s impossible to think of all the scenarios up front, before you are able to see the system…
- Sympathetic Testing: Similar to “happy path” testing, but with an extra element of being even more gentle with the system. The main aim is not to try to find bugs but to build a model of the system, to think about possible sufficient benefits of the software before you start trying to find bugs. Automated scripts can’t be sympathetic.
- Lateral Thinking: A major part of testing software is being able to think laterally about the functional areas of the software being tested, about all of the different ways that the functional areas might be used… Did you know that there are over 50 ways that a plain input field might break? Computers can’t think for themselves.
- Bug Investigation: It is simply not possible for a computer to investigate bugs to find any details around the situations that caused the bug or even the root cause of the bug. They can only provide information that will allow us to manually investigate the problem.
- Perspective/Prospective: Perspective is something that needs to be thought about with every bug found. You have to think about the defect from multiple perspectives to be able to determine the severity of the issue or to determine if it is actually a defect or not. Prospective is another key element in testing. This is where you have an expectation of the system (based on various factors; customer’s needs, customer’s wants, cost, time, quality, value, etc), many of which you might not be able to gather until you have a product in front of you… And this affects your testing as it sets your consistency oracles running.
- Consistency Oracles: This is the niggle in your head that make you think “that’s a bug” or make you recognise a problem when you see it.
- Playing: It’s important to “play” with the system to: explore, learn, question, break and generally get to know the system to be able to test it better.
- Claims Testing: This is where you verify every claim made about the product (from every source available – implicit or explicit). Considering SLAs, advertisements, specifications, help text, manuals, communication with developers and customers, etc.
- Galumphing: The act of randomness. But using true randomness for creating test data and for doing random actions during testing (“true” as opposed to the sub-conscious fake randomness that we all do…).
And these are just a handful of the activities surrounding what’s involved in manual testing…
So to conclude, there will always be a need for manual, sapient testing in the software industry, but being able to utilise automation for the checking activities is highly beneficial. In order to have an effective process that focuses on building quality products in a fast paced environment, both testing methods are important for being able to achieve this! You need to utilise the benefits of having automation, in the fact that it will free up time and effort that a tester would normally have to spend performing the “checking” tasks, and allows the testers to focus more on the sapient testing that is required to be done in order to be able to discover an accurate level of quality that the software bestows.
Markus Gartner (the author of the “ATDD by Example” book) summed it up nicely when he said: “While automated tests focus on codifying knowledge we have today, exploratory testing helps us discover and understand stuff we might need tomorrow”. That statement sums it up very nicely indeed!
Dan Ashby 
In my opinion…
Automation cannot test, so referring to something as “automated testing” is unhelpful, and exactly why we make the distinction between testing and checking.
Automation can be used for far more purposes than for checking, so referring to automated testing as “a form of testing that utilises scripts to automatically run a set of procedures on the software under test, to check that the steps which are coded in the script work” is pretty unhelpful too.
No oracle, “manual” or “automated” can show that something works or that something is functioning correctly. An oracle can help us to recognize problems, but no number of passing tests can show correctness. (Ask a turkey before Thanksgiving if the farmer is a kind, considerate, caring man, with the best interests of turkeys at heart.)
“Manual” testing may or may not (but usually does) use machinery and software tools, so referring to it as “manual” is pretty unhelpful too.
It’s also unhelpful to consider “automated testing” as something that is somehow not embedded in “manual testing”. If checking is to have any value, it must be preceded by testing and followed by it too.
We have a way to get around all this: we can give “manual testing” and “automated testing” their retirement dinners, and start referring to your long list of activities as testing, and to the checking stuff as checking. When we start seeing testing as something that can only be done by humans, we can get rid of “manual, sapient” in your second-last paragraph above.
—Michael B.
LikeLike
Great points Michael! Thanks!
I think it’d be great if we could all just stop using that terminology, but I don’t think that would necessarily stop people that still use that terminology from having their misconceptions…
For me, I think it’s primarily more important to dispel the misconceptions around the meaning of the terminology, rather than the terminology itself…
Additionally, in terms of my blog, I’ve intentionally used that terminology in my heading and in the blog tags, as I know that
anyone looking to learn about automation will turn to google and type “automated testing” more than they would type “automated checking”… So it has been a bit tactical on that side… the more people I can get to read that scripts ran by a computer (regardless of terminology) can only “check”, the better!
“Automation can be used for far more purposes than for checking” – I totally agree with you here. I mentioned in a previous blog about automation being used by testers as throwaway scripts from a tester’s perspective.
I was aiming this blog from the “framework perspective” side of the coin though… I know people who actually believed that having an automation framework replaces the need for testing!
I also agree with what you say about the word “Manual” too, so started calling it “Sapient” testing. But I got frustrated in having to explain what “sapient” meant to everyone… I’m still struggling to find a better word when talking about “Testing” in the same sentence as talking about “Automation”… I’ve tried using this exact terminology and people can get confused due to them knowing “Automation” as “Automated Testing” right now (although I am dispelling their beliefs around what automation is and isn’t, which is good)… Any ideas around this?
Re: “no number of passing tests can show correctness” – I recently watched a presentation on youtube from an academic in America about Critical Thinking (cant remember who the speaker’s name was), and this point was drilled across when she was talking about asking questions and hunting for answers. She said that the only way that you can know if any answer is correct is by having some “pre-information” regarding the question and also by knowing the expectations, perspective, context and meaning/reasoning behind the question. I can completely relate this to software testing and it relates to your statement too!
LikeLike
Very informative post and it was quite helpful to me. I also wrote something similar lines on myths about test automation – http://bit.ly/1nhPKmc
LikeLike
“For me, I think it’s primarily more important to dispel the misconceptions around the meaning of the terminology, rather than the terminology itself.”
Yes, it is. But if you’re serious about that, you can’t ignore the connections between words and ideas. When I find people getting stuck because of the words they’re using, I can use the tactic of changing the words to help get them unstuck,
“I’ve tried using this exact terminology and people can get confused due to them knowing “Automation” as “Automated Testing” right now (although I am dispelling their beliefs around what automation is and isn’t, which is good)… Any ideas around this?”
Which exact terminology? One hint is to try using smaller, clearer words. Instead of saying “automation”, say tools. Instead of “terminology”, try “words”, or “names” or “labels”. Instead of saying “test automation”, try “testing tools” (when you mean that) or “machine checking” (when you mean that). I suspect that if people called them “checking tools” instead of “automation frameworks”, we could lose a bunch of pretentiousness and gain some clarity.
With regard to “sapience”, is it possible you’ve missed reading this? http://www.satisfice.com/blog/archives/856
LikeLike
Something you’ve not really mentioned here is the automated testing (or rather, checking) of system integrations with other systems, i.e. verification of systems adhering to contracts at the API level.
Although clearly this is not the same overall discussion as probing a user interface, it’s quite interesting and I don’t see it discussed a lot. From my interpretation of discussions in this area, it’s an emerging industry and is almost completely automated.
I’d be interested to hear your take on whether manual / human testing can be applied so readily to this area as it can be to systems that have direct user interaction.
LikeLike
Hi Dana,
I’m glad that you are trying to grapple with this material. But I’m concerned that you have edited one of my slides in such a way as to introduce a bug. We don’t “check that a product works.” We can’t do that. We can check specific facts about it.
For instance, you can check that a murder suspect is not currently murdering someone as he is sitting in jail, but you cannot conclude from any amount of such checking that he “works” in the sense of not being liable to murder anyone. Checking is useful, but it doesn’t magically teleport you to knowledge that “the product works.”
— james
LikeLike
Hi James! Thanks for your comment!!
You make a great point… I’ll update the image to reflect that – by “works” I really meant “conforms to it’s expectation at that particular point in time”.
I have a question about your analogy with the murder suspect though. If the police regularly check up on a suspect, but each time they check they found that the suspect didn’t commit any crime, then this would potentially give them a certain level of confidence that the suspect wasn’t a criminal (at least over time).
In this instance, can the police not say that since the suspect hasn’t committed any crime, then he/she is not a criminal (at least up until the point where the suspect does eventually commit a crime, if that was to happen)?
I guess what I’m trying to say is that it’s understandable that things change. What once was functioning software can break in the future with newly introduced code. But right now, for testing the previously released function, we do seem to rely on “checks” to allow us to remain confident that the previous functionality is “still meeting that existing expectation at the point of checking it again”. But taking into account the fact that we can’t ‘check’ that the function works, is there a more effective (yet still time effective) alternative for performing such checking tasks?
Thanks,
Dan
LikeLike
“by “works” I really meant “conforms to it’s expectation at that particular point in time”.
You mean “apparently conforms to some expectation out of a universe of possible expectations and that we have recognized in advance of the observation at that particular point in time and not necessarily at any other point in time.”
each time they check they found that the suspect didn’t commit any crime, then this would potentially give them a certain level of confidence that the suspect wasn’t a criminal.
Yes; this is precisely why con men are called that; “confidence men”. They take advantage of people’s (misplaced) confidence. The police cannot reasonably say that the person has not committed any crime; only that they are not aware of the person having committed a crime. This is why police have investigators and detectives, not just clerks. Without evidence, it is foolish to make the determination that a person either is or is not a criminal.
But taking into account the fact that we can’t ‘check’ that the function works, is there a more effective (yet still time effective) alternative for performing such checking tasks?
There is a wider practice that we call “testing” that covers that nicely.
—Michael B.
LikeLiked by 1 person
Hi Michael,
Thanks for the response and for your insight.
I find it really interesting what you’re saying around regression testing. Its something that I’ve been thinking about recently when I heard someone say that they should *re-run* an ET session for regression testing purposes… I questioned why they wouldn’t run a *new* ET session for regression. It turned out that their emphasis for regression was more about attempting to run repeatable tests and checks rather than focusing on the risks surrounding their software regressing.
I think automation can be very valuable for having the repeatability that it brings, but for me the purpose is to investigate the risks of the software regressing. I think automation checks play a big part in this, but I do see the value in exploring alongside those automated checks though to discover new risks.
LikeLike
Hello Dan, you make a good point that automation should be used for checking. It does provide a good help to speed up with the checking of things like validations and for the rest it is good practice to imply manual testing. There’s some many people who think that automation “testing” is always better than manual and soon it will eat up manual testing completely. I tried to put up my thoughts on this here http://milinpatel17.wordpress.com/2014/07/life-is-too-short-for-manual-testing-or-is-it
LikeLike
Good explanation on Manual Testing. Keep posting 🙂
LikeLike
Excellent explanation
LikeLike
HI Dan,
Good Thought, You mentioned by automation we can’t test UI / look and feel of application, it could be done by Selenium RC, Visual Studio, and Telerik’s Test Studio. But the complication is we need to set the coordinations, this could also be overcome in future soon, like wise many can’t will going to be can in automation.
R.Karthik
Test Automation
LikeLike
Hi Karthik,
Thanks for the comment.
If we know what to expect of these things, then I guess you are right in saying that we /can/ automate them. Thats key though… you cant know everything up front about how something looks and feels.
There is also the big question surrounding VALUE… Is it valuable in your context to attempt to automate such checks? The answer completely depends on each of our own situations.
LikeLike
Tools can perform checks. Only humans can test (which might include checking). If I were to reply to Kathik’s issue, I would ask “what things about *look* or *feel* can be checked? What things require human evaluation and learning?
LikeLike
Its a good question. I like to ask the question “what can’t we automate”…
I did make a big assumption when I responded, as I based my response on my knowledge of the tools. I assumed that he was talking about checking pixels and colours, which I know you can do with these tools. They can’t check the *feel* of the system though.
You are right with the fact that tools can perform checks, but we need to know what we expect in the first place to be able to have the tool check it, which is the point I was trying to make. And then there is the question of there being value in checking this information.
Here’s a question – can we even test the *feel* of the software?
I mean, yes we can test it based on our own perspective and the expectations that we learn about from speaking to the customers and users and from other sources. But there are always multiple users and customers, who will have different expectations, preferences and feelings about the software… so what does that mean/how does that affect that aspect of the *feel* of the software?
LikeLike
Very informative post about manual and automated testing.
automation testing
LikeLike
This was loveely to read
LikeLike