So… Another blog post stemmed from the depths of the LinkedIn forums. This time someone was asking for advice about answering an ISTQB exam question.
The question is as follows (with this exact wording):
If you are flying with an economy ticket, there is a possibility that you may get upgraded to business class, especially if you hold a gold card in the airline’s frequent flier program. If you don’t hold a gold card, there is a possibility that you will get ‘bumped’ off the flight if it is full and you check in late.
Three tests have been run:
Test 1: Gold card holder who gets upgraded to business class
Test 2: Non-gold card holder who stays in economy
Test 3: A person who is bumped from the flightWhat is the statement coverage of these three tests?
A. 60%
B. 70%
C. 80%
D. 90%
There are so many problems with this question.
Lets start with the first problem regarding terminology. In a software development world, where communication is vital, we need to use terminology that people understand. And if we are creating documentation, we should simplify our terminology as much as possible to avoid confusion and misunderstanding from others reading the documentation. So I ask, what do you think when you read the words “statement coverage”? What does it mean? Especially in the context of this ISTQB question. Even when it appears that the question previously used the term “tests” when talking about them being run (but in fact, they would actually be “checks”, right?). Do statements relate to “tests” (/”checks”)?
Either way. I’m confused by this terminology. I’m guessing that its asking for the level of coverage for the 3 checks it’s supplying regarding the feature or system it’s describing.
This brings me to my second problem regarding the context. We’re asked to read an initial paragraph of information. From this we can gather that its regarding an airline’s frequent flier program and the possibility of getting upgraded or “bumped” off a flight based on the frequent flier program and gold card membership and whether the flight is full or not. The name of the airline is not supplied.
My problem is that there is a major risk of falling into the “assumption trap” regarding only the explicit information.
But before I go into more detail on these assumptions, let me mention my third problem with the question as it ties in with it: The supplied A, B, C and D answers are all wrong.
If we look at the explicit information, we can see that we have a few variables:
- The ticket type
- The upgrade option
- The airline membership card
- The flight being overbooked
When we look at only the explicit information about these variables that is supplied in the paragraph, we might think that there are only 2 types of tickets (Economy and Business). But in reality, anyone who has flown with any major airline, might know that there are actually other ticket types available. For example, Premium Economy and First Class tickets can be purchased.
Now, taking that extra information into account, we have explicit information from the paragraph about upgrading to business class, but with there being other class types, what about upgrading to Premium Economy or First Class too from Economy? And of course the extra scenarios around upgrading from Premium Economy to either Business Class or First Class and even upgrading from Business Class to First Class.
And what about airline membership cards? Well, I’ve been a member of a few airlines collecting air miles for some time, and I know that there are various levels of membership cards: Blue, Silver, Gold, Platinum, etc. Most major airlines offer at least 3 levels of membership, not just a Gold Card. And then of course there is also that explicit information of the possibility of not having a membership card.
And we have the 4th variable of the flight being overbooked. But really, are we assuming that this means the full flight is overbooked, or does it mean just the seats for the class that your ticket is for?
I once flew in economy, but the economy section was overbooked, so they moved some people up to Premium Economy free of charge rather than “bumping” them off the flight completely.
So lets collate all of this information together…
You can see from the mind-map (clicking on it should expand it to make it readable), that there are actually around 92 different scenarios if we take into account this extra information within the variables that are sourced from our real life experiences with flights and airline memberships. This means that the 3 “checks” mentioned would actually equate to about 2.76% coverage. (Of course, the caveat is that this is based only on the scenarios that I could think of… so far at this point in time… based on my own experiences and beliefs. There might be more information that I don’t have).
The thing is… If we were treating the ISTQB’s system as if it was our own real system that we were working on, then one of the most important things we need to do is to use our thinking skills to think about as many variables and questions about those variables in order to stem more information on the “what”, “why”, “who”, “where”, “when” and “how” of the feature or system that we are being asked to develop and test.
We need to dispel the assumptions of the requirement, and not just fall into the trap of relying solely on the explicit information from the single paragraph that we receive. We need to use multiple resources, such as our experiences and perhaps sales and marketing material or even competitor products, etc.
If we jump on only the explicit information supplied in the 2 sentences, like the ISTQB question supplies, then you’ll fall into that “assumption trap”.
And this brings me to my final problem with the question. Testing is about investigating ands questioning. When we have a requirement, we need to question the assumptions and investigate the variables and question them some more in order to refine the requirement so that we can start building and testing the “right thing”. When we have built a product based on that refined, tested requirement, we question and investigate it through operation and observation so that we can test (and check) that the “right thing” is “right” (if that makes any sense 🙂
ISTQB are completely missing this in their exam question. Instead they are focussing on “Statement coverage” and their answers rely on you NOT questioning the requirement in the initial paragraph.
Have you seen any similarly appalling ISTQB questions?
How would you have answered this one?
Dan Ashby 
Dan, I saw this question floating around in some LinkedIn group a while ago. It was posed in the same format you use here and my answer was “you cannot determine the statement coverage in absence of either a diagram flow or the actual code”. Other people insisted on other answers and that made me research the question on the internet and I found out that the version you report is incomplete, in fact the question also includes a flow diagram (see http://www.softwaretestingclub.com/forum/topics/i-have-a-doubt-for-an-istqb-ctfl-related-question?id=751045%3ATopic%3A220753&page=2#comments)
Believe me, I am far from defending ISTQB questions and ISTQB itself, but I thought I should point this out.
I believe that most of the points you make in your post still apply
LikeLiked by 1 person
Hi Gus! Thanks for the comment and the link to the fuller question including the state diagram.
Yes, I’d say that the same points still apply. We could draw a bigger state diagram based off of the extra information gathered from my real life experiences and other sources of information to include the extra scenarios around it
My main concern about the question, even with the state diagram, is that it’s requiring you not to think of the variables and build up the information. In fact, it’s relying on you not doing that in order to pass the question, which would actually hinder your testing if it was real life. ISTQB should be promoting asking questions and building information, not hindering it.
LikeLike
I agree Dan. The question is based on an old fashioned way of testing, that is testing the code but not the assumptions.
LikeLiked by 1 person
You’ve hit the nail on the head there Gus.
LikeLike
Interesting post Dan, and great ideas on dealing with assumptions.
I think we all agree that the ISTQB view of the world seems a bit simplistic at times. Declaring somebody to be a “Certified Tester” based on it, and filtering job applicants on whether they have a cert or not is something that I agree should be discouraged.
However; I feel you are slightly overthinking this one (but I might be wrong). The way I see it this (simple and limited) question/scenario is is designed to determine whether the student understands what the ISTQB means when talking about Statement Coverage… in the form of a multiple choice question. It is not a question about how you would suggest to think about testing such a system if hired to do so… to be answered in essay form.
While you’re answer would be very interesting when getting to know you and learning about your approach as a tester, it would be tediously long, and more than a little bit off topic if I just wanted to know if you agreed with my definition of Statement Coverage.
Back when I did the BBST foundations I fell into the same trap (IMHO) when we got a question about testing fonts. I had recently attended RST with James and he asked us about some regression testing of fonts in Notepad (or something like that). In that case the answer was that Microsoft already has a huge team working with fonts, so the only thing we were really expected to do was to se that Notepad wasn’t obviously broken and that you could select different fonts.
So back in BBST I felt really clever and went off on a tangent about how maybe other teams was doing this work already and bla bla bla… which might be relevant in a “real world” scenario, like your first class tickets and membership cards.
But that was not the question!
The question was how *I* would test this, in the limited and perhaps simplistic scenario given in that specific question.
So, in conclusion, or TL;DR (which probably should have been at the beginning of my comment 🙂
I think the question is not about good testing in general, but about the definition of Statement Coverage. In this context the simplistic scenario might be good enough.
LikeLiked by 1 person
Thanks for the comment Geir!
I completely agree. I initially was looking at it from the perspective of it testing the student’s knowledge on Statement Coverage, but the more I looked at it, the more I thought it was hindering how I would approach testing this feature if it was right in front of me.
And that’s why I wrote the post. A company that is teaching about testing should be teaching how to test rather than focussing on checking that students understand terminology.
But I can see how it might read like a bit too much of a rant… 🙂
LikeLiked by 1 person
Nothing wrong with a good rant 🙂
It even made me think about it again (I remember seeing the forum post).
Thanks
LikeLike
The fact that we don’t really know what the question is *for* does not speak well for the quality of the question. The question doesn’t actually mention any software whatsoever, just that “tests have been run”. Not by whom, not on what, just that some testing apparently happened to something. There may, in fact, not be a single line of code involved. What percentage coverage would that imply? What does that even mean?
Here’s an interesting question to ponder with regards to this: What answer could be given to this question that would prove any depth of understanding about the ISTQB definition of statement coverage?
Here’s an interesting follow up: And why is that valuable to anyone?
I put it to you that not only is this question useless for pragmatic, contextual testing (not forgetting that the context **defines** the value of the testing), but it is useless to establish any understanding of the definition of statement coverage. I don’t think this question is effective to establish anything in particular, as it’s poorly worded and nonsensical. If there was an intent I think it’s lost to the ages. Whatever we bend our minds to imply about this the actual wording of the actual question is, in my opinion, utter, utter nonsense.
Out of interest, what do we all think the ISTQB has on their answer sheet to this question?
LikeLiked by 1 person
Guess what Dan, you failed! I’m sorry to say that this means you must be a crap tester, if you can even call yourself a tester. Pff, shame on you.
Your answer, while in-depth, informative, thoughtful, etc. is not any of the supplied answers (A, B, C or D) and therefore must be wrong. Oh, and that picture in included (I think it’s a mind map?) just confuses the hell out of me.
Please unlearn all the good stuff in this post and just circle A, B, C or D. We really don’t want testers in the world to think too much… they may end up being dangerous!
Yours sincerely,
David “The Examinator” Greenlees
LikeLiked by 1 person
Heed this man! I failed the heck out of the ISTQB practice paper by thinking too much (http://secondsignofmadness.blogspot.co.uk/2014/06/i-failed-istqb-practice-exam-part-i.html)
LikeLiked by 1 person
OMG! Please make sure our paths NEVER cross. I don’t want to associate with, let alone work with, such a crap tester as yourself!
😉
LikeLike