So… Another blog post stemmed from the depths of the LinkedIn forums. This time someone was asking for advice about answering an ISTQB exam question.
The question is as follows (with this exact wording):
If you are flying with an economy ticket, there is a possibility that you may get upgraded to business class, especially if you hold a gold card in the airline’s frequent flier program. If you don’t hold a gold card, there is a possibility that you will get ‘bumped’ off the flight if it is full and you check in late.
Three tests have been run:
Test 1: Gold card holder who gets upgraded to business class
Test 2: Non-gold card holder who stays in economy
Test 3: A person who is bumped from the flight
What is the statement coverage of these three tests?
There are so many problems with this question.
Lets start with the first problem regarding terminology. In a software development world, where communication is vital, we need to use terminology that people understand. And if we are creating documentation, we should simplify our terminology as much as possible to avoid confusion and misunderstanding from others reading the documentation. So I ask, what do you think when you read the words “statement coverage”? What does it mean? Especially in the context of this ISTQB question. Even when it appears that the question previously used the term “tests” when talking about them being run (but in fact, they would actually be “checks”, right?). Do statements relate to “tests” (/”checks”)?
Either way. I’m confused by this terminology. I’m guessing that its asking for the level of coverage for the 3 checks it’s supplying regarding the feature or system it’s describing.
This brings me to my second problem regarding the context. We’re asked to read an initial paragraph of information. From this we can gather that its regarding an airline’s frequent flier program and the possibility of getting upgraded or “bumped” off a flight based on the frequent flier program and gold card membership and whether the flight is full or not. The name of the airline is not supplied.
My problem is that there is a major risk of falling into the “assumption trap” regarding only the explicit information.
But before I go into more detail on these assumptions, let me mention my third problem with the question as it ties in with it: The supplied A, B, C and D answers are all wrong.
If we look at the explicit information, we can see that we have a few variables:
- The ticket type
- The upgrade option
- The airline membership card
- The flight being overbooked
When we look at only the explicit information about these variables that is supplied in the paragraph, we might think that there are only 2 types of tickets (Economy and Business). But in reality, anyone who has flown with any major airline, might know that there are actually other ticket types available. For example, Premium Economy and First Class tickets can be purchased.
Now, taking that extra information into account, we have explicit information from the paragraph about upgrading to business class, but with there being other class types, what about upgrading to Premium Economy or First Class too from Economy? And of course the extra scenarios around upgrading from Premium Economy to either Business Class or First Class and even upgrading from Business Class to First Class.
And what about airline membership cards? Well, I’ve been a member of a few airlines collecting air miles for some time, and I know that there are various levels of membership cards: Blue, Silver, Gold, Platinum, etc. Most major airlines offer at least 3 levels of membership, not just a Gold Card. And then of course there is also that explicit information of the possibility of not having a membership card.
And we have the 4th variable of the flight being overbooked. But really, are we assuming that this means the full flight is overbooked, or does it mean just the seats for the class that your ticket is for?
I once flew in economy, but the economy section was overbooked, so they moved some people up to Premium Economy free of charge rather than “bumping” them off the flight completely.
So lets collate all of this information together…
You can see from the mind-map (clicking on it should expand it to make it readable), that there are actually around 92 different scenarios if we take into account this extra information within the variables that are sourced from our real life experiences with flights and airline memberships. This means that the 3 “checks” mentioned would actually equate to about 2.76% coverage. (Of course, the caveat is that this is based only on the scenarios that I could think of… so far at this point in time… based on my own experiences and beliefs. There might be more information that I don’t have).
The thing is… If we were treating the ISTQB’s system as if it was our own real system that we were working on, then one of the most important things we need to do is to use our thinking skills to think about as many variables and questions about those variables in order to stem more information on the “what”, “why”, “who”, “where”, “when” and “how” of the feature or system that we are being asked to develop and test.
We need to dispel the assumptions of the requirement, and not just fall into the trap of relying solely on the explicit information from the single paragraph that we receive. We need to use multiple resources, such as our experiences and perhaps sales and marketing material or even competitor products, etc.
If we jump on only the explicit information supplied in the 2 sentences, like the ISTQB question supplies, then you’ll fall into that “assumption trap”.
And this brings me to my final problem with the question. Testing is about investigating ands questioning. When we have a requirement, we need to question the assumptions and investigate the variables and question them some more in order to refine the requirement so that we can start building and testing the “right thing”. When we have built a product based on that refined, tested requirement, we question and investigate it through operation and observation so that we can test (and check) that the “right thing” is “right” (if that makes any sense 🙂
ISTQB are completely missing this in their exam question. Instead they are focussing on “Statement coverage” and their answers rely on you NOT questioning the requirement in the initial paragraph.
Have you seen any similarly appalling ISTQB questions?
How would you have answered this one?