The Science of Testing

Previously posted on the PQA blog.

The best way to approach a problem is typically to look at it from different angles, to turn it over and to discuss it until a solution can be found.  Similarly, it is important to try to bring different perspectives into your work to develop your skills and extend your toolbox.  This article explores the parallels between software testing and science, and highlights what testers can learn from the scientific method.

What is the Scientific Method?

The ultimate goal of all sciences is knowledge, and to acquire new knowledge, scientists make observations and analyze data – activities we normally refer to as research.  The scientific method is simply a collection of techniques that scientists use to investigate phenomena and gain new information.  For a method to be considered scientific, it must be based on gathering empirical evidence.  Empirical means acquired through observation or experimentation – making claims without experimental evidence is science fiction, not science.

Here, we can already draw our first parallel to testing.  We test software to try to learn how it works; like a researcher, our goal is to gain new knowledge.  If we already knew everything about the software, there would be no reason to test it!  When we test software, we are in fact experimenting and observing the results.  Testing is simply gathering empirical evidence.  Hence, we can draw the logical conclusion that good testing adheres to the scientific method!

Simplified, the scientific method involves the following workflow:

  1. Collect data through observation
  2. Propose a hypothesis and make predictions based on that hypothesis
  3. Run experiments to corroborate the hypothesis

If the experiments corroborate the hypothesis, additional predictions can then be made and tested.  If the experiments instead refute the hypothesis, it is necessary to go back and propose a new hypothesis, given the additional knowledge gained from the experiment.

A trivial example would be:

  1. We observe a mouse eating cheddar cheese.
  2. Based on this observation, we propose the hypothesis that our mouse will eat all sorts of cheese and, in particular, we predict that our mouse will also eat Swiss cheese.
  3. We give our mouse Swiss cheese and eagerly wait to see if the cheese will be eaten.

If the mouse eats the Swiss cheese, our hypothesis has been corroborated and we can predict other consequences, for example, that the mouse will also eat goat cheese.  If the mouse does not eat the Swiss cheese, we have to go back and suggest a new hypothesis.  Maybe the mouse only likes cheese without holes in it?

The scientific method is cyclic and dynamic; it involves continuous revision and improvement.  Based on observations, a hypothesis is proposed and the consequences of that hypothesis are predicted.  Experiments are setup and run to test the hypothesis, and the results are evaluated and used to propose a revised – and improved – hypothesis.

The scientific method.

The scientific method. Based on observations, a hypothesis is proposed and a prediction is made. Experiments are setup and run to test the hypothesis, and the results are evaluated and used to propose a revised hypothesis and so on.

How does the scientific method apply to software testing?  Let’s assume we are testing a simple text editor.  The workflow of the scientific method maps to software testing as follows:

  1. Learn the product and observe how it behaves
  2. Identify risks and predict potential failures
  3. Execute tests to reveal failures

Based on the results of our tests, we will identify other risks and use this new knowledge to design additional tests.  The process of testing a software product has striking similarities with the workflow typically adopted by scientists. But does software testing enjoy the same level of credibility as science?

Credibility

The word science comes from the Latin scientia, meaning knowledge, and when we talk about science, we refer to a systematic search for, and organization of, knowledge.  Typically, the word science is associated with authority, expertise and – last but certainly not least – credibility.

Whether we find something credible or not depends on:

  1. What we know about the issue – evidence
  2. How compatible something is to our own worldview
  3. Reliability of the source
  4. What are the consequences of accepting or refuting the issue

We are often more likely to believe statements if we have little knowledge of the topic – we simply do not have any counter evidence.  Some sources are also seen as more credible than others; if we read something in the morning paper, we are more likely to believe it than if it is posted on Facebook.

In testing, what we know about the issue equates to our test result.  How compatible the behaviour of a piece of software is with our prior experiences has an impact on our expected test result, and therefore might make us biased.  The reliability of the test result depends on who tested the product and how it was tested.  Finally, there may be direct consequences of reporting or rejecting any particular bug, which could affect our objectivity.

Is the testing done at your workplace seen as credible, and who assesses that credibility?  The first step in increasing test credibility is to look at the factors that will raise the likelihood that we will believe in both the process employed and the actual result obtained.

The Science of Testing

What characterises science is that it aims at falsifiable claims, whereas pseudo-science, or non-science, typically makes claims that cannot be verified.  Here, we can draw a second parallel between science and testing.  Software testing that embraces the scientific method tries to find ways in which the software fails rather than trying to prove that the software works.  How do we recognise the testing equivalent of pseudo-science? How can we protect ourselves, as testers, from being fooled by false claims? The best way is to nurture our inner scientist, and strive for an unbiased and reflective approach to testing.

xBTM: Harnessing the Power of Exploratory Testing

Previously posted on the PQA blog.

xBTM is a topic that I keep coming back to, and I think that is all in good order. Nothing is perfect from day one – we need to go back and rethink and revise our ideas continuously.

I’m a big fan of exploratory testing – it provides both flexibility and speed, characteristics that have become increasingly important, especially with the quick pace of short agile iterations.  But, how do you retain traceability in exploratory testing without losing your creativity? How do you, as a manager, actually manage testing that is unscripted and improvised? One answer is to use a combination of session-based test management (SBTM) and thread-based test management (TBTM) called xBTM.

Session-based Test Management (SBTM)

SBTM (Jonathan Bach, 2000) is the reply to the common misconception that exploratory testing is by its very nature always unplanned, undocumented and unmeasurable. Using SBTM, exploratory testing is done in time-boxed sessions. Each session has a mission that is specified in the test charter, for example, the sessions are actually planned ahead (but not scripted). A session results in a session report that provides a reviewable record of the testing that was done. Based on the session report, the test manager can derive various metrics to track the progress of the testing. SBTM doesn’t have much in common with ”ad hoc” testing, in fact – SBTM is quite strict, sometimes even too strict.

Thread-based Test Management (TBTM)

There are environmens too chaotic or hectic for SBTM to work properly, and this led to the introduction of TBTM (James Bach, 2010). TBTM is a generalization of SBTM that allows for more freedom and flexibility. The core of TBTM is threads and, similarily to sessions, threads are test activities intended to reach a certain test goal. The big difference is that whereas a session is a committment to complete a specific task in a given amount of time, threads come with no such committments. A thread can be interrupted, resumed, canceled or go on indefinitely. SBTM defines both where to start and where to end, but TBTM only definies where to start. In TBTM, activities are allowed to change over time. TBTM is the free-spirited sister of SBTM.

SBTM -> TBTM -> xBTM

SBTM and TBTM both have their strengths and weaknesses, and rather than having to choose one over the other, xBTM was created. xBTM unites the two exploratory techniques to get the full advantage of both, focusing on increased test efficiency and creation only of artifacts that actually add value. The name xBTM highlights that it is combination of SBTM and TBTM (x = S or T).

The xBTM workflow is centered around a mind map:

  1. List all test ideas and activities – these are your threads.
  2. Arrange all threads in a mind map grouped by function area or test technique. This constitutes the test plan.
  3. Prioritize the threads.
  4. Use SBTM when possible. Group threads to create suitable charters and create session reports once the charters are executed.
  5. Use TBTM when SBTM is not an option. Follow the threads where they take you and add new threads when needed.
  6. Update the mind map continuously, showing the progress and state of the threads. The mind map is the status report.

Using a mind map, rather than a traditional text document, makes conveying the test plan easier, and it helps communication — not only within the test team but across the whole project team. Figure 1 shows a simplified example of what the xBTM mind map for testing an e-shop might look like.

Image

Figure 1: Example of an xBTM mind map. Here, an e-shop is being tested, and the mind map shows some of the threads. For example, “Create user account” is being tested in a session, whereas “Add item to shopping cart” is being tested as a thread.

Want more?

Want to know more about xBTM? A detailed walk-through of applying xBTM on a real project is given on my old blog.

Also, look for our workshop ”Exploratory Testing on Agile Projects” at STAREAST in 2013.

Links

There are many different mind map tools to choose from and, in most cases, the core features are very similar. Which one you end up picking will mainly be a question of personal taste. The author has had very good experiences using the following three tools:

  • —XMind: A powerful tool with a good selection of icons and markers. The base version is free and there is a Pro version available for purchase. XMind was used to create the example in this article.
  • —Mindmeister: Collaborative tool that stores mind maps online and allows multiple users to edit the same mind map simultaneously. The free version limits the number of mind maps you can create and store.
  • —FreeMind: The simpler of the mind mapping tools, but still very useful and completely free.

Additionally there are a number of tools that help you manage your session-based testing. Most of these tools focus on recording the session reports. Three tools which the author has used in the past are:

  • —Rapid Reporter: A note-taking tool for exploratory testing sessions.
  • —Session Tester: A tool for recording and managing exploratory testing session.
  • —SBTExecute (scroll down to bottom of page for English): A tool that produces summary reports and calculates metrics from an Excel session report template.

Stay tuned for the next episode…

 

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

The new Boeing 787 Dreamliner can carry about 250 passengers. This blog was viewed about 1,200 times in 2012. If it were a Dreamliner, it would take about 5 trips to carry that many people.

Click here to see the complete report.

Are you a tester Labrador, or a Chihuahua?

I often think of software testers as being either Labradors or Chihuahuas.

Labradors are friendly creatures. All they want is for the pack to be together and everyone to be happy. Labs just want to help everyone out, even if as a human I don’t always find a drooling dog snout on my keyboard helpful. That wagging tail is just a display of love and joy, even if it happens to send your best china crashing to the floor. There’s a reason why people use Labs for service dogs.

What strikes me as the most typical Lab characteristic, though, is their curiosity. Labs view all things new as potential toys. And if you can’t play with it, you might be able to eat it. Sometimes you’re lucky and can do both. Labs happily embrace new things with their tails wagging full blast.

Chihuahuas are slightly different. In general they tend to be anxious little bundles of nerves. A Chihuahua faced with something new often reverts to incessant yapping while running around in circles, or hiding behind a larger object (which is basically anything apart from another Chihuahua). Chihuahuas also seem to be more territorial. They will ferociously guard what they consider theirs and might even go for the throat. Which in reality means they might reach your calf. I can’t help but sometimes wondering what the purpose of Chihuahuas is, apart from, well…being lap dogs.

Chihuahuas also seem to need more accessories than Labs. There are unlimited fashion items such as sweaters, jackets, boots etc available. Labradors just need the world to explore. In dire cold and meter deep snow you may give a Lab a blanket to sit on outside. You do not dress Labs up just for the sake of dressing them up. Travelling with a Lab means packing food, a leash and if you’re feeling extravagant maybe a tennis ball.

My apologies to all Chihuahua fans out there.

20120904-212258.jpg

Update: Short Notes from Test Coach Camp 2012

Role Models
During the Test Coach Camp role models session I made some notes on a flip chart. I tried to add these (in green) to the mind map. I would appreciate comments/additions from other discussion participants.

20120717-084545.jpg

Short Notes from Test Coach Camp 2012

It was fantastic, it exceeded my expectations, I learnt how to make a monkey braid and I got to spend time with a bunch of great people.

I had suggested two sessions, and I was fortunate enough to have enough people interested in the proposed topics to make the sessions happen. These are just some brief initial notes for documentation purposes.

Role Models
I wanted to talk about how to be a good role model, not only for testers. I had arranged my thoughts on the topic in a mid map, and I used part of it as a starting point for the discussion.

20120716-134240.jpg

We had a great session where amongst other things we talked about (in no particular order):

  • Victim mentality
  • Judgement
  • Getting too personal
  • Dealing with concflict
  • Being a role model from a technical vs human perspective
  • Give and take, or Give and receive
  • Replacing “but” with “and”
  • Admitting to not knowing the answer
  • Asking the right questions
  • Being humble
  • Not making people feel bad

It is all fun and games!
I also wanted to talk about having fun at work and how to engage and motivate testers. We ended up having a session that focused on games and puzzles. I finally got to play the dice game, I am now able to answer the question “Is this a pen?” and I got hooked on The Set Game.

My original thoughts on the topic were:

20120716-134823.jpg

Let’s Test 2012

I’m writing this as I’m occupying seat 32C on a flight from London back to Vancouver, 8839 m above ground. My trip started earlier today when I left Stockholm after having attended the first, but surely not last, Let’s Test conference.

Several excellent reviews of the conference have already been posted, so I thought I’d keep this post short and light-weight, but there are a few things that I’d like to comment upon:

Setting: The conference was held at Runö, beautifully located by the water, far away from the city bustle. The spring sun shining, vitsipporna (Google it) in full blossom, sheep grazing the pastures, bird song keeping people awake – the whole package. Semi-secluded venues like these make escape difficult, thereby promoting interaction and socializing, which is what conferences should be about.

Food: Those of us who arrived on Sunday had the pleasure of experiencing the festival of Swedish flavours: salmon, pickled herring, smoked prawns and the Swedish version of cheese cake. There seemed to be some scepticism regarding the pickled herring though. Try it next time, it’s the Swedish sushi!

Mood: So much energy, and so much enthusiasm! I feel invigorated and energized and ready to go out and save the world! Whether it needs saving or not. The mood at the conference was absolutely fantastic – people were friendly, encouraging and the discussions were creative and inspiring. It was an humbling experience, being allowed to meet and talk to all these brilliant testers readily sharing their experiences. Most impressive was how positive everyone was – it was a refreshingly bright picture of the future that was painted.

People: It did feel a little bit like a SWET (Swedish Workshop on Exploratory Testing) reunion, but it was great to also see a lot of new faces. There’s certainly good hope for the future of testing.

Lasting impression: In retrospect, the common thread of the conference seems to have been drawing parallels between testing and other disciplines – covering everything from hypnosis to military service. I like the idea of not looking at testing in isolation, but relating testing to the world in which the testing actually takes place and learning not only from our own community but also from non-testers.

Closing words: A big thank you to the organizers, and also to all the attendees for making it a fantastic conference. I’ll be back next year. Not organizing the conference again is not an option.

How I choose conferences

I love going to conferences. After a good conference I always come back to work full of energy and with a thousand new ideas to try out. Conferences inevitably boost up my enthusiasm a couple of levels. Usually my network has expanded quite a bit too, and I’ve met people that are not only friendly and interesting, but also amazing sources of useful tips, advice and insight.

Even though I’m fortunate enough to work with a great bunch of testers, nothing beats the intensity of the testing conferences where approaches and techniques are scrutinized, refuted and worshipped into the wee hours.

What do I look for in a conference?

  • Enthusiasm: Attandees that are there because they’re passionate about the topic, and not because the conference is at a fancy resort, or because they want to skip work for a couple of days. Judging the level of enthusiasm at a conference is hard to do in advance if it’s a conference you’ve not been to before, but I have faith in my peers and ask around – someone is bound to have been there.
  • Topic: I want to be enthusiastic too, so of course the topic of the conference has to be something I care about strongly.
  • Socialising: The time outside the sessions is usually at least as important. There must be time for just hanging out and talking about testing. That’s one reason why I usually prefer to stay clear of the really huge conferences – there seems to be an inverse relationship between the number of attendees and the number of productive interactions.
  • Price: Yes, the price matters, but I’m also prepared to pay for quality (come on, I’m a tester!).

What’s on my confence schedule for the next couple of months? Well, no less than three conferences I’m very much looking forward to:

  1. Let’s Test, Sweden, May 7-9
  2. CAST 2012, USA, July 16-18
  3. TesTrek Vancouver 2012, Canada, July 25-26

I hope to bump into you!

20120404-210902.jpg

Follow

Get every new post delivered to your Inbox.