Illusions on Testing - Part I (Chimera)

I don’t know yet how many parts this post will have. I only know it will be more than one so I already started numbering the titles. My initial idea is to have 3 different parts (Chimeras, Reasons and Future), but I won’t promise anything as I write like I live my life; which is pretty exploratory and not based on too much preplanning.

Recently I made a small-scale research for example on different concepts of testing and how testers approach testing problems. I call them testing problems because I presented them as problems, questions. I’ll present and analyze the results in this series of posts.

The first observation is related to pre-scripting testers’ work. There were a few problems where the testers needed to create test cases, test ideas, minimalistic test plans and so forth. Some of the people had really nice ideas with interesting inputs they might use in testing. The biggest shortcoming s came from forgetting testing outputs and “the big picture” (as in, you have a dialog box, but how does actions on it affect on OS, browser, file system …). Basic things like user rights, file system connections and “simultaneous” manipulation was forgotten. As said earlier, I’ll go to the reasons in the next post.

A really interesting observation is most people thought a “check” is a “test”. I could understand how the words get confused with “common” people, but it was a surprise coming from software testers. I see this as something that should be addressed as soon as possible. Checking is something I can do with a computer program.  Testing needs skills, imagination and intelligence – something way over the league of current computers and software. Testing is about learning and questioning.

Recently I had a wonderful discussion with Tobias Karlsson (https://twitter.com/#!/tobias_karlsson) about the difference between testing and checking.  He is against the distinction and feels it’s useless. The discussion was wonderful for two reasons. Firstly, because he admitted he understands the difference between checks and tests. Secondly, because his logic was wrong, which gives confidence my own logic was working well.

He mentioned calling UAT not testing is like saying thrash/speed metal is not metal. This is clearly a play with words instead of a real comparison between the distinctions. Checking has a one-dimensional result; there needs to be a presumption/belief before a check can be done. A test, however, finds new information and it answers to questions like “what will happen if”.

A third remark I made was not surprising at all. Women were doing much better than men. When I say “much better”, I mean *much* better. Simply put, women had way more context in their answers, they analyzed the problems from many points of view, they challenged me by leaving questions in their answers and they used online search engines only to confirm some facts they already thought of. There is an enormous difference between most replies from men in this respect.

The fourth point was something I was prepared for. Except that I wasn’t ready on the scale of confusion with “exploratory testing”. I found it alarming when testers don’t see exploratory testing in the way it should be respected. For some reason (I blame mostly ISTQB Foundation certificate for this, but more about that later) ET is seen as something testers do when they don’t have anything better to do. The love many have for test cases is slightly distressing, too.

Fifth, and last, observation I am going to list was the colossal amount of claims without explanations/arguments.  A few examples: “test cases show how much testing is covered”, “testing efficiency depends on (code) design plans” and what I really enjoy “test cases can be run by inexperienced testers”. The last was repeated by many testers. I don’t mention here any of the “best ways” to do things, because I used that as a trap in a few problems.

With all the saddening replies on some of the testing problems, I still believe there is a lot of potential in each tester in question. They only need guidance, motivation and someone to walk along them. What could be a bigger challenge than to improve others to a level they were not even aware of?