Reasons of Illusions on Testing - Part II (Reasons)

In the previous (http://jarilaakso.blogspot.com/2012/02/illusions-on-testing-part-i-chimera.html) post I noted five different chimeras of testing. They were:

1.       Pre-scripting testing work

2.       Using terminology (test vs. check)

3.       Women are better on an average

4.       Comprehending exploratory testing

5.       Writing claims without arguments

As some of you already speculated, I am going to put some blame on ISTQB etc. for all these problems. I am not saying it’s all because of them. I actually blame the testers for all this, they are responsible of their own answers, but it’s quite clear how “certification” is visible in their answers. I don’t want to turn my blog as a rampage against any kind of “authority”; I am only pointing out for example what kind of problems certifications can cause.

When we think about the problems with pre-scripting testing, as in creating test cases, we can rapidly see the testers were able only to cover very shallow cases. There were indeed really good ideas too, but the problem I see is that testers weren’t able to write down some really high risk cases, things I would call “basic tests”. On the other hand, I am pretty sure many of the testers would perform these tests. Maybe not on the first “round”, but evidently when given time. Possible reasons why this problem occurred: too hasty work, poor documentation skills and lack of imagination.

The second problem I noted was lack of differentiating testing and checking. I’ve talked lately with quite many people about this and pretty much everyone seemed to think it’s not a necessary differentiation. I know this might sound like a troll, however, I claim everyone serious about testing understands the difference and uses it accordingly. I don’t really care if non-testers use “test” in any way as long as testers use it properly. All professions have their own terminology (jargon) for a reason. An additional rationale for differentiating the words is for example to avoid anyone thinking that testing can be automated. A few reasons for the problem: not being serious about testing, using ISTQB terminology and working with customers who use a certain kind of terminology.

On the third point I illustrated how women were doing better than men in the test. It’s true that some of the best answers came from men, but in overall women had better results. (As a side note, I remember reading about quite many surveys on various domains where the same patters repeats.) There are many possible reasons for this result and I think one big part is that generally women analyze situations more than men. Men seemed to lack confidence (online searches for answers and even misquoted) and have short answers, which could imply they either were lazy or want to use more time with hands-on activities.

The fourth chimera was about exploratory testing. Everyone appears to write it in their CV, but few understand what it is. It’s seen as something that will be done after all test cases have been executed and there is time to look for other issues, it’s widely described as experience-based technique and commonly understood it doesn’t have any means for providing information what has been done. My blame-list for reasons: ISTQB, lack of professional aspiration and the culture in the company they are working in.

I saved the best part for the last. There seems to be a strong culture of giving claims without justification. If the claims would make sense (i.e. I would agree with the claim without hesitation and I would know how the tester came to the conclusion), I could accept some of them. But in this case there were a lot of claims I just can’t accept. Most likely not even with strong argumentation, which would be the minimum requirement for me to consider the statements as valid ones. (Before sending the problems to testers, I mentioned there are no correct answers, but they need to be able to convince me their answer is a good one.) The main reason behind claims without explanations: lack of critical thinking.

What do you think are the reasons for the mentioned issues? Maybe you want to share another general problem you have seen in testers thinking?

Illusions on Testing - Part I (Chimera)

I don’t know yet how many parts this post will have. I only know it will be more than one so I already started numbering the titles. My initial idea is to have 3 different parts (Chimeras, Reasons and Future), but I won’t promise anything as I write like I live my life; which is pretty exploratory and not based on too much preplanning.

Recently I made a small-scale research for example on different concepts of testing and how testers approach testing problems. I call them testing problems because I presented them as problems, questions. I’ll present and analyze the results in this series of posts.

The first observation is related to pre-scripting testers’ work. There were a few problems where the testers needed to create test cases, test ideas, minimalistic test plans and so forth. Some of the people had really nice ideas with interesting inputs they might use in testing. The biggest shortcoming s came from forgetting testing outputs and “the big picture” (as in, you have a dialog box, but how does actions on it affect on OS, browser, file system …). Basic things like user rights, file system connections and “simultaneous” manipulation was forgotten. As said earlier, I’ll go to the reasons in the next post.

A really interesting observation is most people thought a “check” is a “test”. I could understand how the words get confused with “common” people, but it was a surprise coming from software testers. I see this as something that should be addressed as soon as possible. Checking is something I can do with a computer program.  Testing needs skills, imagination and intelligence – something way over the league of current computers and software. Testing is about learning and questioning.

Recently I had a wonderful discussion with Tobias Karlsson (https://twitter.com/#!/tobias_karlsson) about the difference between testing and checking.  He is against the distinction and feels it’s useless. The discussion was wonderful for two reasons. Firstly, because he admitted he understands the difference between checks and tests. Secondly, because his logic was wrong, which gives confidence my own logic was working well.

He mentioned calling UAT not testing is like saying thrash/speed metal is not metal. This is clearly a play with words instead of a real comparison between the distinctions. Checking has a one-dimensional result; there needs to be a presumption/belief before a check can be done. A test, however, finds new information and it answers to questions like “what will happen if”.

A third remark I made was not surprising at all. Women were doing much better than men. When I say “much better”, I mean *much* better. Simply put, women had way more context in their answers, they analyzed the problems from many points of view, they challenged me by leaving questions in their answers and they used online search engines only to confirm some facts they already thought of. There is an enormous difference between most replies from men in this respect.

The fourth point was something I was prepared for. Except that I wasn’t ready on the scale of confusion with “exploratory testing”. I found it alarming when testers don’t see exploratory testing in the way it should be respected. For some reason (I blame mostly ISTQB Foundation certificate for this, but more about that later) ET is seen as something testers do when they don’t have anything better to do. The love many have for test cases is slightly distressing, too.

Fifth, and last, observation I am going to list was the colossal amount of claims without explanations/arguments.  A few examples: “test cases show how much testing is covered”, “testing efficiency depends on (code) design plans” and what I really enjoy “test cases can be run by inexperienced testers”. The last was repeated by many testers. I don’t mention here any of the “best ways” to do things, because I used that as a trap in a few problems.

With all the saddening replies on some of the testing problems, I still believe there is a lot of potential in each tester in question. They only need guidance, motivation and someone to walk along them. What could be a bigger challenge than to improve others to a level they were not even aware of?

Internet Banking Experiences From Tester’s Point of View

Obs: I am not going to reveal the names of the banks in question for the time being. However, I might change my mind later and add them here. My intentions are not to show how bad these banks are but to show how bad experiences I’ve had. I’d appreciate if readers would respect this and not even try to guess the names of the banks in the comment section. However, I do recommend readers to share their experiences and I am not imposing any rules on what you should or should not tell.

I have experiences with Internet banking from S (Finnish), R (Austrian) and T (Romanian). The experiences differentiate widely from UX point of view. In this post I will put in plain words some of the obscurities I have seen with the Internet services of these banks. Please note already at this point that in my opinion an Internet bank should be top-notch from quality, usability and user experience point of view. Not only because my examples are hugely profitable and wealthy organizations, but also for example because they have so diverse end users in their systems. Respecting end users is a must for them.

S

There is a person who can access my account details. I asked from the bank about this and was told they don’t know how it’s possible. I updated my service agreement with them and was told the problem is not going to remain. The problem remains.

There was a big integration project to D (Danish) system which was a disaster. (They lost in Finland almost 30 000 personal and over 2000 corporate customers in about half a year.) Even ATM’s stopped working and gave Danish messages in Finland. The online banking integration resulted in people not being able to sign in the system, getting duplicate payments, receiving error messages in Danish… After this, S has had their fare share of problems with an average of over 1 major/critical issue each year.

R

I started using them around the beginning of 2008. At this point, the system didn’t need a token or anything that changes; it was enough to know a username and password.

I forgot my password at some point as I didn’t use their services in a long time. I noticed quickly two things:

1) The system was clever enough to give a different error message for a wrong username and a wrong password.

2) The login page didn’t have a restriction for unsuccessful login attempts.

Later on in 2010 or so I thought to return as their Internet banking customer. I went to the bank and realized how nicely they had adopted using tokens. I felt security was finally taken care of. Unfortunately, this time the bureaucracy to actually get it working was too much, so I left after 30 – 60 min of trying to get an account.

T

I wanted to leave the ultimate example as the last. I have seriously considered many times to change my bank because of the problems I have had. I still am. I just haven’t found a (local) bank with much better ensemble.

First was to take Internet banking in usage. This happened either with Token/SMS or installing a certificate on the PC and using that for credentials. I wasn’t given a token, but thought to try to SMS option. Surprisingly the SMS option wasn’t implemented. All I saw was “login with password/token”.

Then I thought to try the certificate authentication, but that page returns “SSL peer was unable to negotiate an acceptable set of security parameters.” when trying to access it. I tried to request some certificate via an online service, but never received anything.

Finally I got a token from the bank and I was ready to give another try. I changed the language to English and noticed the bank cared only to translate part of the essentials. I thought to keep a consistent language all over so I moved back to Romanian. This logs you off their service and asks to login again.

An interesting note is that the login screen has Romanian, English, Hungarian and French as the language options. Other parts of their web page had only Romania, English and Greek (!).

Then some issues from inside the banking system. When you are on the Messages view, the oldest message is on top of messages by default. Always. Change the order, read a message, return to the Messages and notice the default order is again applied.

I paid two bills with great success. The first bill was accepted (I was even prompted “Operation successful!” and presented a green circle to represent “OK”) and I was happy how easy it all was. Until today, 8 days later, when I found I had a new message in the Messages section (there was no alert for it, nothing in main page to get my attention etc.). The Message was that the operation was not successful because a mandatory input was missing.

The second bill was a success story, too. I had to pay a gas bill of 133 lei and 77 bani (1 ban is a cent of 1 leu), so I logged again to the online bank. I used the numpad for inputting numbers, because it’s more fluent for me like that, so I input 133,77 lei. I authorized the payment and noticed something odd on the screen: I just signed a payment of 13,377.00 lei. Happens that comma is forcing the amount to kilo-amounts. What a wonderful feature!

Now I needed to cancel the transaction before it’s done. (I forgot to mention that I did this at 8:30 AM and their Internet banking is performing transactions only 09:00 – 17:00 Monday to Friday. Yes, I am not lying.) I found “Order Tracking”, but there one is allowed only to view transactions, not to do anything on them. Finally from “Operations” -> “Standing orders” I found the payment. I was happy to see there is a Cancel checkbox; until I noticed it’s grayed out. I didn’t have user rights to cancel a payment I did.

Regarding user rights, when I got my Internet banking working, in a way, I had access only to other of my accounts. It took 3 personal visits to the bank to have rights to see details of my other account.

Tips for CV's and Job Applications

The following is slightly idealistic view on applying for software testing jobs. Reader is recommended to reflect his own values and think what kind of compromises he is ready to make.

What to considering avoiding on your CV/cover letter

1)      Lying

2)      Using (for ex. ISTQB) certificates as a proof of testing skills

3)      Explaining experience without details (“6 years in web testing” doesn’t tell really anything)

4)      Explaining experience with too many/waste details

5)      Exaggerating your experience (don’t put “MySQL” in your skills if you didn’t work directly with it)

6)      Typing mistakes

7)      “Empty holes” in the timeline

8)      Abbreviations and jargon

9)      Confidential information (for example customer name)

What to consider including

1)      Explaining your skills and how you have improved on them

2)      Explaining what you bring to the company

3)      Explaining your way of testing

4)      Giving arguments to your approach to testing

5)      Showing a capability to critical thinking

6)      Showing contextual talent

7)      Humor (this is quite a bit so-and-so case)

8)      Extra curriculum activities such as blogging, writing, public speaking, visiting conferences

9)      A sample test report as an attachment

What jobs to consider passing up

1)      If they have “certification” written in them without a negate

2)      If the job includes creating/aiming at one-dimensional metrics (bug counts, test case amounts, …)

3)      If the application has a lot of buzzwords or nonsense in it

4)      If the job/company doesn’t meet your values

5)      If you are not even close to what they are asking for

How would you continue/edit the lists? Let me know about your thoughts!

Next post most likely about Internet banking experiences from a tester’s point of view. Stay tuned.

Tester Salaries – 3 Steps for Raises

I hear every now and then comments like “Google pays testers as much as developers”. On the other side, as I look around, I often see testers with much smaller salaries than fellow developers. Now what is causing this difference? I’ll provide 3 different explanations and look forward on readers to comment their own views on the matter.

1)      You don’t separate from others.

Ask from your managers (depending on company structure, this could be for example test manager, unit manager and HR manager) how you are different than other testers in the company. If they can’t give good answers, you need to start working on getting those answers for them. Now! You need to be able to stand out. You don’t want to be transparent for those people who decide your salary and affect on your career. Be creative and show passion to your line of work. Don’t settle for asking what is expected from you but amaze people with your ideas.

2)      You are working in a wrong company.

I understand we need to make choices in our lives and not everyone can work in the best company. Don’t get discouraged because of this. Everyone needs to start from somewhere and get food on the table. You can try to change the company, but if that is not going to happen, start looking for new opportunities. There are lots of great companies in the world and you can even make one yourself. Here are a few signs of a company you want to get rid of:

·         Managers and/or Developers don’t value testing

·         You don’t have the chance to participate conferences or other means to increase your skills and network

·         There is a lot of documentation involved (test case design, test planning, …) and metrics (numbers) are appreciated as the main measurement of testing efficiency/quality/performance

·         You are often told to do extra hours because “there is a release on Friday”

3)      You are not as good as you think you are.

Testing is not about being the bug hunting hero. It’s not about being the guy who finds the "best" bugs. Of course, part of the job is to find issues, but when we are talking about being a professional, we are talking about many other things. Your value to the company is increased significantly when you start training others, participate conferences, read books, recommend books to others, help developers to avoid mistakes… Especially the information sharing part is often overlooked. Make others better and your importance to the company will grow. Be more important than the fake hero of the test lab.

Test is dead vs. ISTQB kills people

Some time ago I wrote this http://www.satisfice.com/blog/archives/664#comment-265390 comment on James Bach’s blog. I would like to open up a bit more the whole theme and how I think about it.

There was a whole lot of discussion of “test is dead” this year. My claim is that “ISTQB kills people”. I don’t even need to use that as a cheap marketing trick adopted from Nietzsche. I don’t need to wear a black robe to say it. The only thing needed is to look news and see when poor software testing causes the loss of lives. (I am not claiming all that is directly because of ISTQB, but I am not too far from the claim either.)

As all clever readers understood, the formulation of the claim is made according to the “test is dead” nonsense. I do criticize ISTQB, and sometimes I do it loud, but surely it’s easy to comprehend I could not possibly have enough information to make claims like that with certainty. It’s a bait, a highly simplified declaration of an extremely complex chain of events. Ironically, that is exactly what ISTQB is doing to testing for example when they define “boundary value testing” like they do.

Many testers are against ISTQB partially because “it’s just a way to make money”. I have nothing against them making money. (I don’t know how much money they make, actually, I only know they don’t make it in Finland) I’d like to have a discussion with someone about this in fact because I think either I am missing something or those testers are; or maybe both. Nevertheless, I am against ISTQB for various other reasons, such as it doesn’t improve skills to think/question (actually quite the opposite), it isn’t about testing skills, it’s based on multiple choice answers and it advocates poor metrics.

I asked also:

How often have you seen a professional use scripts to be able to do his work? They might have some sort of checklists for helping not to forget important things, for example. But if you can imagine some profession where serious pre-scripting is needed, please let me know, so I can think about this again.

And got a reply from Jesper L Ottosen:

Moviemaking. … That is Blockbuster Movie Making.
But then there are also http://en.wikipedia.org/wiki/Dogme_95 movies etc.

I think this is a good answer for two reasons. Firstly, it’s not accurate in the sense of “serious pre-scripting” (which was not defined by me at all). Secondly, simplification of (blockbuster) movie making can be seen like this, just like is often happening to testing in the eyes of non-testers. When we think about a movie script, we are seeing something that could be thought to be for example user stories. Screenplay can contain for example expressions and movements of actors, but most movies have lightning, sound, cameras, make up, clothing, etc. which are not pre-scripted. But yes, there is indeed a wide variety of movies with a diverse amount of scripting done. I’ve seen quite a few Finnish movies where even the dialogues are not scripted – only some guidelines are given to actors.

Furthermore, I found out this comment by Michael Pilaeten (http://www.pilaeten.be/?p=1622) with Google:

In the comments of one of James Bach’s posts, Jari Laakso claimed that ISTQB is killing people. I can only assume that he’s joking, since no human with an IQ of less than 2 digits would read James’ blog.  But that does not mean that the whole certification discussing has little to no value.

It’s not quite a joke what I wrote, but as I said above, it’s neither a “this is how it is” statement. The essence is showing the foolishness of sentences like “test is dead”. I won’t dig in deeper in this if I don’t get any further comments. However, I would like to take a pick on his analogy between ISTQB certification and car driving license.

I understand how people confuse these to have something in common. At some point I thought this too. I will explain how I don’t see them anymore similar in any way. Please note, I am considering a driving license only from my own context as I have no idea of other driving licenses than a Finnish one. (Ok, I do know how it goes in Romania also, but I haven’t gone through the process personally and I don’t want to use second-hand information.)

ISTQB Foundation Level certification is basically about theory and vocabulary. (Not to mention how far away from reality they go with that.) A car driving license examination is about knowing theory (physics, traffic signs and law) and putting it into practice (driving in traffic, slippery road driving, avoiding obstacles, parking between two obstacles and so forth).

From driving license examination process you get feedback on how well you did, where you can improve, how to lower gas consumption, etc. If you pass the theory part (mandatory before you can even try practical part) you are allowed to try the driving exam. Obviously, you can’t pass the exam if you don’t know how to drive well enough to survive in traffic. (More context, I never did a driving exam is some 1000 person town, but possibly there it’s easier to pass.)

I don’t want to say all certificates and certifications are useless. I don’t even think everything about ISTQB is waste. But as an entity, with all parts included, it is pretty much futile. I hope this clarifies on my current view on the subject. I also hope to get lots of feedback and comments so I can review my own thoughts and get a grasp on how other people see this.