Thursday, November 29, 2012

Software Testers Don’t Break Code – or Do They

(Update: As the blog text came at 6 AM, it seems only appropriate to wake up at 4:30 AM to update the post. I fixed some inconsistencies, rewrote phrases and clarified my own point of view in the subject.)

Short description around the discussion before I go to rephrase what I wrote earlier. This all started from a semi-joke from my side to Erik Davis. I thought it's funny to say I am a tester who sometimes break code. I had the intention behind that I also write code sometimes. As the discussion didn't stop there, I thought first to write how to defend "testers break code" but while writing it all collapsed in my mind. This is why the original post was really confusing and my replies to the comments about it. I tried to keep the body of the original post here so there still can be apparent mistakes. I'd be glad to hear about those.

I have discussed about this topic a few times, but I never gave it much of a thought. I did think different aspects on a slightly superficial level, however, as I didn't write about those sides, I decided finally today at 6 AM it's time to update my blog. This won't be an exhaustive research on the subject, but it's intended to show how I think about the topic (and how I see discussion around it) currently. My thinking will change the more I dig into this.

When I talk about testers in this case, I talk about people who mainly do software testing. I don't include for example programmers and managers in the definition of a software tester, even if they would do testing. I can include them in the definition if testing is what they get paid for and they are continuously trying to improve their skills as a software tester. But I think in this case their title should be a tester or so.

Next logical step would be to think what is breaking. For looking into different meanings of that word, I used The Free Dictionary by Farlex. I won't go copy/pasting the text here, please just take a glance and see how many ways there are to use that word. I take a few snippets that I think apply nicely for software world:
  1. To force or make a way through; puncture or penetrate => break into software /  crack it
  2. To find an opening or flaw in => bug
  3. To render useless or inoperative => pretty major bug
  4. Go down, crash => software breaks during a test / over a period of time

In this respect, it does indeed /seem/ like testers would break software. This sounds counter-intuitive? Yes, definitely I have the same thought. (As I didn't find any better translations, I leave it here from the dictionary point of view. In my opinion, breaking would imply making the code unusable/working wrong.) But on the other hand, as these are used as arguments by some people, let's look into what they say: "If we have software running and by our actions it becomes useless, who actually broke it? Sure, the code was not resistant to *all* possible use cases, but please, show me an application that is."

There are many levels of breaking, one could argue to us. "There is the misuse, happy flow and how ever people want to call ways to use an application." This doesn't really add content to the topic. The definition of breaking is not subject to different ways to use an application. In my opinion, we can’t have in a long time, if ever, applications that handle every possible scenario without “bugs”. However, when we find those, we don't break the code, we find ways to expose threats to the value of the application.

There are two other points of view I still want to write about. I used some definitions of breaking above, but what if we say "testers receive the code broken"? Now this changes the discussion closer to existentialism, where many thorough deductions seem to end. The argument we would hear could be: "Let's say we have a light bulb, drop it on the floor and it shatters to pieces. Did we receive the light already broken or who/what broke it? One could say we broke it, one blames the manufacturing company, someone the engineers or faulty machinery. And there are always those who said the ground brake it with a devilish co-operation with gravity. Damn you Newton - why you just didn't eat your apples!"

In reality, software is not like a physical object that breaks by dropping it on the ground. Software breaks because there are problems with the code - and sometimes it seems to break because of problems in the infrastructure. If you hear analogies with breaking physical objects, changes are the discussion needs to change focus from "what is breaking" into "what is software" before the first question can be answered.

The second point is about how we actually use these words. I hear this really often and sometimes fall into it myself. Language is not static; it's constantly evolving to many directions. It's different if we write or say aloud the words. It's a lot different if we shout or whisper. For these reasons I don't consider it always that important what words people use if we know what they mean. (And obviously, we need to ask them about this if we are not "sure"; and even then in some cases.) But this is not the case with people such as scientists and engineers; they need to be accurate with their wordings. Beware of arguments that back up on interpretations!

I want to note here I prefer saying “I like to find ways how the application/system seems to reveal a threat to the value of the product” (shorter version: I like to see stuff break) or something in those lines. In other words, I prefer telling I want to /find/ something that I /believe/ is a threat to /some/ value. (As value is a relationship, I understand my “important” findings are not valuable to all stakeholders.) I don't like talking about breaking code or code being broken. However, I like breaking *things*. 

Friday, June 29, 2012

(Dark) Secret of a Great Tester

I recently read a Finnish article ( from a news paper which said the “dark secret of great employees is free overtime work”. This started a chain of thoughts in my mind that I would like to open up in this blog post. In the same time, I am hoping I will clarify my thoughts while writing.

The article is referring to another article where is estimated the workers of a Finnish labor union Pro are making about 2 million free hours yearly. Specialists and managerial roles are told to be with most unpaid extra hours. Some questions that popped up my mind:
-          Are they working at their fullest through the whole day or do they take additional “breaks” (like check their Facebook messages)?
-          Is this a problem because of bad management or for example employee’s own time management skills?
-          Do these people record their hours or this is based on their gut feeling? If they record the hours, how accurately it’s done?
-          What are the reasons they work extra hours? Why they don’t get paid for them?
-          Is anyone looking after what gets done instead of how many hours are done?

When I look back at my work history, I see myself being one of those doing long days, being enthusiastic about the product we have been building/testing, having very late and early meetings/e-mails/discussions with customers, taking responsibility for doing a great job and working for the team. So, did I work extra hours? Yes, always when I saw it was needed. So, did I get appreciation from it? Yes, always when it was seen by the team/management/customer. So, did I get paid for those hours? Yes, always when I reported the hours in the ERP. Obviously, I could not be paid for hours I didn’t mark.

Do I study “on my own time”? Of course! Do I think my employer should pay for this? No, because it’s something I am doing mostly for myself. I don’t except them to compensate for what they didn’t ask me to do, but I will greatly appreciate if they do so. I don’t want to be someone who is just hanging around there and executing test cases someone else has written. I don’t want mediocrity. I want to carry kittens from burning houses. I want to throw myself over an exploding grenade. I want to become great in whatever I am going to do.

So, what’s the problem here? According to the article, this should not be the case and even a person who is not going to do all that should be considered a great member of staff.

The article does rise up also the concern of people not getting any kind of compensation for their work. I don’t see this as relevant to the “dark secret” because clearly it’s not a secret of someone who is successful. That is a person who is being abused. That is about bad management instead of greatness. Maybe this is what the journalist even meant, but in this case she should re-write the article.

I am not saying everyone has to do it (= work hard; really hard). I am saying you have to do it if you want to be great. It’s not a dark secret all great people have worked huge amounts. It’s easy to say “oh but Einstein was so intelligent” and do nothing to achieve even a bit of what he did. If you don’t want to do it, don’t do it. But don’t come and ask for the same recognition those get who do it.

Thursday, June 28, 2012

Testing Challenge - Puzzle #1, part 2

This is the second part of the first ( puzzle I wrote. 

The reason for the second part is that James Marcus Bach ( found a solution to the original one. His solution was much different than what I had in mind and he adviced me how to change the wording. Nicely solved and thanks for the tip!

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks! 

So here goes!

What comes next in the series (you need to replace X's with correct letters to continue the series) and present your logic:
gra, avar, rvtXX

Wednesday, June 27, 2012

15 CAPTCHA Testing Ideas

This is a continuation to my friend Santhosh Tuppad's blog entry ( He decided to make a fantastic list what could be used when testing a CAPTCHA. I read it, liked it, and decided I want to add my few cents to the topic! Before I go to the test ideas, I want to note CAPTCHA will not increase security and it should not be used for such purpose.

Now I am sure some of these things will overlap with either each other and/or with Santhosh's list. This is fine by me because I wrote the list as things were coming to my mind:

1. Not to pass the information to the server in plain text.
2. Not to have the same CAPTCHA repeat (in a reasonable interval).
3. Try sending a decoded old CAPTCHA value with an old CAPTCHA/Session ID.
4. Catch the HTTP(S) request and check all parameters (for example possible ecnrypted CAPTCHA ID).
5. Server-side validation for inputs (for ex. injections like here
6. Using dynamic noise in the CAPTCHA is harder to break automatically than static noise (however, obscurity is not security).
7. Avoid possibility to "random success" like selecting an answer from a list.
8. If you are using a visual CAPTCHA, check out from here if it's of any good
9. Test the CAPTCHA with some CAPTCHA breaking tools to see if it's any good.
10. If you have access to the code (or someone can tell you technical details), verify from Open Source Vulnerability DataBase if the CAPTCHA has known issues
11. Test if the session is destroyed after a correct phrase is entered. Reusing session ID of a known image could make it possible to automate requests to the page.
12. Should try to avoid using I, l, 1, 0, o, O etc. because users have problems with them.
13. Can your granny register? Should she be able to?
14. Will the CAPTCHA make people disappear from the service?
15. Do you actually need a CAPTCHA or could you use another means for the purpose you had on your mind?

Saturday, June 23, 2012

Challenging claims, part 1 – Test Automation

I was reading yesterday this article and noticed it contains many claims I don’t (fully) agree with. Before I started writing this post, I chose to read a few more from them to have a better idea what they are saying – maybe I had missed something. The more I read, the more I am convinced this is a company selling fake testing (inputting “QA” in the name of the company doesn’t help at all with this feeling). I hope I am wrong with that and there is some misunderstanding somewhere. However, in the meantime, I will challenge some claims from their blog, starting from the article I quoted above.

“Beginning the automated testing before the software can support the testing will only make more work through additional maintenance and rework.”

Firstly, sentences like this are usually of no value to testing, in my opinion. To me, it’s like saying “don’t write code too early” or “wasting money on irrelevant documentation is useless”. Nonetheless, there is a mistake also. Test automation can be written before any product code is done and it can still be rather maintenance and rework free. If you don’t believe me, ask from a TDD evangelist.
(Yes, I am making an assumption here that the author meant “beginning writing the test automation” when he wrote “beginning the automated testing”. I am also making an assumption he means test automation scripts when he writes about test automation. I wish to be corrected if this is not what he meant.)

“Eventually, automated testing takes the daily monotonous work of conducting the same action over and over away from software testers.”

I have heard this multiple times. It’s usually said by testers who execute test cases someone else told them to do. Sometimes it’s said by testers who for example want to create a lot of user accounts at the start of each sprint.
Testing is not monotonous. It’s monotonous only when done wrong (in my opinion). Think of it like sex. It most likely gets boring if you only do the missionary position under the blanket on Saturday evenings when the lights are off. Maybe not my best analogy, but surely you see the connection?
What I would like to ask in this case is why I am doing the same things over and over again, could I stop doing it, what other means there are etc. I would also like to ask from the author what he considers as “automated testing” in this context.

“Automation will repeat test after test for days on end, never failing to conduct them in exactly the equal way.”

I’ve never seen this happen in real life. I’ve never heard this happens in real life. Automation fails because of source code changes, infrastructure updates, porting to different systems, timing errors… Even for the same reasons the code fails it’s supposed to test! There are false positives, false negatives, blockages, crashes etc. And just to mention it, in this universe I am seeing around myself, it’s pretty much impossible to conduct a test in the exact same way as it was done previously. Sounds philosophical? Good, it means you started thinking of it!

“Automated testing never gets tired or burnt out or forgets to do a step.”

Indeed, automation doesn’t get tired or forget, but it does, however, fail. We don’t call it computer getting tired; we call it for example a memory leak or taking over CPU. We don’t call it forgetting to do a step; we call it missing a step, having a step that was changed, getting a timeout for a step… Testing is (in most cases I am aware of) not about endless struggle to repeat the same things and hoping something will (not) break at some point.

“Automated testing can just confirm that the software is as good today as it was yesterday.”

Automated testing cannot confirm that. Just like testers don’t assure quality. This was the initial claim which led me to believe you are selling fake testing. If you really claim you/your automation can confirm this, you are either lying or ignorant. Good test automation can "provide some confidence that nothing really big and obvious broke", like Matt Heusser wrote.

Wednesday, June 20, 2012

Metrics - reply to Mike Talks

This post was originally supposed to be a comment in Mike Talks' blog ( but it became a bit lengthy so I decided to post it here...

Hi Mike,

There are great posts about metrics already available so I don't want to dig in too deeply in the subject. Nonetheless, I would like to comment your interesting blog post!

I like the start with "are we there yet"! I wish more bloggers would make such associations between "real life" and software projects / testing.

When you mentioned you are estimating how long testing will take, do you mean a case where you and the project manager have a long history together and you both know what kind of testing you will do in that time? I am asking because testing is never done and I don't think that is too secure way for a PM to create a testing budget. Will he/you have the code done at this part? Do you (as in, you with the PM) have a lot of history with the same product?

"I know some people hate recording hours on a project. I personally think it's vital, because it helps a manager to determine spend on a project."

Yes, in some (possibly even most) cases this is very good. From testing point of view, I am usually interested to know how much of time (e.g. 90 min sessions) is used to test certain features/functionalities. For example, if there are no bugs found in a week, I could start asking questions like why we are not finding bugs and could we use testing somewhere else more effectively. (If the goal would be to raise bugs.)

"What about the number of test requirements tested and the number passed? Personally I like this metric, as it gives a good feel of how many paths and features we've tested, and I do think it's useful to keep track of this (as long as it's relatively painless)."

In many cases this can be a good thing to track. There can be for example legal requirements that need to "pass". However, like you point out in your text, it can be also very misleading. Even if 100% of the requirements pass, it doesn't mean a product is good or doesn't have critical bugs.

"Another metric I've seen is simple number of test cases run, and number passed. ... However it's more than likely a lot easier to track this number than the number of requirements if you're running manual test scripts which are just written up in Word (unless you're an Excel wizard)."

This is easy to track, but I don't see it telling really anything interesting. What would be interesting to know is how you use this metric. Is it used to raise up questions or make decisions (inquiry or control)?

One thing I want to stress. Passed tests can be dangerous to follow because they don't tell us much (if any) of the product. They might even give false confidence of the stability/quality of the product to management.

"What about measuring defects encountered for each build?"

I like to see how these change over time and what questions the change might raise up. Just like your text explains a situation where earlier bugs were blocking further testing.

When it comes to regularity of metrics, I think automated/scripted systems would be great to give numbers in many cases. Numbers can raise up good questions, but I would prefer not to use too much testing time to collect them. Depending for example on project size of course.

"I had to keep daily progress number updates. After day 5 I had still not finished script 1. In fact after day 8 I still wasn't done on script 1. On day 10 all 3 of my scripts were completed."

Maybe the progress should have been communicated differently? Sounds like there was "too big piece" to be reported for 1 progress step. I don't know what the script involved, but here is an example for a UI check:
- Needed (navigation) controls are coded
- The flow between controls is automated
- The assertions/verifications/requirements of each successful step are automated
- Minor changes to produce script #2
- Minor changes to produce script #3
- Testing, fixing and re-factoring the scripts

As we saw, because your comments helped the manager to understand there is nothing to worry, the metric was (close to) useless. Your explanation was a better report and the number could have been tossed away.

Thursday, June 14, 2012

Testing Challenge - Puzzle #6

My mind was full of riddles when I wrote up these puzzles!

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

10 philosophers meet and one decides to make a bet. He says he will put all his ten 100 euro bills in a basket. Then each of the philosophers will take one and only one 100 euro bill. He will be the last one. After he takes his bill, the basket will still have a 100 euro bill inside. If he succeeds, the other philosophers will need to pay him 100 euro each. If he doesn't succeed, he needs to give them the money. What is he going to do?

Testing Challenge - Puzzle #5

So here we go again with a puzzle that will require you to send me questions in order to solve this one. I'll start these with an easy one so you might get this even with the first question.

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

There was a road construction and a lot of people didn’t like it. After a while of constructions, people on pension started calling an old lady and complain to her about the construction, nevertheless she wasn’t part of the firm or had anything to do with them. Can you explain why this happened?

Testing Challenge - Puzzle #4

This time we will talk about trains. Some of you are more familiar with them than others which might give a helping edge, but anyone with good questioning skills will solve this.

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

Two trains are heading each others and will crash in a matter of seconds. There are no secondary tracks and the brakes don't work. How can the accident be avoided?

Testing Challenge - Puzzle #3

This is the second lateral puzzle. I got huge help from Ilari Henrik Aegerter (, James Bach (, Pekka Marjamäki ( and Michael Bolton ( I'd like to thank them for helping with the setup, clarifying a lot of questions, bringing insights and of course a lot of good time!

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

There is a 15 year old boy studying in a high school. He loves ice hockey and is the best of the team from his year. The team has been excellent in the high school championships. Recently, the dean and the teacher’s council had a meeting where they decided he is so good they must dismiss him from the team. Explain why.

Testing Challenge - Puzzle #2

This is the first (they might get a bit harder after the easy start) of the "yes/no/not relevant" kind of lateral puzzle I am publishing. More will follow. Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

In the world championships of  relay running in 2654, the Chinese team will be the last to cross over the finish line (as in, the slowest team). However, they still won. Explain why this happened.

Testing Challenge - Puzzle #1

After thinking about this for a long time, I decided I will start publishing puzzles I have made. Because I keep coming up with new ones also, most likely I will add them here every now and then.

I have not yet fully decided, but my initial idea was to have problem solving/mathematical/logical puzzles in the blog so that everyone can try to solve them here and lateral/creative puzzles only presented with the setup. If a reader would be interested to solve a puzzle of the latter kind, we could do it for example over Skype or Twitter. I am also planning to add these to the TdT Cluj-Napoca (if you don't know what that is, check out workshops, but maybe more about that later.

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

So here is the first logical one!

Continue the series (as in, replace X's with correct letters) and present your logic:
gra avar rvtXX

Monday, May 14, 2012

My Answers to 18 Testing Challenges from Santhosh Tuppad

My friend and a great tester Santhosh Tuppad (!/santhoshst) got an idea of making a testing competition. He put the questions on his blog ( and mentioned everyone can participate. I thought it would be a cool thing for practicing my thinking and seeing things from another perspective. My answers were written without too much effort on the visual side, more like a collection of thoughts. They seem to be rather lengthy too. I’d love to hear your comments on my answers so please take the time to read and reply. Here are the questions and answers:
  1. What if you click on something (A hyperlink) and to process or navigate to that webpage you need to be signed in? Currently, you are not signed in. Should you be taken to Sign up form or Sign in form? What is the better solution that you can provide?
Firstly, this answer should be included in all answers below: using passwords is an outdated way to handle authorization. Has been for years already. I would have passwords, if someone sees value in using them, for example with TV Guides and online magazines, but in no system that includes sensitive information. Now to the questions in hand!
Making a few assumptions here to get started… Let’s say the “something” would be: you choose an item to shopping cart and click “pay”. The site would require user to be logged in to continue. The first-come-to-mind option would be to have “Login with credentials here” view with an additional option “Don’t have an account yet? Click here!” for registering a new account.
If I would be somewhere in a completely different place, would be redirected to another site and asked to login, I would prefer for example to see what I am about to enter/access. In the shopping cart example I already knew that, but it’s not the same for all hyperlinks.
The question comes down to “will the webpage know if I have an account or not”. If the webpage doesn’t know whether you have an account or not, both options should be visible. If the webpage knows you have an account, login would be visible. If the webpage knows you don’t have an account, registering would be visible. Considering, the webpage has little knowledge who is actually using the computer, knowing if you have an account or not is tricky. A cookie might be present, but that could actually let a “wrong” user to login.
  1. Using “Close” naming convention to go back to the homepage is good or it should be named as “Cancel” or it is not really required because there is a “Home” link which is accessible. What are your thoughts?
I assume this in some web page because you mention “homepage”. What would you close in this case to return to homepage? Specifically, if you close something, do you need to return to homepage or would it be open in the background and you would close a popup window? If this is the case, but homepage would not be on the background, what would be there?
Commonly, I would say it’s good to have a few different options to return to homepage because people are used to navigate in different manners.
Returning back to the Close vs. Cancel. Close could be usable when there is actually something to be closed, such as a popup window. Cancel could be usable when user is for example filling up a registration form and decides he doesn’t want to complete it. A context where both could be used would be for example a Flash app appearing on the page. (A concrete example: open a car manufacturer web page, choose a car model, click “customize” which opens a Flash app over the page where you can adjust the configuration of the car. In this case, we could have buttons Close and Cancel – maybe even Back and Back to Home Page.)
  1. Logout should be placed on top right hand side? What if it is on the top left hand side or in the left hand sidebar which is menu widget like “My Profile”, “Change Password” etc. – Is it a problem or what is your thought process?
I am used to have logout on the top right, this is how most web pages work. I have asked around from people to put their finger on paper where they think certain functionality exists and “logout finger” goes on top right 100% of the times. This makes sense because many people think “logout” is a way to close the application they are using and most (GUI-based) operating systems provide closing functionality from the top right corner.
Same goes for profile etc. They tend to be on top right. I think this is good for example because people tend to look a bit on the up left (not top left, but a bit higher than center). So when focus is on the left side, it’s better to put insignificant information on the right so there is nothing extra on the concentration area. Another reason is that we look up right when we access so called “visually remembered images”, so when we want to remember something, we tend to move the eyes on top right. This, with the addition of “logout is on top right” to be almost an industry standard, speaks on behalf of keeping logout on top right. When looking right in general, we are trying to remember something instead of using our imagination to figure it out.
  1. Current design of forgot password asks for username and security answer and then sends a link to e-mail inbox to set new password. How does “security answer” increase the cost of operations? Also, what questions do you frame for security questions?
I am not sure what “increase the cost of operations” means. Nevertheless, I will think about this situation from usability and security point of view. Maybe you will later explain what the original question meant. :-) (Note: later on I understood this, but as the answer was already written, I thought it would be fun to leave it as it is.)
Let’s consider I own an account in and my username is JariLaakso. I use Amazon rarely so sometimes I need to reset my password because I want it to be unique compared to any other password I use, I want it to be long etc. I go to “forgot password” in, enter “JariLaakso” as my ID and get a prompt about the security question. As this is my account, I want to be able to remember the answer always. I would most likely choose something from my life, such as names or places.
Now let’s consider a malicious user (for example ex-girlfriend) would want to break into my Amazon account. The user would need to know how to reply to that question in order to get my password reset. This is not such a big deal in case she doesn’t have access to my e-mail. Basically, I would be screwed (in the bad sense) if she would be able to access my e-mail already, so the risk doesn’t increase here too much. There would be a problem if the site would directly allow changing the password, but when e-mail is needed in between the risks are lower.
How about if the “forgot password” is for the e-mail? Where the link would be sent when answering correctly to the question? We have found a gold vein! Ultimately, when using this “security question” pipe, we would find out the account what we need to break in order to gain access to pretty much everything else. This rises up alarming concerns. Now when we add “Internet knows everything about you” spice to the soup, we have made a dinner with 5 courses. (From here on, everything is depending how “the last line of defense” is protected. A bad design is to allow a user to change the password when answering correctly to a security question. This is because there is so much information about a general user online already. A better option would be for example via SMS, but phone numbers change etc. so it’s not without risk either. A completely new method will be needed in the near future.)
  1. If you had to design “Forgot Password” working, how would you do it and why? You are free to give different many functional designs.
I’ll start again by describing a sort of starting point. I want to do this to explain what kind of context I am thinking. In a different context, the feature could lean more towards security instead of usability. The feature exists on a web page (non-webmail). The page doesn’t store too much personal and/or sensitive information, however, identity thieves are not welcomed warmheartedly. Every user has a unique username and registration happens via a form on the page. I want to login to the site, but I have forgotten my password, so I click “Forgot Password” link. Layout and graphical part is not considered as I focus on how things would work.
There would be a text fields (Note: the information input on the text fields should not be remembered by the browser) where to write your e-mail address and username to get a “reset password” link in your e-mail. The sent e-mail would only have a link to reset the password if needed; it would not be done before clicking that link. Why? Because I want to prevent others from resetting my password. I would also restrict the amount one can reset the password consecutively. There would not be any sense to send multiple “you can reset your password from this link” e-mails to a user.
I would not add captcha because those can be circumvented and they annoy users. They might work for some registrations, but mostly irritate in this function.
In case the user doesn’t remember the e-mail address assigned to the ID or username for that site… most sites would have “security question” feature. I am not too fond of them, as described on previous answer, but I could still have a similar feature if the webpage would be for example something rather meaningless like “online TV Guide”. Even in this case, I would like answering correctly to this question to send an e-mail, but not reset the password etc. just like above. I still don’t want other people to reset my passwords or allow them to spam me from a service I am registered into.
… And just for the sake of argument, for a system which contains sensitive information, such as online bank, the abovementioned is not adequate enough. Basically, all current security systems can be cheated, but I think it’s satisfactory to demand a customer to visit an office in certain situations. This rises up so many branches of discussion, I better continue to the next question. :-)
  1. There is neither account lockout policy nor captcha for the login or security answer forms; what kind of problems do you see with the current implementation and what do you propose?
This reminds me of the online banking issues I blogged ( earlier about. :-)
Not that captcha actually increases security, but let’s say not all jerks in the neighborhood can attack your service automatically if you add a captcha check. However, like said before, captcha can be circumvented (by machine and human force) and there are examples of both online.
The biggest problem comes from so called brute-force attack. The brute-force doesn’t have to be a “stupid one”, but can be firstly based on common password lists etc. It doesn’t even matter, because if you allow a user to guess all the way, they will figure out all usernames and passwords from the database(s).
What the question doesn’t mention is if there is a waiting algorithm between login attempts and if there is one, how will it function. For example, between 1st and 2nd login would be a mandatory 5 sec pause. Between 2nd and 3rd login would be a 25 sec pause. I guess you got the algorithm. This could be usable for a system where it is not of interest to let someone find out the usernames and passwords. However, it would also cause some usability issues, so I would not recommend it for “everyday systems”.
Not having a security answer form is not a problem itself. It becomes a problem if there is no other way to obtain username or password reset.
My suggestion would depend heavily on how secure the system should be. For example, if has to be really secure, I could also consider limiting access to certain IP ranges, having a certificate on the accessing machine, etc.
  1. Well, it is about context and there are no best practices in general. What are your thoughts on usage of captcha? Where should they be used and why?
Oh so I wrote 3 pages of text to arrive here and see I have replied to some of the questions already above. Great!
Captcha is good if you want to limit the amount of potential hackers. It’s not good if you think it prevents anything else. In most cases, I hate captcha. I have actually seen some interesting research articles where computers have gotten better results than humans with captcha. :-)
If it would be up to me, I would not use captcha pretty much anywhere. The benefit is smaller than the loss of usability. If nothing else, I would come up with a completely new way to “verify” the user is not a machine.
  1. If you are the solution architect for a retail website which has to be developed; what kind of questions would you ask with respect to “Scalability” purpose with respect to “Technology” being used for the website?
Do you mean hardware with technology? If we include also programming languages, what else do we add? What do other retail websites use? Why? Why not something else? (For example, Facebook goes with LAMP and that seems to work for them. Why? How about Amazon and eBay or maybe the local shop in my town?)
Things to consider about: user amount growth, user amount growth in different countries/continents, what kind of requests are sent from different actions, how long their processing takes, what loads the servers most, how to handle load balancing, what kind of user profiles we will have, what are most common functions and pages (for caching and optimization), etc.
Nevertheless (especially early) users will abandon the system if the response times are not magnificent, context will matter a lot. If your purpose is to sell locally and you expect 99% of traffic from a certain city, you might want to scale for that. However, remember also to read this carefully.
More questions? Ok here are a few: Are we talking about administrative scalability, geographical scalability, load scalability or functional scalability? Maybe all? Maybe a combination? How will we scale out? Do we need to scale up? How will the database affect on this? How about system design?
  1. How do you think “Deactivate Account” should work functionally keeping in mind about “Usability” & “Security” quality criteria?
Hopefully by deactivating the desired account from a user. :-)
There are 2 common good ways to handle deactivation:
1) Allow it for a user who has an active session
a. Benefit: Nobody would deactivate your account if they can’t use it.
b. Detriment: It’s more than easy to forget an active session for a computer which can be accessed by other people.
c. Solution: Either the confirmation as I describe next or canceling the deactivation if trying to login again within a certain amount of time.
2) Allow it for a user via a confirmation (e-mail for example)
a. Benefit: It’s not possible to deactivate someone’s account without confirming it.
b. Detriment: People tend to dislike “extra” confirmations.
c. Solution: (Considering this would not be a common use case someone anyway does often, it’s not a big problem, imo.) Explain the user clearly why the confirmation is needed. Ease up the confirmation process for example with a simple clickable link in an e-mail. Include still a “remorse time” during which the user would reactivate the account when signing in again (and maybe receiving a “do you want to keep this account active” question).
All in all, in my opinion, deactivating an account should not remove the data of that person from a system. The information might be needed/usable in the future.
  1. For every registration, there is an e-mail sent with activation link. Once this activation link is used account is activated and a “Welcome E-mail” is sent to the end-users e-mail inbox. Now, list down the test ideas which could result in spamming if specific tests are not done.
I assume you are asking “which could result in spamming if specific code is not done” or something like that as testing won’t prevent anything. :-) So let’s start this from design point of view.
Prevent spamming a single user: The system should allow only 1 Activation Link e-mail to be sent to a specific e-mail address. The system should allow only 1 Welcome E-mail to be sent to a specific e-mail address. Exception: If a user deactivates/deletes the account, he should be allowed to register again with the same e-mail.
Prevent spamming multiple users: The system should have a limitation for incoming requests so a malicious user can’t register lots of accounts automatically.
Next step is how to test these. For single user point of view, you would test registering same account a few times and checking if your e-mail received more than 1 e-mail (ideally we would not print “this e-mail address already exists in the system” because it allows malicious users to gain information what e-mail addresses are used in the system). You would also test if it’s possible to register again after deactivation/deleting the account.
Spamming multiple users would require tests such as using many computers from different IP ranges, multiple computers from a small IP range (even from a single IP, like behind a NAT), single computer and to see if there is a delay in consecutive registrations.
Slightly out of the provided context, but closely related: When we are talking about registrations and such, we would also need to consider for example XSS, CSRF and SQL Injection tests. Not as a direct consequence from the Welcome E-mail and the link, but when requesting and storing user data, it would be good if the system would prevent giving those to malicious users.
  1. In what different ways can you use “Tamper Data” add-on from “Mozilla Firefox” web browser? If you have not used it till date then how about exploring it and using it; then you can share your experience here.
Phew, finally a shorter answer! I am assuming the question is more about what different tests I do with Tamper Data.
I use it mostly for editing POST parameters, but it’s also usable to tracing HTTP requests/responses. There is a time provided and it can be used for example to see if some actions are causing more load on a server when a single user accesses the service. An example of a check that *might* be useful: measure response times when logging in with different usernames, longer wait could imply the username was found if the code firstly checks if the user exists and then compares the password.
I use Tamper Data also to view headers and sometimes to modify them. Cookie manipulation can be done, however, there are other tools as well for this purpose.
  1. Application is being launched in a month from now and management has decided not to test for “Usability” or there are no testers in the team who can perform it and it is a web application. What is your take on this?
Firstly, I would say testers are not responsible for managing a project/product or making release decisions. Testing is done for obtaining information for decision-making. I could advocate what consequences this could have, but the decision to launch and bear the risks is not mine to carry.
That being said, I would question the question: What means to decide not to test for usability? Why the testers can’t perform usability testing? Are there any testers in the team? Who else understands usability? Was it considered in the design? Why it matters if the application is a web app? Why it matters if the application will be launched in the future if the decision is made no usability testing will be done?
In a real-life situation, in my context, working with a customer I am currently working in a project I am currently working, I would of course do something different. If the advocating would not result in convincing the management how much needed the usability testing is, I would:
1) Try to understand the decision and either agree or find a new way to persuade them (if still rejected, one could see it’s time to back off)
2) Use a short while of my own time to collect usability issues, thus either gaining confidence it’s in a good shape or to show what are the major problems I see
3) Talk with the devs or their lead(s) to see what actually could be used, whose responsibility the decision to fix would be, etc.
If the team could not handle usability testing, but the management would like it to be done, I would either involve myself on it, find other people from the company, get a third party involved (if allowed)… In this case, when the team can’t handle the testing, I see it so that I am not part of the team. This would greatly limit my options, however, as I already wrote, a lot of things can be done. The challenge would be to see what usability issues could be fixed before the release considering there might be still features to be implemented and other bugs to be fixed.
  1. Share your experience wherein; the developer did not accept security vulnerability and you did great bug advocacy to prove that it is a bug and finally it was fixed. Even if it was not fixed then please let me know about what was the bug and how did you do bug advocacy without revealing the application / company details.
I can’t recall a case where my security bugs would have been marked “invalid” or something else suggesting they are “not accepted” by a developer. I remember cases where I have defended other testers’ bugs, but I can’t recall this happening for myself. I tend to write the risk and other relevant information on security bugs because it saves time on a long run. (So I have been told, lol.)
As a guideline, if you don’t know the dev and/or have history to know how he might understand your bugs, it’s a good idea to include your deductions and claims already in the initial bug report. Describing what problems a bug might cause, how they can be abused and what other kind of risks there are usually helps the dev to make the correct decision.
I also would like to note that in 99% of my context the devs don’t mark bugs “invalid” or “won’t fix” before a meeting is held with more people.
  1. What do you have in your tester’s toolkit? Name at least 10 such tools or utilities. Please do not list like QTP, LoadRunner, SilkTest and such things. Something which you have discovered (Example: Process Explorer from SysInternals) on your own or from your colleague. If you can also share how you use it then it would be fantastic.
Considering this is a blog post that has been in my queue for a long time, I will try to just summarize a few things here. The way how the question is put makes me believe the tools you are referring to should not be test tools, but other tools which can be used in testing. If I misunderstood, please correct me. List is in order how they came to my mind.
#1 tool coming into my mind is FreeMind It is a great free tool for making mindmaps.
#2 tool has to be Excel. Excel is fantastic for keeping notes, making reports, collecting data, comparing data etc.
#3 could be Firebug & Web Developer together because I use them so much. I use then for example to manipulate hidden elements, modify JS from a page, change input validation and enter all kind of values to forms. I put them in the same category for the fun of it, no particular logic.
#4 is Twitter as I use it sometimes to find out what people say about the companies/products we are testing. Twitter is not the only tool for this, but a really good one.
#5 shall be Paint as it’s a very lightweight application for simple picture modification. I could maybe write a blog post later about using pictures in web testing.
#6 must be something for test/check automation purposes: SVN. It just makes your life so much easier. SVN combined with a CI system is a really good combination for many projects.
#7 place goes for … paper and pencil! I love drawing pictures, writing fast notes, storing words/ideas etc. with paper and pencil. I have been thinking to buy a tablet of some kind for this, but not yet decided on a product.
#8 seems to be Total Commander. This is a fantastic lightweight application for Windows users who want to compare/synchronize folders, copy/delete files etc.
#9 I am not pointing to any single application, but applications that capture “video” of what you do with your computer as sometimes really awesome to track down ways to reproduce a bug, show what really happened etc.
#10 is YSlow which I haven’t used in a while. It’s good for measuring performance of different functionalities of a web page easily while you do other testing. (By the way, I was really tempted to put “my brain” as the last one. J)
  1. Let us say there is a commenting feature for the blog post; there are 100 comments currently. How would you load / render every comment. Is it one by one or all 100 at once? Justify.
A few things to consider: how many people read your blog, what devices people use to read the blog, what kind of internet connection they have, do you want to have a compromise solution for everyone or optimize for a certain group, how long the comments are and do they contain other things than plain text too. There are more variables, but these seem to be the ones directing this kind of decision the most.
Now to the loading itself. If you choose to load one-by-one, you might face a situation where the server is getting a lot of requests just for the sake of loading text (if that is the case). That could lead to performance issues with many concurrent users. If you would like to be really clever, you could let the user decide this by clicking “how many comments you want to load at once” selection where you could have a few different options. Default being, for example, 10.
Basically, any decent server should be able to handle loading 100 messages (depending on size of course) really easy, but as we don’t know any details of the environment etc., I need to abstain from giving a clear answer on this one.
  1. Have you ever done check automation using open-source tools? How did you identify the checks and what value did you add by automating them? Explain.
I’ve done GUI automation tests/checks for a few reasons. The reasons and analysis are too much to write here at the moment. Maybe I’ll write a blog post about it. Internet already has great writings from this subject and I’d like to recommend “Test automation snake oil” for a starter. Main idea is that I don’t have anything against or pro test automation without an analysis. The answer is as multifold as if the term “automation” would be replaced with “manual”. (Note: I’ve used for example Selenium for check automation. One can get pretty rapid feedback for smoke tests with it when combined with a continuous integration server.)
  1. What kind of information do you gather before starting to test a software? (Example: Purpose of this application)
Depends a lot of the application, platform, test “phase” (security, performance etc.) and many other factors, like the customer. Let’s say we would have a web site to test and our job is to see how the functionalities work etc. I would most likely start with CIDTESTD mnemonic. Not because it’s the best one, but it’s a good starting point if you don’t have anything else to compare. CIDTESTD includes information about who customers are, manuals, documents, history, developers, test team, equipment and tools, schedule, test items and deliverables. That is a pretty comprehensive list to start with, but not everything needs to be specified. However, it is usually better the more you know.
In a more general manner, I feel it’s important to understand who uses the software, why they use it, what is my mission (what is expected from me), how much time I have, what kind of reporting is needed etc. I could also want to know if there are legal requirements for using/testing the software, restrictions on what systems it works/should work with, severe impacts on society due a bug (for example a nuclear weapon launch system) and for example if the software is working together with other systems such as banking software.
  1. How do you achieve data coverage (Inputs coverage) for a specific form with text fields like mobile number, date of birth etc? There are so many character sets and how do you achieve the coverage? You could share your past experience. If not any then you can talk about how it could be done.
Firstly, I would note the coverage includes also outputs, not only inputs. Secondly, I would like to note I have used a lot of “checklists” for this and I review them with colleagues to see if someone comes up with new test ideas. That is great fun always! Thirdly, I must stress that this is somewhat case-by-case basis for example because with web services one can do so many different things with inputs.
One common way for me is to use automation for storing + giving variance for inputs. Second is that I tend to categorize (XSS, SQL injection, empty, too small, too big, way too big etc.) the tests and use sort of “equivalence classes” in the tests, as in I make assumptions “if X and Y passes, the class they represent is less-likely to be risky”.
I always add some sort of random tests in those equivalence classes if making the tests is cheap. For example, with a web service, you could leave your test computer send different kinds of inputs overnight and check fast in the morning if any input caused strange behavior/errors.

Overcoming Illusions on Testing – Part III

This entry (first part and second part has been on hold for quite a while; mostly because I have been focusing on other things. (I actually changed even the topic because I thought to write about the future later, in the future!) As we are building the Romanian testing community, I have actual work to do and I’m now a father, I didn’t feel motivated enough to keep on writing. The motivation is back and I’m hoping to start adding more posts in the near future. (If you wonder why I didn’t say “I didn’t have enough time”, it’s simply because it’s a lie. Time is a matter if prioritization. We have it, but not for everything.)

I this post, I will be referring to RST quite many times. This is because I like to use examples in my writing and there were fantastic examples in the course. I won’t, however, tell those examples so you don’t lose anything from the experience.

Without further ado, let’s dig in to the topic at hands. So we have seen detailed pre-scripting doesn’t work too well because a) it kills creativity, b) it doesn’t tolerate change too well, and c) people are just not that good in writing detailed instructions. Actually it does work. Just that it doesn’t work for excellent testing, but it works great for invoicing customers and setting up a fake quality assessment done by “even a stranger from the streets”. The latter means the turnover for testers doesn’t matter for the company.

RST has a few exercises around this topic. One was from the tester’s point of view and one from who was trying to write the script for the tester. The exercises are designed to fail with simple answers, which is fantastic from my point of view. So what to do in this case? Firstly, don’t start writing very detailed test scripts for testers. Secondly, to help improve your (data) coverage, you can always ask help from others. Great testers love to help other testers; and in most organizations I have seen, the developers are keen on working with this.

I mentioned in the previous posts that terminology seems challenging for testers. RST has a lot of keywords you should understand. Otherwise it’s hard to follow the discussions. At least James explained all the terms he was using and based on what I have seen for example here, I am sure Michael is doing the same. The thing with terminology is that you don’t need a word listing for this. What you need to do is to start reading and talking with others and find a common language.

There was an observation earlier that women did better because they had more sense of context in their replies. I understand this easily from a Finnish point of view where we are told that women analyze things while men drink beer, eat sausages and find The Best Way. I could easily see myself fitting in that form still some years ago. How I got out from it? I think it was just a phase actually, but at the end it’s about what kind of people you gather around yourself and what do you do on your free time. I am advocating on reading and writing, but remember that you can get ideas from pretty much anywhere if you keep your eyes/ears open.

Yes, I covered only 3 of the 5 things I mentioned on earlier posts. Why? Because I want you to tell me what you think about them!