Metrics - reply to Mike Talks

This post was originally supposed to be a comment in Mike Talks' blog (http://testsheepnz.blogspot.co.nz/2012/02/are-we-there-yet-metrics-of-destination.html) but it became a bit lengthy so I decided to post it here...



Hi Mike,

There are great posts about metrics already available so I don't want to dig in too deeply in the subject. Nonetheless, I would like to comment your interesting blog post!

I like the start with "are we there yet"! I wish more bloggers would make such associations between "real life" and software projects / testing.

When you mentioned you are estimating how long testing will take, do you mean a case where you and the project manager have a long history together and you both know what kind of testing you will do in that time? I am asking because testing is never done and I don't think that is too secure way for a PM to create a testing budget. Will he/you have the code done at this part? Do you (as in, you with the PM) have a lot of history with the same product?

"I know some people hate recording hours on a project. I personally think it's vital, because it helps a manager to determine spend on a project."

Yes, in some (possibly even most) cases this is very good. From testing point of view, I am usually interested to know how much of time (e.g. 90 min sessions) is used to test certain features/functionalities. For example, if there are no bugs found in a week, I could start asking questions like why we are not finding bugs and could we use testing somewhere else more effectively. (If the goal would be to raise bugs.)

"What about the number of test requirements tested and the number passed? Personally I like this metric, as it gives a good feel of how many paths and features we've tested, and I do think it's useful to keep track of this (as long as it's relatively painless)."

In many cases this can be a good thing to track. There can be for example legal requirements that need to "pass". However, like you point out in your text, it can be also very misleading. Even if 100% of the requirements pass, it doesn't mean a product is good or doesn't have critical bugs.

"Another metric I've seen is simple number of test cases run, and number passed. ... However it's more than likely a lot easier to track this number than the number of requirements if you're running manual test scripts which are just written up in Word (unless you're an Excel wizard)."

This is easy to track, but I don't see it telling really anything interesting. What would be interesting to know is how you use this metric. Is it used to raise up questions or make decisions (inquiry or control)?

One thing I want to stress. Passed tests can be dangerous to follow because they don't tell us much (if any) of the product. They might even give false confidence of the stability/quality of the product to management.

"What about measuring defects encountered for each build?"

I like to see how these change over time and what questions the change might raise up. Just like your text explains a situation where earlier bugs were blocking further testing.

When it comes to regularity of metrics, I think automated/scripted systems would be great to give numbers in many cases. Numbers can raise up good questions, but I would prefer not to use too much testing time to collect them. Depending for example on project size of course.

"I had to keep daily progress number updates. After day 5 I had still not finished script 1. In fact after day 8 I still wasn't done on script 1. On day 10 all 3 of my scripts were completed."

Maybe the progress should have been communicated differently? Sounds like there was "too big piece" to be reported for 1 progress step. I don't know what the script involved, but here is an example for a UI check:
- Needed (navigation) controls are coded
- The flow between controls is automated
- The assertions/verifications/requirements of each successful step are automated
- Minor changes to produce script #2
- Minor changes to produce script #3
- Testing, fixing and re-factoring the scripts

As we saw, because your comments helped the manager to understand there is nothing to worry, the metric was (close to) useless. Your explanation was a better report and the number could have been tossed away.

Testing Challenge - Puzzle #6

My mind was full of riddles when I wrote up these puzzles!

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

10 philosophers meet and one decides to make a bet. He says he will put all his ten 100 euro bills in a basket. Then each of the philosophers will take one and only one 100 euro bill. He will be the last one. After he takes his bill, the basket will still have a 100 euro bill inside. If he succeeds, the other philosophers will need to pay him 100 euro each. If he doesn't succeed, he needs to give them the money. What is he going to do?

Testing Challenge - Puzzle #5

So here we go again with a puzzle that will require you to send me questions in order to solve this one. I'll start these with an easy one so you might get this even with the first question.

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

There was a road construction and a lot of people didn’t like it. After a while of constructions, people on pension started calling an old lady and complain to her about the construction, nevertheless she wasn’t part of the firm or had anything to do with them. Can you explain why this happened?

Testing Challenge - Puzzle #4

This time we will talk about trains. Some of you are more familiar with them than others which might give a helping edge, but anyone with good questioning skills will solve this.

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

Two trains are heading each others and will crash in a matter of seconds. There are no secondary tracks and the brakes don't work. How can the accident be avoided?

Testing Challenge - Puzzle #3

This is the second lateral puzzle. I got huge help from Ilari Henrik Aegerter (www.ilari.com/blog/), James Bach (www.satisfice.com/blog/), Pekka Marjamäki (www.how-do-i-test.blogspot.com/) and Michael Bolton (www.developsense.com/blog/). I'd like to thank them for helping with the setup, clarifying a lot of questions, bringing insights and of course a lot of good time!

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

There is a 15 year old boy studying in a high school. He loves ice hockey and is the best of the team from his year. The team has been excellent in the high school championships. Recently, the dean and the teacher’s council had a meeting where they decided he is so good they must dismiss him from the team. Explain why.

Testing Challenge - Puzzle #2

This is the first (they might get a bit harder after the easy start) of the "yes/no/not relevant" kind of lateral puzzle I am publishing. More will follow. Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

In the world championships of  relay running in 2654, the Chinese team will be the last to cross over the finish line (as in, the slowest team). However, they still won. Explain why this happened.

Testing Challenge - Puzzle #1

After thinking about this for a long time, I decided I will start publishing puzzles I have made. Because I keep coming up with new ones also, most likely I will add them here every now and then.

I have not yet fully decided, but my initial idea was to have problem solving/mathematical/logical puzzles in the blog so that everyone can try to solve them here and lateral/creative puzzles only presented with the setup. If a reader would be interested to solve a puzzle of the latter kind, we could do it for example over Skype or Twitter. I am also planning to add these to the TdT Cluj-Napoca (if you don't know what that is, check out http://tabaradetestare.ro/) workshops, but maybe more about that later.

Please remember, I don't want you to ruin the puzzle for anyone in the comments section so if you want to solve this, send me an e-mail or ping over Twitter so we can sort out the details. Thanks!

So here is the first logical one!

Continue the series (as in, replace X's with correct letters) and present your logic:
gra avar rvtXX

My Answers to 18 Testing Challenges from Santhosh Tuppad

My friend and a great tester Santhosh Tuppad (https://twitter.com/#!/santhoshst) got an idea of making a testing competition. He put the questions on his blog (http://www.tuppad.com/blog/) and mentioned everyone can participate. I thought it would be a cool thing for practicing my thinking and seeing things from another perspective. My answers were written without too much effort on the visual side, more like a collection of thoughts. They seem to be rather lengthy too. I’d love to hear your comments on my answers so please take the time to read and reply. Here are the questions and answers:

1. What if you click on something (A hyperlink) and to process or navigate to that webpage you need to be signed in? Currently, you are not signed in. Should you be taken to Sign up form or Sign in form? What is the better solution that you can provide?

Firstly, this answer should be included in all answers below: using passwords is an outdated way to handle authorization. Has been for years already. I would have passwords, if someone sees value in using them, for example with TV Guides and online magazines, but in no system that includes sensitive information. Now to the questions in hand!

Making a few assumptions here to get started… Let’s say the “something” would be: you choose an item to shopping cart and click “pay”. The site would require user to be logged in to continue. The first-come-to-mind option would be to have “Login with credentials here” view with an additional option “Don’t have an account yet? Click here!” for registering a new account.

If I would be somewhere in a completely different place, would be redirected to another site and asked to login, I would prefer for example to see what I am about to enter/access. In the shopping cart example I already knew that, but it’s not the same for all hyperlinks.

The question comes down to “will the webpage know if I have an account or not”. If the webpage doesn’t know whether you have an account or not, both options should be visible. If the webpage knows you have an account, login would be visible. If the webpage knows you don’t have an account, registering would be visible. Considering, the webpage has little knowledge who is actually using the computer, knowing if you have an account or not is tricky. A cookie might be present, but that could actually let a “wrong” user to login.

2. Using “Close” naming convention to go back to the homepage is good or it should be named as “Cancel” or it is not really required because there is a “Home” link which is accessible. What are your thoughts?

I assume this in some web page because you mention “homepage”. What would you close in this case to return to homepage? Specifically, if you close something, do you need to return to homepage or would it be open in the background and you would close a popup window? If this is the case, but homepage would not be on the background, what would be there?

Commonly, I would say it’s good to have a few different options to return to homepage because people are used to navigate in different manners.

Returning back to the Close vs. Cancel. Close could be usable when there is actually something to be closed, such as a popup window. Cancel could be usable when user is for example filling up a registration form and decides he doesn’t want to complete it. A context where both could be used would be for example a Flash app appearing on the page. (A concrete example: open a car manufacturer web page, choose a car model, click “customize” which opens a Flash app over the page where you can adjust the configuration of the car. In this case, we could have buttons Close and Cancel – maybe even Back and Back to Home Page.)

3. Logout should be placed on top right hand side? What if it is on the top left hand side or in the left hand sidebar which is menu widget like “My Profile”, “Change Password” etc. – Is it a problem or what is your thought process?

I am used to have logout on the top right, this is how most web pages work. I have asked around from people to put their finger on paper where they think certain functionality exists and “logout finger” goes on top right 100% of the times. This makes sense because many people think “logout” is a way to close the application they are using and most (GUI-based) operating systems provide closing functionality from the top right corner.

Same goes for profile etc. They tend to be on top right. I think this is good for example because people tend to look a bit on the up left (not top left, but a bit higher than center). So when focus is on the left side, it’s better to put insignificant information on the right so there is nothing extra on the concentration area. Another reason is that we look up right when we access so called “visually remembered images”, so when we want to remember something, we tend to move the eyes on top right. This, with the addition of “logout is on top right” to be almost an industry standard, speaks on behalf of keeping logout on top right. When looking right in general, we are trying to remember something instead of using our imagination to figure it out.

4. Current design of forgot password asks for username and security answer and then sends a link to e-mail inbox to set new password. How does “security answer” increase the cost of operations? Also, what questions do you frame for security questions?

I am not sure what “increase the cost of operations” means. Nevertheless, I will think about this situation from usability and security point of view. Maybe you will later explain what the original question meant. :-) (Note: later on I understood this, but as the answer was already written, I thought it would be fun to leave it as it is.)

Let’s consider I own an account in Amazon.com and my username is JariLaakso. I use Amazon rarely so sometimes I need to reset my password because I want it to be unique compared to any other password I use, I want it to be long etc. I go to “forgot password” in Amazon.com, enter “JariLaakso” as my ID and get a prompt about the security question. As this is my account, I want to be able to remember the answer always. I would most likely choose something from my life, such as names or places.

Now let’s consider a malicious user (for example ex-girlfriend) would want to break into my Amazon account. The user would need to know how to reply to that question in order to get my password reset. This is not such a big deal in case she doesn’t have access to my e-mail. Basically, I would be screwed (in the bad sense) if she would be able to access my e-mail already, so the risk doesn’t increase here too much. There would be a problem if the site would directly allow changing the password, but when e-mail is needed in between the risks are lower.

How about if the “forgot password” is for the e-mail? Where the link would be sent when answering correctly to the question? We have found a gold vein! Ultimately, when using this “security question” pipe, we would find out the account what we need to break in order to gain access to pretty much everything else. This rises up alarming concerns. Now when we add “Internet knows everything about you” spice to the soup, we have made a dinner with 5 courses. (From here on, everything is depending how “the last line of defense” is protected. A bad design is to allow a user to change the password when answering correctly to a security question. This is because there is so much information about a general user online already. A better option would be for example via SMS, but phone numbers change etc. so it’s not without risk either. A completely new method will be needed in the near future.)

5. If you had to design “Forgot Password” working, how would you do it and why? You are free to give different many functional designs.

I’ll start again by describing a sort of starting point. I want to do this to explain what kind of context I am thinking. In a different context, the feature could lean more towards security instead of usability. The feature exists on a web page (non-webmail). The page doesn’t store too much personal and/or sensitive information, however, identity thieves are not welcomed warmheartedly. Every user has a unique username and registration happens via a form on the page. I want to login to the site, but I have forgotten my password, so I click “Forgot Password” link. Layout and graphical part is not considered as I focus on how things would work.

There would be a text fields (Note: the information input on the text fields should not be remembered by the browser) where to write your e-mail address and username to get a “reset password” link in your e-mail. The sent e-mail would only have a link to reset the password if needed; it would not be done before clicking that link. Why? Because I want to prevent others from resetting my password. I would also restrict the amount one can reset the password consecutively. There would not be any sense to send multiple “you can reset your password from this link” e-mails to a user.

I would not add captcha because those can be circumvented and they annoy users. They might work for some registrations, but mostly irritate in this function.

In case the user doesn’t remember the e-mail address assigned to the ID or username for that site… most sites would have “security question” feature. I am not too fond of them, as described on previous answer, but I could still have a similar feature if the webpage would be for example something rather meaningless like “online TV Guide”. Even in this case, I would like answering correctly to this question to send an e-mail, but not reset the password etc. just like above. I still don’t want other people to reset my passwords or allow them to spam me from a service I am registered into.

… And just for the sake of argument, for a system which contains sensitive information, such as online bank, the abovementioned is not adequate enough. Basically, all current security systems can be cheated, but I think it’s satisfactory to demand a customer to visit an office in certain situations. This rises up so many branches of discussion, I better continue to the next question. :-)

6. There is neither account lockout policy nor captcha for the login or security answer forms; what kind of problems do you see with the current implementation and what do you propose?

This reminds me of the online banking issues I blogged (http://jarilaakso.blogspot.com/2012/02/internet-banking-experiences-from.html) earlier about. :-)

Not that captcha actually increases security, but let’s say not all jerks in the neighborhood can attack your service automatically if you add a captcha check. However, like said before, captcha can be circumvented (by machine and human force) and there are examples of both online.

The biggest problem comes from so called brute-force attack. The brute-force doesn’t have to be a “stupid one”, but can be firstly based on common password lists etc. It doesn’t even matter, because if you allow a user to guess all the way, they will figure out all usernames and passwords from the database(s).

What the question doesn’t mention is if there is a waiting algorithm between login attempts and if there is one, how will it function. For example, between 1^st and 2^nd login would be a mandatory 5 sec pause. Between 2^nd and 3^rd login would be a 25 sec pause. I guess you got the algorithm. This could be usable for a system where it is not of interest to let someone find out the usernames and passwords. However, it would also cause some usability issues, so I would not recommend it for “everyday systems”.

Not having a security answer form is not a problem itself. It becomes a problem if there is no other way to obtain username or password reset.

My suggestion would depend heavily on how secure the system should be. For example, if has to be really secure, I could also consider limiting access to certain IP ranges, having a certificate on the accessing machine, etc.

7. Well, it is about context and there are no best practices in general. What are your thoughts on usage of captcha? Where should they be used and why?

Oh so I wrote 3 pages of text to arrive here and see I have replied to some of the questions already above. Great!

Captcha is good if you want to limit the amount of potential hackers. It’s not good if you think it prevents anything else. In most cases, I hate captcha. I have actually seen some interesting research articles where computers have gotten better results than humans with captcha. :-)

If it would be up to me, I would not use captcha pretty much anywhere. The benefit is smaller than the loss of usability. If nothing else, I would come up with a completely new way to “verify” the user is not a machine.

8. If you are the solution architect for a retail website which has to be developed; what kind of questions would you ask with respect to “Scalability” purpose with respect to “Technology” being used for the website?

Do you mean hardware with technology? If we include also programming languages, what else do we add? What do other retail websites use? Why? Why not something else? (For example, Facebook goes with LAMP and that seems to work for them. Why? How about Amazon and eBay or maybe the local shop in my town?)

Things to consider about: user amount growth, user amount growth in different countries/continents, what kind of requests are sent from different actions, how long their processing takes, what loads the servers most, how to handle load balancing, what kind of user profiles we will have, what are most common functions and pages (for caching and optimization), etc.

Nevertheless (especially early) users will abandon the system if the response times are not magnificent, context will matter a lot. If your purpose is to sell locally and you expect 99% of traffic from a certain city, you might want to scale for that. However, remember also to read this http://www.zdnet.com/blog/foremski/report-51-of-web-site-traffic-is-non-human-and-mostly-malicious/2201 carefully.

More questions? Ok here are a few: Are we talking about administrative scalability, geographical scalability, load scalability or functional scalability? Maybe all? Maybe a combination? How will we scale out? Do we need to scale up? How will the database affect on this? How about system design?

9. How do you think “Deactivate Account” should work functionally keeping in mind about “Usability” & “Security” quality criteria?

Hopefully by deactivating the desired account from a user. :-)

There are 2 common good ways to handle deactivation:

1) Allow it for a user who has an active session

a. Benefit: Nobody would deactivate your account if they can’t use it.

b. Detriment: It’s more than easy to forget an active session for a computer which can be accessed by other people.

c. Solution: Either the confirmation as I describe next or canceling the deactivation if trying to login again within a certain amount of time.

2) Allow it for a user via a confirmation (e-mail for example)

a. Benefit: It’s not possible to deactivate someone’s account without confirming it.

b. Detriment: People tend to dislike “extra” confirmations.

c. Solution: (Considering this would not be a common use case someone anyway does often, it’s not a big problem, imo.) Explain the user clearly why the confirmation is needed. Ease up the confirmation process for example with a simple clickable link in an e-mail. Include still a “remorse time” during which the user would reactivate the account when signing in again (and maybe receiving a “do you want to keep this account active” question).

All in all, in my opinion, deactivating an account should not remove the data of that person from a system. The information might be needed/usable in the future.

10. For every registration, there is an e-mail sent with activation link. Once this activation link is used account is activated and a “Welcome E-mail” is sent to the end-users e-mail inbox. Now, list down the test ideas which could result in spamming if specific tests are not done.

I assume you are asking “which could result in spamming if specific code is not done” or something like that as testing won’t prevent anything. :-) So let’s start this from design point of view.

Prevent spamming a single user: The system should allow only 1 Activation Link e-mail to be sent to a specific e-mail address. The system should allow only 1 Welcome E-mail to be sent to a specific e-mail address. Exception: If a user deactivates/deletes the account, he should be allowed to register again with the same e-mail.

Prevent spamming multiple users: The system should have a limitation for incoming requests so a malicious user can’t register lots of accounts automatically.

Next step is how to test these. For single user point of view, you would test registering same account a few times and checking if your e-mail received more than 1 e-mail (ideally we would not print “this e-mail address already exists in the system” because it allows malicious users to gain information what e-mail addresses are used in the system). You would also test if it’s possible to register again after deactivation/deleting the account.

Spamming multiple users would require tests such as using many computers from different IP ranges, multiple computers from a small IP range (even from a single IP, like behind a NAT), single computer and to see if there is a delay in consecutive registrations.

Slightly out of the provided context, but closely related: When we are talking about registrations and such, we would also need to consider for example XSS, CSRF and SQL Injection tests. Not as a direct consequence from the Welcome E-mail and the link, but when requesting and storing user data, it would be good if the system would prevent giving those to malicious users.

11. In what different ways can you use “Tamper Data” add-on from “Mozilla Firefox” web browser? If you have not used it till date then how about exploring it and using it; then you can share your experience here.

Phew, finally a shorter answer! I am assuming the question is more about what different tests I do with Tamper Data.

I use it mostly for editing POST parameters, but it’s also usable to tracing HTTP requests/responses. There is a time provided and it can be used for example to see if some actions are causing more load on a server when a single user accesses the service. An example of a check that *might* be useful: measure response times when logging in with different usernames, longer wait could imply the username was found if the code firstly checks if the user exists and then compares the password.

I use Tamper Data also to view headers and sometimes to modify them. Cookie manipulation can be done, however, there are other tools as well for this purpose.

12. Application is being launched in a month from now and management has decided not to test for “Usability” or there are no testers in the team who can perform it and it is a web application. What is your take on this?

Firstly, I would say testers are not responsible for managing a project/product or making release decisions. Testing is done for obtaining information for decision-making. I could advocate what consequences this could have, but the decision to launch and bear the risks is not mine to carry.

That being said, I would question the question: What means to decide not to test for usability? Why the testers can’t perform usability testing? Are there any testers in the team? Who else understands usability? Was it considered in the design? Why it matters if the application is a web app? Why it matters if the application will be launched in the future if the decision is made no usability testing will be done?

In a real-life situation, in my context, working with a customer I am currently working in a project I am currently working, I would of course do something different. If the advocating would not result in convincing the management how much needed the usability testing is, I would:

1) Try to understand the decision and either agree or find a new way to persuade them (if still rejected, one could see it’s time to back off)

2) Use a short while of my own time to collect usability issues, thus either gaining confidence it’s in a good shape or to show what are the major problems I see

3) Talk with the devs or their lead(s) to see what actually could be used, whose responsibility the decision to fix would be, etc.

If the team could not handle usability testing, but the management would like it to be done, I would either involve myself on it, find other people from the company, get a third party involved (if allowed)… In this case, when the team can’t handle the testing, I see it so that I am not part of the team. This would greatly limit my options, however, as I already wrote, a lot of things can be done. The challenge would be to see what usability issues could be fixed before the release considering there might be still features to be implemented and other bugs to be fixed.

13. Share your experience wherein; the developer did not accept security vulnerability and you did great bug advocacy to prove that it is a bug and finally it was fixed. Even if it was not fixed then please let me know about what was the bug and how did you do bug advocacy without revealing the application / company details.

I can’t recall a case where my security bugs would have been marked “invalid” or something else suggesting they are “not accepted” by a developer. I remember cases where I have defended other testers’ bugs, but I can’t recall this happening for myself. I tend to write the risk and other relevant information on security bugs because it saves time on a long run. (So I have been told, lol.)

As a guideline, if you don’t know the dev and/or have history to know how he might understand your bugs, it’s a good idea to include your deductions and claims already in the initial bug report. Describing what problems a bug might cause, how they can be abused and what other kind of risks there are usually helps the dev to make the correct decision.

I also would like to note that in 99% of my context the devs don’t mark bugs “invalid” or “won’t fix” before a meeting is held with more people.

14. What do you have in your tester’s toolkit? Name at least 10 such tools or utilities. Please do not list like QTP, LoadRunner, SilkTest and such things. Something which you have discovered (Example: Process Explorer from SysInternals) on your own or from your colleague. If you can also share how you use it then it would be fantastic.

Considering this is a blog post that has been in my queue for a long time, I will try to just summarize a few things here. The way how the question is put makes me believe the tools you are referring to should not be test tools, but other tools which can be used in testing. If I misunderstood, please correct me. List is in order how they came to my mind.

#1 tool coming into my mind is FreeMind http://freemind.sourceforge.net/wiki/index.php/Download. It is a great free tool for making mindmaps.

#2 tool has to be Excel. Excel is fantastic for keeping notes, making reports, collecting data, comparing data etc.

#3 could be Firebug & Web Developer together because I use them so much. I use then for example to manipulate hidden elements, modify JS from a page, change input validation and enter all kind of values to forms. I put them in the same category for the fun of it, no particular logic.

#4 is Twitter as I use it sometimes to find out what people say about the companies/products we are testing. Twitter is not the only tool for this, but a really good one.

#5 shall be Paint as it’s a very lightweight application for simple picture modification. I could maybe write a blog post later about using pictures in web testing.

#6 must be something for test/check automation purposes: SVN. It just makes your life so much easier. SVN combined with a CI system is a really good combination for many projects.

#7 place goes for … paper and pencil! I love drawing pictures, writing fast notes, storing words/ideas etc. with paper and pencil. I have been thinking to buy a tablet of some kind for this, but not yet decided on a product.

#8 seems to be Total Commander. This is a fantastic lightweight application for Windows users who want to compare/synchronize folders, copy/delete files etc.

#9 I am not pointing to any single application, but applications that capture “video” of what you do with your computer as sometimes really awesome to track down ways to reproduce a bug, show what really happened etc.

#10 is YSlow which I haven’t used in a while. It’s good for measuring performance of different functionalities of a web page easily while you do other testing. (By the way, I was really tempted to put “my brain” as the last one. J)

15. Let us say there is a commenting feature for the blog post; there are 100 comments currently. How would you load / render every comment. Is it one by one or all 100 at once? Justify.

A few things to consider: how many people read your blog, what devices people use to read the blog, what kind of internet connection they have, do you want to have a compromise solution for everyone or optimize for a certain group, how long the comments are and do they contain other things than plain text too. There are more variables, but these seem to be the ones directing this kind of decision the most.

Now to the loading itself. If you choose to load one-by-one, you might face a situation where the server is getting a lot of requests just for the sake of loading text (if that is the case). That could lead to performance issues with many concurrent users. If you would like to be really clever, you could let the user decide this by clicking “how many comments you want to load at once” selection where you could have a few different options. Default being, for example, 10.

Basically, any decent server should be able to handle loading 100 messages (depending on size of course) really easy, but as we don’t know any details of the environment etc., I need to abstain from giving a clear answer on this one.

16. Have you ever done check automation using open-source tools? How did you identify the checks and what value did you add by automating them? Explain.

I’ve done GUI automation tests/checks for a few reasons. The reasons and analysis are too much to write here at the moment. Maybe I’ll write a blog post about it. Internet already has great writings from this subject and I’d like to recommend “Test automation snake oil” for a starter. Main idea is that I don’t have anything against or pro test automation without an analysis. The answer is as multifold as if the term “automation” would be replaced with “manual”. (Note: I’ve used for example Selenium for check automation. One can get pretty rapid feedback for smoke tests with it when combined with a continuous integration server.)

17. What kind of information do you gather before starting to test a software? (Example: Purpose of this application)

Depends a lot of the application, platform, test “phase” (security, performance etc.) and many other factors, like the customer. Let’s say we would have a web site to test and our job is to see how the functionalities work etc. I would most likely start with CIDTESTD mnemonic. Not because it’s the best one, but it’s a good starting point if you don’t have anything else to compare. CIDTESTD includes information about who customers are, manuals, documents, history, developers, test team, equipment and tools, schedule, test items and deliverables. That is a pretty comprehensive list to start with, but not everything needs to be specified. However, it is usually better the more you know.

In a more general manner, I feel it’s important to understand who uses the software, why they use it, what is my mission (what is expected from me), how much time I have, what kind of reporting is needed etc. I could also want to know if there are legal requirements for using/testing the software, restrictions on what systems it works/should work with, severe impacts on society due a bug (for example a nuclear weapon launch system) and for example if the software is working together with other systems such as banking software.

18. How do you achieve data coverage (Inputs coverage) for a specific form with text fields like mobile number, date of birth etc? There are so many character sets and how do you achieve the coverage? You could share your past experience. If not any then you can talk about how it could be done.

Firstly, I would note the coverage includes also outputs, not only inputs. Secondly, I would like to note I have used a lot of “checklists” for this and I review them with colleagues to see if someone comes up with new test ideas. That is great fun always! Thirdly, I must stress that this is somewhat case-by-case basis for example because with web services one can do so many different things with inputs.

One common way for me is to use automation for storing + giving variance for inputs. Second is that I tend to categorize (XSS, SQL injection, empty, too small, too big, way too big etc.) the tests and use sort of “equivalence classes” in the tests, as in I make assumptions “if X and Y passes, the class they represent is less-likely to be risky”.

I always add some sort of random tests in those equivalence classes if making the tests is cheap. For example, with a web service, you could leave your test computer send different kinds of inputs overnight and check fast in the morning if any input caused strange behavior/errors.

Overcoming Illusions on Testing – Part III

This entry (first part http://jarilaakso.blogspot.com/2012/02/illusions-on-testing-part-i-chimera.html and second part http://jarilaakso.blogspot.com/2012/02/reasons-of-illusions-on-testing-part-ii.html) has been on hold for quite a while; mostly because I have been focusing on other things. (I actually changed even the topic because I thought to write about the future later, in the future!) As we are building the Romanian testing community, I have actual work to do and I’m now a father, I didn’t feel motivated enough to keep on writing. The motivation is back and I’m hoping to start adding more posts in the near future. (If you wonder why I didn’t say “I didn’t have enough time”, it’s simply because it’s a lie. Time is a matter if prioritization. We have it, but not for everything.)

I this post, I will be referring to RST quite many times. This is because I like to use examples in my writing and there were fantastic examples in the course. I won’t, however, tell those examples so you don’t lose anything from the experience.

Without further ado, let’s dig in to the topic at hands. So we have seen detailed pre-scripting doesn’t work too well because a) it kills creativity, b) it doesn’t tolerate change too well, and c) people are just not that good in writing detailed instructions. Actually it does work. Just that it doesn’t work for excellent testing, but it works great for invoicing customers and setting up a fake quality assessment done by “even a stranger from the streets”. The latter means the turnover for testers doesn’t matter for the company.

RST has a few exercises around this topic. One was from the tester’s point of view and one from who was trying to write the script for the tester. The exercises are designed to fail with simple answers, which is fantastic from my point of view. So what to do in this case? Firstly, don’t start writing very detailed test scripts for testers. Secondly, to help improve your (data) coverage, you can always ask help from others. Great testers love to help other testers; and in most organizations I have seen, the developers are keen on working with this.

I mentioned in the previous posts that terminology seems challenging for testers. RST has a lot of keywords you should understand. Otherwise it’s hard to follow the discussions. At least James explained all the terms he was using and based on what I have seen for example here http://www.developsense.com/blog/2012/04/all-oracles-are-heuristic/, I am sure Michael is doing the same. The thing with terminology is that you don’t need a word listing for this. What you need to do is to start reading and talking with others and find a common language.

There was an observation earlier that women did better because they had more sense of context in their replies. I understand this easily from a Finnish point of view where we are told that women analyze things while men drink beer, eat sausages and find The Best Way. I could easily see myself fitting in that form still some years ago. How I got out from it? I think it was just a phase actually, but at the end it’s about what kind of people you gather around yourself and what do you do on your free time. I am advocating on reading and writing, but remember that you can get ideas from pretty much anywhere if you keep your eyes/ears open.

Yes, I covered only 3 of the 5 things I mentioned on earlier posts. Why? Because I want you to tell me what you think about them! 

Getting a Career in Software Testing

I was recently approached by Alfred (name changed to protect his identity as he removed the comment from my blog) who asked if I have any advice on what steps to take in order to train for and start a career in Software Testing. Well, Alfred, thank you for asking! You have already taken important steps; reading blogs and asking questions.

Your concern about ISTQB is a valid one. They do claim they are the industry standard, but that’s only based on the amount of people they have certified, instead of what actually matters. Don’t get fooled by their sales pitch, just focus on learning. Especially a beginner tester might seriously harm his learning when taking their certification. Essentially, they will teach a bad way to use their vocabulary, advocate there are best practices and require that a question can be answered correctly with only one choice.

If you are interested to learn software testing at home, you could do some of the following (and add your own ideas of course):

1)      Start writing a blog about your learning (please let me also know about it so I can read and comment)

2)      Read online material (as the Internet is full of itself, I recommend to stay with only a few sources in the beginning, such as BBST)

3)      Read and comment blogs on software testing (to get you started, here are some http://www.developsense.com/blog/, http://visible-quality.blogspot.com/, http://www.satisfice.com/blog/ and http://scott-barber.blogspot.com/)

4)      Join Twitter and start following testers (you can take a look for example on my list https://twitter.com/#!/jarilaakso/following as a starting point)

5)      Test software, write a report and either leave it on your blog publicly for everyone to comment or ask other testers, like me, to comment via e-mail or whatever you prefer
(Addition after posting already: forgot completely that you could search for conference videos)

If you want to work in software testing, you want to know companies, possibly join recruitment sites etc. Twitter works as a great channel to get to know about new job openings, to find out who is recruiting and for example to get yourself known. But no matter how good Twitter is, you will need to work a lot to be successful. It’s true there are a lot of people who don’t thrive for excellence and they do pretty well in their life. It’s up to you to decide which path you will take.

You mentioned you are based in England. There is a huge software testing market (go ahead for example to Monster.com and verify my claim) and you will have a lot of competition. Think how you want to separate from the crowd and use it in your advantage. For example, if I’d be recruiting at the moment, I would love to receive for example test reports and/or Youtube videos instead of the typical CV’s I tend to see. But hey, that’s just me, I can’t say what you should do, besides that follow your heart in your decisions.

I noticed you mentioned the “test is dead” part in your comment. Don’t take it literally. There is a continuously growing demand for good testing in the world. Think of it as an opportunity for people who want to do a good job, or even yet, an excellent job. The more there is software around us, there more there is request for testing.

If you have open questions, please leave a comment below or for example e-mail me. I will be glad to help everyone I can.

(This post was written after a suggestion in Twitter from "Mike Talks" https://twitter.com/#!/TestSheepNZ/status/199099553587273729)