My Answers to 18 Testing Challenges from Santhosh Tuppad

My friend and a great tester Santhosh Tuppad (https://twitter.com/#!/santhoshst) got an idea of making a testing competition. He put the questions on his blog (http://www.tuppad.com/blog/) and mentioned everyone can participate. I thought it would be a cool thing for practicing my thinking and seeing things from another perspective. My answers were written without too much effort on the visual side, more like a collection of thoughts. They seem to be rather lengthy too. I’d love to hear your comments on my answers so please take the time to read and reply. Here are the questions and answers:

1. What if you click on something (A hyperlink) and to process or navigate to that webpage you need to be signed in? Currently, you are not signed in. Should you be taken to Sign up form or Sign in form? What is the better solution that you can provide?

Firstly, this answer should be included in all answers below: using passwords is an outdated way to handle authorization. Has been for years already. I would have passwords, if someone sees value in using them, for example with TV Guides and online magazines, but in no system that includes sensitive information. Now to the questions in hand!

Making a few assumptions here to get started… Let’s say the “something” would be: you choose an item to shopping cart and click “pay”. The site would require user to be logged in to continue. The first-come-to-mind option would be to have “Login with credentials here” view with an additional option “Don’t have an account yet? Click here!” for registering a new account.

If I would be somewhere in a completely different place, would be redirected to another site and asked to login, I would prefer for example to see what I am about to enter/access. In the shopping cart example I already knew that, but it’s not the same for all hyperlinks.

The question comes down to “will the webpage know if I have an account or not”. If the webpage doesn’t know whether you have an account or not, both options should be visible. If the webpage knows you have an account, login would be visible. If the webpage knows you don’t have an account, registering would be visible. Considering, the webpage has little knowledge who is actually using the computer, knowing if you have an account or not is tricky. A cookie might be present, but that could actually let a “wrong” user to login.

2. Using “Close” naming convention to go back to the homepage is good or it should be named as “Cancel” or it is not really required because there is a “Home” link which is accessible. What are your thoughts?

I assume this in some web page because you mention “homepage”. What would you close in this case to return to homepage? Specifically, if you close something, do you need to return to homepage or would it be open in the background and you would close a popup window? If this is the case, but homepage would not be on the background, what would be there?

Commonly, I would say it’s good to have a few different options to return to homepage because people are used to navigate in different manners.

Returning back to the Close vs. Cancel. Close could be usable when there is actually something to be closed, such as a popup window. Cancel could be usable when user is for example filling up a registration form and decides he doesn’t want to complete it. A context where both could be used would be for example a Flash app appearing on the page. (A concrete example: open a car manufacturer web page, choose a car model, click “customize” which opens a Flash app over the page where you can adjust the configuration of the car. In this case, we could have buttons Close and Cancel – maybe even Back and Back to Home Page.)

3. Logout should be placed on top right hand side? What if it is on the top left hand side or in the left hand sidebar which is menu widget like “My Profile”, “Change Password” etc. – Is it a problem or what is your thought process?

I am used to have logout on the top right, this is how most web pages work. I have asked around from people to put their finger on paper where they think certain functionality exists and “logout finger” goes on top right 100% of the times. This makes sense because many people think “logout” is a way to close the application they are using and most (GUI-based) operating systems provide closing functionality from the top right corner.

Same goes for profile etc. They tend to be on top right. I think this is good for example because people tend to look a bit on the up left (not top left, but a bit higher than center). So when focus is on the left side, it’s better to put insignificant information on the right so there is nothing extra on the concentration area. Another reason is that we look up right when we access so called “visually remembered images”, so when we want to remember something, we tend to move the eyes on top right. This, with the addition of “logout is on top right” to be almost an industry standard, speaks on behalf of keeping logout on top right. When looking right in general, we are trying to remember something instead of using our imagination to figure it out.

4. Current design of forgot password asks for username and security answer and then sends a link to e-mail inbox to set new password. How does “security answer” increase the cost of operations? Also, what questions do you frame for security questions?

I am not sure what “increase the cost of operations” means. Nevertheless, I will think about this situation from usability and security point of view. Maybe you will later explain what the original question meant. :-) (Note: later on I understood this, but as the answer was already written, I thought it would be fun to leave it as it is.)

Let’s consider I own an account in Amazon.com and my username is JariLaakso. I use Amazon rarely so sometimes I need to reset my password because I want it to be unique compared to any other password I use, I want it to be long etc. I go to “forgot password” in Amazon.com, enter “JariLaakso” as my ID and get a prompt about the security question. As this is my account, I want to be able to remember the answer always. I would most likely choose something from my life, such as names or places.

Now let’s consider a malicious user (for example ex-girlfriend) would want to break into my Amazon account. The user would need to know how to reply to that question in order to get my password reset. This is not such a big deal in case she doesn’t have access to my e-mail. Basically, I would be screwed (in the bad sense) if she would be able to access my e-mail already, so the risk doesn’t increase here too much. There would be a problem if the site would directly allow changing the password, but when e-mail is needed in between the risks are lower.

How about if the “forgot password” is for the e-mail? Where the link would be sent when answering correctly to the question? We have found a gold vein! Ultimately, when using this “security question” pipe, we would find out the account what we need to break in order to gain access to pretty much everything else. This rises up alarming concerns. Now when we add “Internet knows everything about you” spice to the soup, we have made a dinner with 5 courses. (From here on, everything is depending how “the last line of defense” is protected. A bad design is to allow a user to change the password when answering correctly to a security question. This is because there is so much information about a general user online already. A better option would be for example via SMS, but phone numbers change etc. so it’s not without risk either. A completely new method will be needed in the near future.)

5. If you had to design “Forgot Password” working, how would you do it and why? You are free to give different many functional designs.

I’ll start again by describing a sort of starting point. I want to do this to explain what kind of context I am thinking. In a different context, the feature could lean more towards security instead of usability. The feature exists on a web page (non-webmail). The page doesn’t store too much personal and/or sensitive information, however, identity thieves are not welcomed warmheartedly. Every user has a unique username and registration happens via a form on the page. I want to login to the site, but I have forgotten my password, so I click “Forgot Password” link. Layout and graphical part is not considered as I focus on how things would work.

There would be a text fields (Note: the information input on the text fields should not be remembered by the browser) where to write your e-mail address and username to get a “reset password” link in your e-mail. The sent e-mail would only have a link to reset the password if needed; it would not be done before clicking that link. Why? Because I want to prevent others from resetting my password. I would also restrict the amount one can reset the password consecutively. There would not be any sense to send multiple “you can reset your password from this link” e-mails to a user.

I would not add captcha because those can be circumvented and they annoy users. They might work for some registrations, but mostly irritate in this function.

In case the user doesn’t remember the e-mail address assigned to the ID or username for that site… most sites would have “security question” feature. I am not too fond of them, as described on previous answer, but I could still have a similar feature if the webpage would be for example something rather meaningless like “online TV Guide”. Even in this case, I would like answering correctly to this question to send an e-mail, but not reset the password etc. just like above. I still don’t want other people to reset my passwords or allow them to spam me from a service I am registered into.

… And just for the sake of argument, for a system which contains sensitive information, such as online bank, the abovementioned is not adequate enough. Basically, all current security systems can be cheated, but I think it’s satisfactory to demand a customer to visit an office in certain situations. This rises up so many branches of discussion, I better continue to the next question. :-)

6. There is neither account lockout policy nor captcha for the login or security answer forms; what kind of problems do you see with the current implementation and what do you propose?

This reminds me of the online banking issues I blogged (http://jarilaakso.blogspot.com/2012/02/internet-banking-experiences-from.html) earlier about. :-)

Not that captcha actually increases security, but let’s say not all jerks in the neighborhood can attack your service automatically if you add a captcha check. However, like said before, captcha can be circumvented (by machine and human force) and there are examples of both online.

The biggest problem comes from so called brute-force attack. The brute-force doesn’t have to be a “stupid one”, but can be firstly based on common password lists etc. It doesn’t even matter, because if you allow a user to guess all the way, they will figure out all usernames and passwords from the database(s).

What the question doesn’t mention is if there is a waiting algorithm between login attempts and if there is one, how will it function. For example, between 1^st and 2^nd login would be a mandatory 5 sec pause. Between 2^nd and 3^rd login would be a 25 sec pause. I guess you got the algorithm. This could be usable for a system where it is not of interest to let someone find out the usernames and passwords. However, it would also cause some usability issues, so I would not recommend it for “everyday systems”.

Not having a security answer form is not a problem itself. It becomes a problem if there is no other way to obtain username or password reset.

My suggestion would depend heavily on how secure the system should be. For example, if has to be really secure, I could also consider limiting access to certain IP ranges, having a certificate on the accessing machine, etc.

7. Well, it is about context and there are no best practices in general. What are your thoughts on usage of captcha? Where should they be used and why?

Oh so I wrote 3 pages of text to arrive here and see I have replied to some of the questions already above. Great!

Captcha is good if you want to limit the amount of potential hackers. It’s not good if you think it prevents anything else. In most cases, I hate captcha. I have actually seen some interesting research articles where computers have gotten better results than humans with captcha. :-)

If it would be up to me, I would not use captcha pretty much anywhere. The benefit is smaller than the loss of usability. If nothing else, I would come up with a completely new way to “verify” the user is not a machine.

8. If you are the solution architect for a retail website which has to be developed; what kind of questions would you ask with respect to “Scalability” purpose with respect to “Technology” being used for the website?

Do you mean hardware with technology? If we include also programming languages, what else do we add? What do other retail websites use? Why? Why not something else? (For example, Facebook goes with LAMP and that seems to work for them. Why? How about Amazon and eBay or maybe the local shop in my town?)

Things to consider about: user amount growth, user amount growth in different countries/continents, what kind of requests are sent from different actions, how long their processing takes, what loads the servers most, how to handle load balancing, what kind of user profiles we will have, what are most common functions and pages (for caching and optimization), etc.

Nevertheless (especially early) users will abandon the system if the response times are not magnificent, context will matter a lot. If your purpose is to sell locally and you expect 99% of traffic from a certain city, you might want to scale for that. However, remember also to read this http://www.zdnet.com/blog/foremski/report-51-of-web-site-traffic-is-non-human-and-mostly-malicious/2201 carefully.

More questions? Ok here are a few: Are we talking about administrative scalability, geographical scalability, load scalability or functional scalability? Maybe all? Maybe a combination? How will we scale out? Do we need to scale up? How will the database affect on this? How about system design?

9. How do you think “Deactivate Account” should work functionally keeping in mind about “Usability” & “Security” quality criteria?

Hopefully by deactivating the desired account from a user. :-)

There are 2 common good ways to handle deactivation:

1) Allow it for a user who has an active session

a. Benefit: Nobody would deactivate your account if they can’t use it.

b. Detriment: It’s more than easy to forget an active session for a computer which can be accessed by other people.

c. Solution: Either the confirmation as I describe next or canceling the deactivation if trying to login again within a certain amount of time.

2) Allow it for a user via a confirmation (e-mail for example)

a. Benefit: It’s not possible to deactivate someone’s account without confirming it.

b. Detriment: People tend to dislike “extra” confirmations.

c. Solution: (Considering this would not be a common use case someone anyway does often, it’s not a big problem, imo.) Explain the user clearly why the confirmation is needed. Ease up the confirmation process for example with a simple clickable link in an e-mail. Include still a “remorse time” during which the user would reactivate the account when signing in again (and maybe receiving a “do you want to keep this account active” question).

All in all, in my opinion, deactivating an account should not remove the data of that person from a system. The information might be needed/usable in the future.

10. For every registration, there is an e-mail sent with activation link. Once this activation link is used account is activated and a “Welcome E-mail” is sent to the end-users e-mail inbox. Now, list down the test ideas which could result in spamming if specific tests are not done.

I assume you are asking “which could result in spamming if specific code is not done” or something like that as testing won’t prevent anything. :-) So let’s start this from design point of view.

Prevent spamming a single user: The system should allow only 1 Activation Link e-mail to be sent to a specific e-mail address. The system should allow only 1 Welcome E-mail to be sent to a specific e-mail address. Exception: If a user deactivates/deletes the account, he should be allowed to register again with the same e-mail.

Prevent spamming multiple users: The system should have a limitation for incoming requests so a malicious user can’t register lots of accounts automatically.

Next step is how to test these. For single user point of view, you would test registering same account a few times and checking if your e-mail received more than 1 e-mail (ideally we would not print “this e-mail address already exists in the system” because it allows malicious users to gain information what e-mail addresses are used in the system). You would also test if it’s possible to register again after deactivation/deleting the account.

Spamming multiple users would require tests such as using many computers from different IP ranges, multiple computers from a small IP range (even from a single IP, like behind a NAT), single computer and to see if there is a delay in consecutive registrations.

Slightly out of the provided context, but closely related: When we are talking about registrations and such, we would also need to consider for example XSS, CSRF and SQL Injection tests. Not as a direct consequence from the Welcome E-mail and the link, but when requesting and storing user data, it would be good if the system would prevent giving those to malicious users.

11. In what different ways can you use “Tamper Data” add-on from “Mozilla Firefox” web browser? If you have not used it till date then how about exploring it and using it; then you can share your experience here.

Phew, finally a shorter answer! I am assuming the question is more about what different tests I do with Tamper Data.

I use it mostly for editing POST parameters, but it’s also usable to tracing HTTP requests/responses. There is a time provided and it can be used for example to see if some actions are causing more load on a server when a single user accesses the service. An example of a check that *might* be useful: measure response times when logging in with different usernames, longer wait could imply the username was found if the code firstly checks if the user exists and then compares the password.

I use Tamper Data also to view headers and sometimes to modify them. Cookie manipulation can be done, however, there are other tools as well for this purpose.

12. Application is being launched in a month from now and management has decided not to test for “Usability” or there are no testers in the team who can perform it and it is a web application. What is your take on this?

Firstly, I would say testers are not responsible for managing a project/product or making release decisions. Testing is done for obtaining information for decision-making. I could advocate what consequences this could have, but the decision to launch and bear the risks is not mine to carry.

That being said, I would question the question: What means to decide not to test for usability? Why the testers can’t perform usability testing? Are there any testers in the team? Who else understands usability? Was it considered in the design? Why it matters if the application is a web app? Why it matters if the application will be launched in the future if the decision is made no usability testing will be done?

In a real-life situation, in my context, working with a customer I am currently working in a project I am currently working, I would of course do something different. If the advocating would not result in convincing the management how much needed the usability testing is, I would:

1) Try to understand the decision and either agree or find a new way to persuade them (if still rejected, one could see it’s time to back off)

2) Use a short while of my own time to collect usability issues, thus either gaining confidence it’s in a good shape or to show what are the major problems I see

3) Talk with the devs or their lead(s) to see what actually could be used, whose responsibility the decision to fix would be, etc.

If the team could not handle usability testing, but the management would like it to be done, I would either involve myself on it, find other people from the company, get a third party involved (if allowed)… In this case, when the team can’t handle the testing, I see it so that I am not part of the team. This would greatly limit my options, however, as I already wrote, a lot of things can be done. The challenge would be to see what usability issues could be fixed before the release considering there might be still features to be implemented and other bugs to be fixed.

13. Share your experience wherein; the developer did not accept security vulnerability and you did great bug advocacy to prove that it is a bug and finally it was fixed. Even if it was not fixed then please let me know about what was the bug and how did you do bug advocacy without revealing the application / company details.

I can’t recall a case where my security bugs would have been marked “invalid” or something else suggesting they are “not accepted” by a developer. I remember cases where I have defended other testers’ bugs, but I can’t recall this happening for myself. I tend to write the risk and other relevant information on security bugs because it saves time on a long run. (So I have been told, lol.)

As a guideline, if you don’t know the dev and/or have history to know how he might understand your bugs, it’s a good idea to include your deductions and claims already in the initial bug report. Describing what problems a bug might cause, how they can be abused and what other kind of risks there are usually helps the dev to make the correct decision.

I also would like to note that in 99% of my context the devs don’t mark bugs “invalid” or “won’t fix” before a meeting is held with more people.

14. What do you have in your tester’s toolkit? Name at least 10 such tools or utilities. Please do not list like QTP, LoadRunner, SilkTest and such things. Something which you have discovered (Example: Process Explorer from SysInternals) on your own or from your colleague. If you can also share how you use it then it would be fantastic.

Considering this is a blog post that has been in my queue for a long time, I will try to just summarize a few things here. The way how the question is put makes me believe the tools you are referring to should not be test tools, but other tools which can be used in testing. If I misunderstood, please correct me. List is in order how they came to my mind.

#1 tool coming into my mind is FreeMind http://freemind.sourceforge.net/wiki/index.php/Download. It is a great free tool for making mindmaps.

#2 tool has to be Excel. Excel is fantastic for keeping notes, making reports, collecting data, comparing data etc.

#3 could be Firebug & Web Developer together because I use them so much. I use then for example to manipulate hidden elements, modify JS from a page, change input validation and enter all kind of values to forms. I put them in the same category for the fun of it, no particular logic.

#4 is Twitter as I use it sometimes to find out what people say about the companies/products we are testing. Twitter is not the only tool for this, but a really good one.

#5 shall be Paint as it’s a very lightweight application for simple picture modification. I could maybe write a blog post later about using pictures in web testing.

#6 must be something for test/check automation purposes: SVN. It just makes your life so much easier. SVN combined with a CI system is a really good combination for many projects.

#7 place goes for … paper and pencil! I love drawing pictures, writing fast notes, storing words/ideas etc. with paper and pencil. I have been thinking to buy a tablet of some kind for this, but not yet decided on a product.

#8 seems to be Total Commander. This is a fantastic lightweight application for Windows users who want to compare/synchronize folders, copy/delete files etc.

#9 I am not pointing to any single application, but applications that capture “video” of what you do with your computer as sometimes really awesome to track down ways to reproduce a bug, show what really happened etc.

#10 is YSlow which I haven’t used in a while. It’s good for measuring performance of different functionalities of a web page easily while you do other testing. (By the way, I was really tempted to put “my brain” as the last one. J)

15. Let us say there is a commenting feature for the blog post; there are 100 comments currently. How would you load / render every comment. Is it one by one or all 100 at once? Justify.

A few things to consider: how many people read your blog, what devices people use to read the blog, what kind of internet connection they have, do you want to have a compromise solution for everyone or optimize for a certain group, how long the comments are and do they contain other things than plain text too. There are more variables, but these seem to be the ones directing this kind of decision the most.

Now to the loading itself. If you choose to load one-by-one, you might face a situation where the server is getting a lot of requests just for the sake of loading text (if that is the case). That could lead to performance issues with many concurrent users. If you would like to be really clever, you could let the user decide this by clicking “how many comments you want to load at once” selection where you could have a few different options. Default being, for example, 10.

Basically, any decent server should be able to handle loading 100 messages (depending on size of course) really easy, but as we don’t know any details of the environment etc., I need to abstain from giving a clear answer on this one.

16. Have you ever done check automation using open-source tools? How did you identify the checks and what value did you add by automating them? Explain.

I’ve done GUI automation tests/checks for a few reasons. The reasons and analysis are too much to write here at the moment. Maybe I’ll write a blog post about it. Internet already has great writings from this subject and I’d like to recommend “Test automation snake oil” for a starter. Main idea is that I don’t have anything against or pro test automation without an analysis. The answer is as multifold as if the term “automation” would be replaced with “manual”. (Note: I’ve used for example Selenium for check automation. One can get pretty rapid feedback for smoke tests with it when combined with a continuous integration server.)

17. What kind of information do you gather before starting to test a software? (Example: Purpose of this application)

Depends a lot of the application, platform, test “phase” (security, performance etc.) and many other factors, like the customer. Let’s say we would have a web site to test and our job is to see how the functionalities work etc. I would most likely start with CIDTESTD mnemonic. Not because it’s the best one, but it’s a good starting point if you don’t have anything else to compare. CIDTESTD includes information about who customers are, manuals, documents, history, developers, test team, equipment and tools, schedule, test items and deliverables. That is a pretty comprehensive list to start with, but not everything needs to be specified. However, it is usually better the more you know.

In a more general manner, I feel it’s important to understand who uses the software, why they use it, what is my mission (what is expected from me), how much time I have, what kind of reporting is needed etc. I could also want to know if there are legal requirements for using/testing the software, restrictions on what systems it works/should work with, severe impacts on society due a bug (for example a nuclear weapon launch system) and for example if the software is working together with other systems such as banking software.

18. How do you achieve data coverage (Inputs coverage) for a specific form with text fields like mobile number, date of birth etc? There are so many character sets and how do you achieve the coverage? You could share your past experience. If not any then you can talk about how it could be done.

Firstly, I would note the coverage includes also outputs, not only inputs. Secondly, I would like to note I have used a lot of “checklists” for this and I review them with colleagues to see if someone comes up with new test ideas. That is great fun always! Thirdly, I must stress that this is somewhat case-by-case basis for example because with web services one can do so many different things with inputs.

One common way for me is to use automation for storing + giving variance for inputs. Second is that I tend to categorize (XSS, SQL injection, empty, too small, too big, way too big etc.) the tests and use sort of “equivalence classes” in the tests, as in I make assumptions “if X and Y passes, the class they represent is less-likely to be risky”.

I always add some sort of random tests in those equivalence classes if making the tests is cheap. For example, with a web service, you could leave your test computer send different kinds of inputs overnight and check fast in the morning if any input caused strange behavior/errors.