My friend and a great tester
Santhosh Tuppad (https://twitter.com/#!/santhoshst) got an idea of
making a testing competition. He put the questions on his blog (http://www.tuppad.com/blog/)
and mentioned everyone can participate. I thought it would be a cool thing for
practicing my thinking and seeing things from another perspective. My answers
were written without too much effort on the visual side, more like a collection
of thoughts. They seem to be rather lengthy too. I’d love to hear your comments
on my answers so please take the time to read and reply. Here are the questions
and answers:
- What if you click on something (A hyperlink) and to
process or navigate to that webpage you need to be signed in? Currently,
you are not signed in. Should you be taken to Sign up form or Sign in
form? What is the better solution that you can provide?
Firstly, this answer should be
included in all answers below: using passwords is an outdated way to handle
authorization. Has been for years already. I would have passwords, if someone
sees value in using them, for example with TV Guides and online magazines, but
in no system that includes sensitive information. Now to the questions in hand!
Making a few assumptions here to get
started… Let’s say the “something” would be: you choose an item to shopping
cart and click “pay”. The site would require user to be logged in to continue.
The first-come-to-mind option would be to have “Login with credentials here”
view with an additional option “Don’t have an account yet? Click here!” for
registering a new account.
If I would be somewhere in a
completely different place, would be redirected to another site and asked to
login, I would prefer for example to see what I am about to enter/access. In
the shopping cart example I already knew that, but it’s not the same for all
hyperlinks.
The question comes down to “will the
webpage know if I have an account or not”. If the webpage doesn’t know whether
you have an account or not, both options should be visible. If the webpage
knows you have an account, login would be visible. If the webpage knows you
don’t have an account, registering would be visible. Considering, the webpage
has little knowledge who is actually using the computer, knowing if you have an
account or not is tricky. A cookie might be present, but that could actually
let a “wrong” user to login.
- Using “Close” naming convention to go back to the
homepage is good or it should be named as “Cancel” or it is not really
required because there is a “Home” link which is accessible. What are your
thoughts?
I assume this in some web page
because you mention “homepage”. What would you close in this case to return to
homepage? Specifically, if you close something, do you need to return to
homepage or would it be open in the background and you would close a popup
window? If this is the case, but homepage would not be on the background, what
would be there?
Commonly, I would say it’s good to
have a few different options to return to homepage because people are used to
navigate in different manners.
Returning back to the Close vs.
Cancel. Close could be usable when there is actually something to be closed,
such as a popup window. Cancel could be usable when user is for example filling
up a registration form and decides he doesn’t want to complete it. A context
where both could be used would be for example a Flash app appearing on the
page. (A concrete example: open a car manufacturer web page, choose a car
model, click “customize” which opens a Flash app over the page where you can
adjust the configuration of the car. In this case, we could have buttons Close
and Cancel – maybe even Back and Back to Home Page.)
- Logout should be placed on top right hand side? What if
it is on the top left hand side or in the left hand sidebar which is menu
widget like “My Profile”, “Change Password” etc. – Is it a problem or what
is your thought process?
I am used to have logout on the top
right, this is how most web pages work. I have asked around from people to put
their finger on paper where they think certain functionality exists and “logout
finger” goes on top right 100% of the times. This makes sense because many
people think “logout” is a way to close the application they are using and most
(GUI-based) operating systems provide closing functionality from the top right
corner.
Same goes for profile etc. They tend
to be on top right. I think this is good for example because people tend to
look a bit on the up left (not top left, but a bit higher than center). So when
focus is on the left side, it’s better to put insignificant information on the
right so there is nothing extra on the concentration area. Another reason is
that we look up right when we access so called “visually remembered images”, so
when we want to remember something, we tend to move the eyes on top right.
This, with the addition of “logout is on top right” to be almost an industry
standard, speaks on behalf of keeping logout on top right. When looking right
in general, we are trying to remember something instead of using our
imagination to figure it out.
- Current design of forgot password asks for username and
security answer and then sends a link to e-mail inbox to set new password.
How does “security answer” increase the cost of operations? Also, what
questions do you frame for security questions?
I am not sure what “increase the
cost of operations” means. Nevertheless, I will think about this situation from
usability and security point of view. Maybe you will later explain what the
original question meant. :-) (Note: later
on I understood this, but as the answer was already written, I thought it would
be fun to leave it as it is.)
Let’s consider I own an account in
Amazon.com and my username is JariLaakso. I use Amazon rarely so sometimes I
need to reset my password because I want it to be unique compared to any other
password I use, I want it to be long etc. I go to “forgot password” in
Amazon.com, enter “JariLaakso” as my ID and get a prompt about the security
question. As this is my account, I want to be able to remember the answer
always. I would most likely choose something from my life, such as names or
places.
Now let’s consider a malicious user
(for example ex-girlfriend) would want to break into my Amazon account. The
user would need to know how to reply to that question in order to get my
password reset. This is not such a big deal in case she doesn’t have access to
my e-mail. Basically, I would be screwed (in the bad sense) if she would be
able to access my e-mail already, so the risk doesn’t increase here too much.
There would be a problem if the site would directly allow changing the
password, but when e-mail is needed in between the risks are lower.
How about if the “forgot password”
is for the e-mail? Where the link would be sent when answering correctly to the
question? We have found a gold vein! Ultimately, when using this “security
question” pipe, we would find out the account what we need to break in order to
gain access to pretty much everything else. This rises up alarming concerns.
Now when we add “Internet knows everything about you” spice to the soup, we
have made a dinner with 5 courses. (From here on, everything is depending how
“the last line of defense” is protected. A bad design is to allow a user to
change the password when answering correctly to a security question. This is
because there is so much information about a general user online already. A
better option would be for example via SMS, but phone numbers change etc. so
it’s not without risk either. A completely new method will be needed in the
near future.)
- If you had to design “Forgot Password” working, how
would you do it and why? You are free to give different many functional
designs.
I’ll start again by describing a
sort of starting point. I want to do this to explain what kind of context I am
thinking. In a different context, the feature could lean more towards security
instead of usability. The feature exists on a web page (non-webmail). The page
doesn’t store too much personal and/or sensitive information, however, identity
thieves are not welcomed warmheartedly. Every user has a unique username and
registration happens via a form on the page. I want to login to the site, but I
have forgotten my password, so I click “Forgot Password” link. Layout and
graphical part is not considered as I focus on how things would work.
There would be a text fields (Note: the information input on the text
fields should not be remembered by the browser) where to write your e-mail
address and username to get a “reset password” link in your e-mail. The sent
e-mail would only have a link to reset the password if needed; it would not be
done before clicking that link. Why? Because I want to prevent others from
resetting my password. I would also restrict the amount one can reset the
password consecutively. There would not be any sense to send multiple “you can
reset your password from this link” e-mails to a user.
I would not add captcha because
those can be circumvented and they annoy users. They might work for some
registrations, but mostly irritate in this function.
In case the user doesn’t remember
the e-mail address assigned to the ID or username for that site… most sites
would have “security question” feature. I am not too fond of them, as described
on previous answer, but I could still have a similar feature if the webpage
would be for example something rather meaningless like “online TV Guide”. Even
in this case, I would like answering correctly to this question to send an
e-mail, but not reset the password etc. just like above. I still don’t want
other people to reset my passwords or allow them to spam me from a service I am
registered into.
… And just for the sake of argument,
for a system which contains sensitive information, such as online bank, the
abovementioned is not adequate enough. Basically, all current security systems
can be cheated, but I think it’s satisfactory to demand a customer to visit an
office in certain situations. This rises up so many branches of discussion, I
better continue to the next question. :-)
- There is neither account lockout policy nor captcha for
the login or security answer forms; what kind of problems do you see with
the current implementation and what do you propose?
Not that captcha actually increases
security, but let’s say not all jerks in the neighborhood can attack your
service automatically if you add a captcha check. However, like said before,
captcha can be circumvented (by machine and human force) and there are examples
of both online.
The biggest problem comes from so
called brute-force attack. The brute-force doesn’t have to be a “stupid one”,
but can be firstly based on common password lists etc. It doesn’t even matter,
because if you allow a user to guess all the way, they will figure out all
usernames and passwords from the database(s).
What the question doesn’t mention is
if there is a waiting algorithm between login attempts and if there is one, how
will it function. For example, between 1st and 2nd login
would be a mandatory 5 sec pause. Between 2nd and 3rd
login would be a 25 sec pause. I guess you got the algorithm. This could be
usable for a system where it is not of interest to let someone find out the
usernames and passwords. However, it would also cause some usability issues, so
I would not recommend it for “everyday systems”.
Not having a security answer form is
not a problem itself. It becomes a problem if there is no other way to obtain
username or password reset.
My suggestion would depend heavily
on how secure the system should be. For example, if has to be really secure, I
could also consider limiting access to certain IP ranges, having a certificate
on the accessing machine, etc.
- Well, it is about context and there are no best
practices in general. What are your thoughts on usage of captcha? Where
should they be used and why?
Oh so I wrote 3 pages of text to
arrive here and see I have replied to some of the questions already above.
Great!
Captcha is good if you want to limit
the amount of potential hackers. It’s not good if you think it prevents
anything else. In most cases, I hate captcha. I have actually seen some
interesting research articles where computers have gotten better results than
humans with captcha. :-)
If it would be up to me, I would not
use captcha pretty much anywhere. The benefit is smaller than the loss of
usability. If nothing else, I would come up with a completely new way to
“verify” the user is not a machine.
- If you are the solution architect for a retail website
which has to be developed; what kind of questions would you ask with
respect to “Scalability” purpose with respect to “Technology” being used
for the website?
Do you mean hardware with
technology? If we include also programming languages, what else do we add? What
do other retail websites use? Why? Why not something else? (For example,
Facebook goes with LAMP and that seems to work for them. Why? How about Amazon
and eBay or maybe the local shop in my town?)
Things to consider about: user
amount growth, user amount growth in different countries/continents, what kind
of requests are sent from different actions, how long their processing takes,
what loads the servers most, how to handle load balancing, what kind of user
profiles we will have, what are most common functions and pages (for caching
and optimization), etc.
More questions? Ok here are a few:
Are we talking about administrative scalability, geographical scalability, load
scalability or functional scalability? Maybe all? Maybe a combination? How will
we scale out? Do we need to scale up? How will the database affect on this? How
about system design?
- How do you think “Deactivate Account” should work
functionally keeping in mind about “Usability” & “Security” quality
criteria?
Hopefully by deactivating the
desired account from a user. :-)
There are 2 common good ways to
handle deactivation:
1) Allow
it for a user who has an active session
a.
Benefit: Nobody would deactivate
your account if they can’t use it.
b.
Detriment: It’s more than easy to
forget an active session for a computer which can be accessed by other people.
c.
Solution: Either the confirmation as
I describe next or canceling the deactivation if trying to login again within a
certain amount of time.
2) Allow
it for a user via a confirmation (e-mail for example)
a.
Benefit: It’s not possible to
deactivate someone’s account without confirming it.
b.
Detriment: People tend to dislike
“extra” confirmations.
c.
Solution: (Considering this would
not be a common use case someone anyway does often, it’s not a big problem,
imo.) Explain the user clearly why the confirmation is needed. Ease up the
confirmation process for example with a simple clickable link in an e-mail.
Include still a “remorse time” during which the user would reactivate the
account when signing in again (and maybe receiving a “do you want to keep this
account active” question).
All in all, in my opinion,
deactivating an account should not remove the data of that person from a
system. The information might be needed/usable in the future.
- For every registration, there is an e-mail sent with
activation link. Once this activation link is used account is activated
and a “Welcome E-mail” is sent to the end-users e-mail inbox. Now, list
down the test ideas which could result in spamming if specific tests are
not done.
I assume you are asking “which could
result in spamming if specific code is not done” or something like that as
testing won’t prevent anything. :-) So let’s start this from design point of
view.
Prevent spamming a single user: The
system should allow only 1 Activation Link e-mail to be sent to a specific
e-mail address. The system should allow only 1 Welcome E-mail to be sent to a
specific e-mail address. Exception: If a user deactivates/deletes the account,
he should be allowed to register again with the same e-mail.
Prevent spamming multiple users: The
system should have a limitation for incoming requests so a malicious user can’t
register lots of accounts automatically.
Next step is how to test these. For
single user point of view, you would test registering same account a few times
and checking if your e-mail received more than 1 e-mail (ideally we would not
print “this e-mail address already exists in the system” because it allows
malicious users to gain information what e-mail addresses are used in the
system). You would also test if it’s possible to register again after
deactivation/deleting the account.
Spamming multiple users would
require tests such as using many computers from different IP ranges, multiple
computers from a small IP range (even from a single IP, like behind a NAT),
single computer and to see if there is a delay in consecutive registrations.
Slightly out of the provided
context, but closely related: When we are talking about registrations and such,
we would also need to consider for example XSS, CSRF and SQL Injection tests.
Not as a direct consequence from the Welcome E-mail and the link, but when
requesting and storing user data, it would be good if the system would prevent
giving those to malicious users.
- In what different ways can you use “Tamper Data” add-on
from “Mozilla Firefox” web browser? If you have not used it till date then
how about exploring it and using it; then you can share your experience
here.
Phew, finally a shorter answer! I am
assuming the question is more about what different tests I do with Tamper Data.
I use it mostly for editing POST
parameters, but it’s also usable to tracing HTTP requests/responses. There is a
time provided and it can be used for example to see if some actions are causing
more load on a server when a single user accesses the service. An example of a
check that *might* be useful: measure response times when logging in with
different usernames, longer wait could imply the username was found if the code
firstly checks if the user exists and then compares the password.
I use Tamper Data also to view
headers and sometimes to modify them. Cookie manipulation can be done, however,
there are other tools as well for this purpose.
- Application is being launched in a month from now and
management has decided not to test for “Usability” or there are no testers
in the team who can perform it and it is a web application. What is your
take on this?
Firstly, I would say testers are not
responsible for managing a project/product or making release decisions. Testing
is done for obtaining information for decision-making. I could advocate what
consequences this could have, but the decision to launch and bear the risks is
not mine to carry.
That being said, I would question
the question: What means to decide not to test for usability? Why the testers
can’t perform usability testing? Are there any testers in the team? Who else
understands usability? Was it considered in the design? Why it matters if the
application is a web app? Why it matters if the application will be launched in
the future if the decision is made no usability testing will be done?
In a real-life situation, in my
context, working with a customer I am currently working in a project I am
currently working, I would of course do something different. If the advocating
would not result in convincing the management how much needed the usability testing
is, I would:
1) Try
to understand the decision and either agree or find a new way to persuade them
(if still rejected, one could see it’s time to back off)
2) Use
a short while of my own time to collect usability issues, thus either gaining confidence
it’s in a good shape or to show what are the major problems I see
3) Talk
with the devs or their lead(s) to see what actually could be used, whose
responsibility the decision to fix would be, etc.
If the team could not handle
usability testing, but the management would like it to be done, I would either
involve myself on it, find other people from the company, get a third party
involved (if allowed)… In this case, when the team can’t handle the testing, I
see it so that I am not part of the team. This would greatly limit my options,
however, as I already wrote, a lot of things can be done. The challenge would
be to see what usability issues could be fixed before the release considering
there might be still features to be implemented and other bugs to be fixed.
- Share your experience wherein; the developer did not
accept security vulnerability and you did great bug advocacy to prove that
it is a bug and finally it was fixed. Even if it was not fixed then please
let me know about what was the bug and how did you do bug advocacy without
revealing the application / company details.
I can’t recall a case where my
security bugs would have been marked “invalid” or something else suggesting
they are “not accepted” by a developer. I remember cases where I have defended
other testers’ bugs, but I can’t recall this happening for myself. I tend to
write the risk and other relevant information on security bugs because it saves
time on a long run. (So I have been told, lol.)
As a guideline, if you don’t know
the dev and/or have history to know how he might understand your bugs, it’s a
good idea to include your deductions and claims already in the initial bug
report. Describing what problems a bug might cause, how they can be abused and
what other kind of risks there are usually helps the dev to make the correct
decision.
I also would like to note that in
99% of my context the devs don’t mark bugs “invalid” or “won’t fix” before a
meeting is held with more people.
- What do you have in your tester’s toolkit? Name at least
10 such tools or utilities. Please do not list like QTP, LoadRunner,
SilkTest and such things. Something which you have discovered (Example:
Process Explorer from SysInternals) on your own or from your colleague. If
you can also share how you use it then it would be fantastic.
Considering this is a blog post that
has been in my queue for a long time, I will try to just summarize a few things
here. The way how the question is put makes me believe the tools you are
referring to should not be test tools, but other tools which can be used in
testing. If I misunderstood, please correct me. List is in order how they came
to my mind.
#2 tool has to be Excel. Excel is
fantastic for keeping notes, making reports, collecting data, comparing data
etc.
#3 could be Firebug & Web
Developer together because I use them so much. I use then for example to
manipulate hidden elements, modify JS from a page, change input validation and
enter all kind of values to forms. I put them in the same category for the fun
of it, no particular logic.
#4 is Twitter as I use it sometimes
to find out what people say about the companies/products we are testing.
Twitter is not the only tool for this, but a really good one.
#5 shall be Paint as it’s a very
lightweight application for simple picture modification. I could maybe write a
blog post later about using pictures in web testing.
#6 must be something for test/check
automation purposes: SVN. It just makes your life so much easier. SVN combined
with a CI system is a really good combination for many projects.
#7 place goes for … paper and
pencil! I love drawing pictures, writing fast notes, storing words/ideas etc.
with paper and pencil. I have been thinking to buy a tablet of some kind for
this, but not yet decided on a product.
#8 seems to be Total Commander. This
is a fantastic lightweight application for Windows users who want to
compare/synchronize folders, copy/delete files etc.
#9 I am not pointing to any single
application, but applications that capture “video” of what you do with your
computer as sometimes really awesome to track down ways to reproduce a bug,
show what really happened etc.
#10 is YSlow which I haven’t used in
a while. It’s good for measuring performance of different functionalities of a
web page easily while you do other testing. (By the way, I was really tempted
to put “my brain” as the last one. J)
- Let us say there is a commenting feature for the blog
post; there are 100 comments currently. How would you load / render every
comment. Is it one by one or all 100 at once? Justify.
A few things to consider: how many
people read your blog, what devices people use to read the blog, what kind of
internet connection they have, do you want to have a compromise solution for
everyone or optimize for a certain group, how long the comments are and do they
contain other things than plain text too. There are more variables, but these
seem to be the ones directing this kind of decision the most.
Now to the loading itself. If you
choose to load one-by-one, you might face a situation where the server is
getting a lot of requests just for the sake of loading text (if that is the
case). That could lead to performance issues with many concurrent users. If you
would like to be really clever, you could let the user decide this by clicking
“how many comments you want to load at once” selection where you could have a
few different options. Default being, for example, 10.
Basically, any decent server should
be able to handle loading 100 messages (depending on size of course) really
easy, but as we don’t know any details of the environment etc., I need to
abstain from giving a clear answer on this one.
- Have you ever done check automation using open-source
tools? How did you identify the checks and what value did you add by
automating them? Explain.
I’ve done GUI automation
tests/checks for a few reasons. The reasons and analysis are too much to write
here at the moment. Maybe I’ll write a blog post about it. Internet already has
great writings from this subject and I’d like to recommend “Test automation snake
oil” for a starter. Main idea is that I don’t have anything against or pro test
automation without an analysis. The answer is as multifold as if the term
“automation” would be replaced with “manual”. (Note: I’ve used for example Selenium for check automation. One can get
pretty rapid feedback for smoke tests with it when combined with a continuous
integration server.)
- What kind of information do you gather before starting
to test a software? (Example: Purpose of this application)
Depends a lot of the application,
platform, test “phase” (security, performance etc.) and many other factors,
like the customer. Let’s say we would have a web site to test and our job is to
see how the functionalities work etc. I would most likely start with CIDTESTD
mnemonic. Not because it’s the best one, but it’s a good starting point if you
don’t have anything else to compare. CIDTESTD includes information about who
customers are, manuals, documents, history, developers, test team, equipment
and tools, schedule, test items and deliverables. That is a pretty
comprehensive list to start with, but not everything needs to be specified.
However, it is usually better the more you know.
In a more general manner, I feel
it’s important to understand who uses the software, why they use it, what is my
mission (what is expected from me), how much time I have, what kind of
reporting is needed etc. I could also want to know if there are legal
requirements for using/testing the software, restrictions on what systems it
works/should work with, severe impacts on society due a bug (for example a
nuclear weapon launch system) and for example if the software is working
together with other systems such as banking software.
- How do you achieve data coverage (Inputs coverage) for
a specific form with text fields like mobile number, date of birth etc?
There are so many character sets and how do you achieve the coverage? You
could share your past experience. If not any then you can talk about how
it could be done.
Firstly, I would note the coverage
includes also outputs, not only inputs. Secondly, I would like to note I have
used a lot of “checklists” for this and I review them with colleagues to see if
someone comes up with new test ideas. That is great fun always! Thirdly, I must
stress that this is somewhat case-by-case basis for example because with web
services one can do so many different things with inputs.
One common way for me is to use
automation for storing + giving variance for inputs. Second is that I tend to
categorize (XSS, SQL injection, empty, too small, too big, way too big etc.)
the tests and use sort of “equivalence classes” in the tests, as in I make
assumptions “if X and Y passes, the class they represent is less-likely to be
risky”.
I always add some sort of random
tests in those equivalence classes if making the tests is cheap. For example,
with a web service, you could leave your test computer send different kinds of
inputs overnight and check fast in the morning if any input caused strange
behavior/errors.
