HCD Qualitative User Testing Project - (40 points.)

Remember, while working on this project, take notes!!
Write down the names of people who help you develop tasks / scenarios.
Later, write down the names of the students who take your tests. (Not the same people!)

Web sites were randomly assigned to students by the professor.

Here is the list of web sites and students. (If you don't like your assigned web site, blame the alphabet.)

Why: We want to study first-time user learn-ability of web sites, creating a benchmark for future designs. (Future web sites should be as usable, or more usable. Not less!)

Purpose: We are looking for an easy to use interface where errors are unlikely and common tasks are easier and faster than rare tasks.

You will be looking for design improvements for the next web site, so pick some nice hard tasks, tasks that will strain the usability of your web site.

Where: Via video-conferencing. You will use 'share screen' tools, in combination with a webcam view of test subjects. That way, the test giver can see the facial expressions and the web browser interactions of the test taker.

What to test: you pick the tasks. Pick 5 to 6 or so tasks, applying insights from the textbook. Pick:

• Common tasks,
• plus at least one rare task
• and your set of tasks must not be all "find X" or "search for X" tasks
  • at most one task may use the search bar -- we are testing web site design, not Google's search engine
  • you may need to simply tell your testers to not use the web site search engine

If possible, try to get your test takers all on the same web browser and, hopefully, the same version of the web site. (Many web sites have a desktop version, and a smart phone version, and automatically display the version depending on a visitor's hardware.) If it is not possible to keep the web browser and web site versions consistent, include in your notes information about your test takers' web browser software and the web site versions they visited.

Ask in class if you want advice picking tasks.

Create one or more scenarios (for two or more tasks each). (More scenarios is better. In the reflections of students who've done usability testing in previous semesters, the most common improvement students have mentioned is "I should have used more scenarios." Some students have said scenarios are more fun and some have said more scenarios would have improved the quality of their testing.)

Required pre-test questions for your test participants are:

• "What is your name?" and
• "What is your web browser?"

Create one or more additional pre-test questions to ask each test participant. (Note that "what is your age?" is an inappropriately personal question, and also provides no useful insights for web site usability analysis.)

Create 2-4 de-briefing questions, to ask each test participant after testing is over. At least one de-briefing questions should, in some way, try to gather "impressions and opinions" from your test takers. Avoid yes/no de-briefing questions. Seek after questions that provoke reflection and will provide you insight into the user's experiences and impressions.

Write down a plan for actions you need to perform in between test participants, to clean up after the last test participant and prepare for a new one. For example, if you have your test participants go through most of the process of purchasing something from the web site, write down that you will need to empty your "cart" on the web site and delete any other records of the purchase process each and every time you prepare for a new test participant.

Keep track of errors - their frequency, type, and severity. We're interested in learn-ability, severity and frequency, errors of omission or commission.

Who: Tester: you, for your web site.

Test subjects: Two or more fellow students. The test results will be best if test subjects have never used your web site before.

(If you are not able to attend class on the testing dates, contact the professor promptly. To protect the welfare of test participants, usability tests performed outside of CS 3500 will not count for your project, unless you get the professor's permission ahead of time.)

We will use "think out loud" observation. "I would like you to think out loud and describe why you do things and how you feel about the way they are done." (RI, Ch 8)

Getting stuck is discouraging for subjects. Try to give test subjects an early success experience. Start with a task you think will be relatively easy. Remind the test subjects that they can quit at any time for any reason

"If subjects get stuck: While the experimenter should not help the subject with the task, there are a few exceptions to this rule.

• If a subject has problems getting started, record their problems and give them a hint to get going. This is OK, because if they can't get started, they will not be able to do the tasks!
• If a subject cannot complete a particular task after a reasonable amount of time, tell them to stop and start them on the next task. Or, give them a hint if they cannot overcome some conceptual problem necessary to trying out other parts of the system. Again, record all problems." (Quotation courtesy Saul Greenberg)

If you have to give your user hints, discuss why in your report -- you may have found a usability problem for new users of your web site.

Remember to de-brief your test participants using your pre-planned debriefing questions. Answer any questions they have.

When you are a test participant: Go slowly, think out loud, and go slowly. The person giving you the test needs time to make observations and take notes.

For more information:
"Write Better Qualitative Usability Tasks: Top 10 Mistakes to Avoid" by Amy Schade at Nielsen Norman Group, on April 9, 2017

The qualitative analysis report will be a MS Word or PDF document (.doc, .docx or .pdf name ending).

The qualitative analysis report will contain:

• Formatting
• Student first and last name and "CS3500" at the top
• Section 1: Setting the record straight
• Identify your web site. Provide title and URL of web site.
• Names of the students (first and last) who worked with you in developing your list of tasks. (If, for some reason, you made up your task list alone, say so.)
• Names of the students (first and last) who were your test participants. (Not the same as the students who helped you create your tests!)
• Section 2: Analysis plan
• Numbered and complete list of tasks.
• All your tasks should be on this list, even if they were part of a scenario given to the test participants as well.
• Clear descriptions of your scenarios, with the tasks each scenario encompassed listed. I.e. "Scenario 1: Tell the test participant: 'You are walking down the street when you see a cute puppy. Overcome by the urge to photograph the puppy, you reach into your right pants pocket for your phone...' In going through Scenario 1, the test participant will complete tasks 3, 6 and 1."
• Do not mix together your list of tasks and your scenario descriptions. First, the list of tasks. After that list, and completely after that, the scenario descriptions.
• Describe your plan for actions to take in between test participants.
• List of your pre-test questions.
• List of your post-test de-briefing questions.
• Section 3: Analysis report
• For each test participant:
  • Observations from that student's testing. Which tasks were easy, which tasks were hard, what that student observed, where errors were made (and why, if you have a hypothesis).
• Section 4: Thoughts about Analysis
• Interpretations: What you learned, inferred, and observed about the usability of the web site. Identify strengths and weaknesses of the web site.

  Remember, we are examining the usability of your web site for new users of the web site only. What a new user has trouble with might be very different from what an experienced user (like you) has trouble with.

• Improvements: Suggest some improvements for the web site. Back your suggestions up with references to your test observations; which test user might your improvement help, with what task(s), and why?

  It may help to remember what we learned from Don Norman. Would your suggested improvement help the user develop a more accurate conceptual model? Improve feedback? Improve mapping? Improve visibility? Reduce the likelihood of some kind of error, or make error recovery easier?

• Conclusions and Reflections
  • Test Reflection: If you were to perform this usability test again in the future, how would you improve your test plan? How would you perform qualitative testing differently?
  • Notes on any other part of your testing plans and results that I have forgotten to mention.

Spelling, grammar and quality of writing always count.

The analysis report, as an MS Word or PDF document (.doc, .docx or .pdf), is to be submitted to the Canvas Assignment.

Grading Sheet:

Total Points (40)

Last Updated on 6/22/2020
By Megan Thomas
With thanks to Saul Greenberg for some ideas.