Usability Test Report With Severity Ratings

Write for two readers: the manager who reads the first page, and the designer or engineer who makes the fixes. Rolf Molich, co-inventor of heuristic evaluation, once compared reports from nine teams that tested the same website. They ran from five pages to 52², and some lacked an executive summary, positive findings or screenshots.

His advice: a one-page executive summary at the front, positive findings, screenshots, and no more than 25 pages. Leave out minor issues if you must, since no test finds them all anyway.

Before you write a single finding, record how each person did on each task. Completion is close to a fact: they reached the agreed end point or they didn't. It also gives every finding a denominator, so "people struggled" becomes "5 of 6 failed." If your sessions were open interviews with no set tasks, user research synthesis is the better fit.

1. 1

   Define done before the first session

   Write what success looks like for each task, such as the new address showing on the order.

2. 2

   Mark every attempt

   For each person and task, record completed, completed with help, or failed, with the timestamp where it was decided.

3. 3

   Set aside broken sessions

   If a server error or a broken prototype ended an attempt, exclude it and say so in the report.

4. 4

   Count against the total

   Write 1 of 6, not 17%. If you give a rate, put the counts beside it.

Small samples can't support precise rates. Jeff Sauro of MeasuringU, a UX research firm, shows why. When 10 out of 10 users complete a task, you can be 95% confident that between 75% and 100% of all users will³. If all six of your participants finish, the range is wider still.

Go back through the sessions and list each distinct problem. Merge the same problem seen on two screens, and split a note that hides two. Describe what people did before you say why: "5 of 6 left without saving" is an observation, while "the button is badly placed" is your reading of it. Give each finding these five parts:

• What happened, in one sentence, naming the screen element
• Who hit it, by participant ID, out of everyone who tried the task
• What it cost them: failed, needed help, or slowed down
• Where to see it: a timestamp in each session, plus a screenshot
• One or two quotes, word for word

Keep a grid of participants against problems behind the findings, so anyone can check who hit what. Problems that stopped nobody go in a short list at the end, not between the ones that did.

Jakob Nielsen's 0 to 4 severity scale runs from 0 (not a usability problem) to 4 (a usability catastrophe). He treats severity as a mix of frequency, impact and persistence⁴: how common a problem is, how hard it is to overcome, and whether it keeps bothering people. He also warns that, in his experience, one evaluator's ratings are too unreliable to trust, and suggests averaging three.

Sauro advises you to treat frequency separately from severity⁵. When possible, his team has a second analyst rate severity without seeing the counts. If you're the only researcher, let what you observed set the order: first how many people a problem stopped, then how many needed help.

Give each problem one recommendation a designer could act on tomorrow. In Molich's CUE-7 study, professional teams wrote fixes for the same six usability problems. The study's takeaways: spell out each recommendation in detail⁶ to avoid "creative misinterpretation," recommend the least possible change, and stay aware of business and technical constraints.

Name the change, not the goal: "Show the Save button without scrolling" beats "Improve the form." A fix ready for the backlog becomes a user story with acceptance criteria. A finding that changes what a feature should do belongs in its product requirements document.

Order the report and the slides the same way, starting with the problem that stopped the most people. Digital.gov asks for findings in "an easy to follow, well prioritized way," not a "basket of issues" or an unprioritized laundry list¹. Put a short clip or screenshot beside each problem, so the team can see it happen.

End with what this round couldn't establish: people you didn't test, tasks you didn't set, sessions that broke. Molich's advice is to never claim or assume a test was exhaustive², so say what the next round should test.

To skip the hand scoring, use the tool at the top of this page. It reads your transcripts, marks every attempt at every task, and ranks the problems by how many people each one stopped.