The short answer
To prioritise documentation fixes, leave out pages with fewer than 30 answers, then score the rest: share of No answers times page views, times how much the page matters (1 to 3), divided by the cost of the fix (1 to 3). Fix from the top of that list. Read the comments first, because some No answers come from readers who landed on the wrong page, and that needs a navigation fix, not a rewrite.
A few weeks after adding "Did this page answer your question?" to every docs page, you have hundreds of No answers over dozens of pages and time to fix perhaps five. Sorting by the count of No answers is the obvious move, and it misleads: a page with 2 No answers out of 3 looks like a disaster, a long reference page collects fewer answers than a short guide, and the count says nothing about what the page is for or what the fix costs. The method below deals with each of those.
Start docs feedback triage with a minimum of 30 answers
Before you score anything, decide how many answers a page needs before its share of No answers counts. This post uses 30 as a rule of thumb: it's a working floor for a docs site with steady traffic, not a figure from research. At 30 answers, one more answer moves the share by about 3 points at most. At 3 answers, it can move it by 25. The sample size calculator shows how much a share can move at other numbers.
Put every page below the floor on a watch list. Widen the date range for those pages, or judge them as a group (all the pages for one SDK, say). Read their comments anyway: if a reader says a command no longer exists, you can check that in a minute.
Prioritising documentation fixes: score each page on four things
The score borrows from two existing methods. Jakob Nielsen's 1994 article on rating usability problems at Nielsen Norman Group combines how often a problem occurs, its impact on the people who hit it, and whether it persists. The RICE method, described in a 2021 Nielsen Norman Group article on prioritisation, multiplies reach, impact and confidence and divides by effort. For docs:
-
Share of No answers
No answers divided by all answers, over the same dates for every page. Only pages above your minimum get a share: the minimum is your confidence check.
-
Readers let down
The share of No answers times the page's views for the same dates, from your analytics. This corrects for pages where few readers answer.
-
Importance, 1 to 3
Give 3 to pages on the path every new user takes (install, set-up, signing in, the first API call) and to billing or your top support topics. Give 2 to common tasks and 1 to reference pages and edge cases.
-
Cost to fix, 1 to 3
An hour of work, such as a new screenshot or a renamed button, is 1. A day, such as rewriting a section or adding a missing step, is 2. A week or more, or a fix that needs an engineer to explain the feature, is 3.
The score is readers let down, times importance, divided by cost to fix. Write down what each weight means for your docs and use the same scale every time, so this month's list compares with next month's.
Tag the comments to find the kind of fix
The score tells you where to look; the comments tell you what to do. Tag every comment on the top pages with one of five reasons:
| Reason | What readers write | Usual fix |
|---|---|---|
| A step is missing | "Then what?", "it skips the part where..." | Add the step, and say what the reader should see after it |
| Out of date | "There is no Settings tab", "this flag was removed" | Match the current screens, names and commands |
| Wrong for my version or platform | "Doesn't work on Windows", "this is the v2 syntax" | Say which version the page covers, or split it by version or platform |
| Answer is there but hard to find | "Took me ages to find the timeout setting" | Better headings, the common case first, a short summary at the top |
| Not what I searched for | "I wanted to cancel my plan, not change it" | Fix navigation and search (next section) |
Then correct the cost in your score. A page tagged mostly "out of date" is usually an hour. A page tagged "a step is missing" may need someone who knows the feature, so it moves to 2 or 3.
Send readers on the wrong page to navigation, not a rewrite
Some pages collect No answers although everything on them is correct: search or a menu sent the reader there, and they wanted something else. Rewriting the page won't help them and can make it worse for the readers it does serve. The signs:
- Comments describe a task that another page already covers.
- Answers to a follow-up such as "What were you trying to do?" name a different feature from the one in the page title.
- The page shares words with a more popular page, such as "change plan" and "cancel plan", or "API key" and "access token".
The fix is a clearer title, a line at the top such as "Looking to cancel? See Cancel your plan", a menu link or a docs search synonym. It costs 1, so these pages score high. Expect their views to drop afterwards as readers go straight to the right page, so judge the fix by the share of No answers, not by traffic.
Example: docs for a deployment tool
Example
Say you run the docs for a command-line tool that deploys web apps. Four weeks after adding the question to every page, you have 300 No answers across 120 pages. You take page views from your analytics for the same four weeks. Six pages hold 160 of the No answers:
| Page | Answers | No share | Views | Let down | Importance | Cost | Score |
|---|---|---|---|---|---|---|---|
| Install on Windows | 90 | 40% | 6,000 | 2,400 | 3 | 1 | 7,200 |
| Authentication tokens | 120 | 25% | 12,000 | 3,000 | 3 | 2 | 4,500 |
| Plans and billing | 60 | 45% | 3,000 | 1,350 | 3 | 1 | 4,050 |
| Webhooks | 40 | 50% | 2,000 | 1,000 | 2 | 1 | 2,000 |
| Config file reference | 150 | 30% | 9,000 | 2,700 | 1 | 3 | 900 |
| Rolling back a deploy | 3 | - | 300 | - | - | - | Watch list |
By count alone, the config file reference would come first, with 45 No answers. Its comments ask for more examples across a long page: a week of work on a page readers look things up in. It drops to last.
The Windows install page comes first. Its comments say the installer download moved last month: an hour of work on a page every new Windows user reads. The billing page looks like a rewrite, but its comments say "how do I cancel?", so you add a pointer to the cancellation page at the top and "cancel" as a search synonym. With authentication tokens (a day of work), that makes three fixes this sprint. The rollback page, with 2 No answers out of 3, stays on the watch list, though you check its one comment ("the --to flag is gone") against the current release.
Four weeks after each fix, the install page has fallen from 40% No to 12%. The billing page has fallen from 45% to 20%, and its views have dropped by a third as readers who want to cancel go straight to the right page.
Check each fix over the same weeks
Note the date of every change, then compare the share of No answers over the same number of whole weeks on each side of it, away from a big release or a holiday.
- Keep the question and its place on the page the same, or the two periods are not comparable.
- Make sure the period after the fix also clears your minimum of answers.
- If the share has not moved, read the newest comments and tag them again. The page often had two problems, and you fixed the louder one.
Then run the score again with the new numbers. A fixed page drops down the list and the next one moves up.
Where Asklet fits
Asklet adds the Yes / No question to every docs page with one snippet, and a pattern such as /docs/** covers new pages too. Results show per page path: the share of readers who found the page useful, the reasons picked in a multiple-choice follow-up, and free-text answers as anonymous comments. Each survey change is a new version, so your before and after figures come from the same questions.
Asklet doesn't count page views, so take the reach part of the score from your analytics. Answers are anonymous, so you can't ask a reader what they meant: put a link to support on the page for readers who need help with their own case.
Your next steps
-
Export four weeks of answers and views per page
Answers and No answers from your feedback tool, page views from your analytics, all over the same dates.
-
Move every page under 30 answers to a watch list
Read its comments anyway, and check any claim you can test in a minute, such as a missing flag.
-
Write your importance and cost scales down
One line per value from 1 to 3, then score each page: share of No times views, times importance, divided by cost.
-
Tag the comments on the top ten pages
Missing step, out of date, wrong version, hard to find, or not what the reader searched for. Correct the cost to match.
-
Fix the top three, then compare
Note the date of each change and compare the same number of whole weeks before and after.
Try these steps on your own site. Asklet adds the questions to every page with one snippet. Compare the plans.
Start freeCommon questions
Should I prioritise docs pages by the number or the share of No answers? Neither alone. Multiply the share of No answers by the page's views to estimate how many readers it let down, and only give a page a share once it has enough answers (this post uses 30).
What if a page has many No answers but nothing on it is wrong? If the comments describe a task another page covers, readers landed on the wrong page. Fix the title, add a pointer to the right page at the top, or add a docs search synonym.
How do I know a docs fix worked? Compare the share of No answers over the same number of whole weeks before and after the change, with the same question in the same place and enough answers in both periods.