Open-Ended Survey Response Coding Tool

Start with three numbers: the people asked the question, the people who typed anything, and the answers left after you set aside non-answers. Decide what counts as a non-answer first, because it depends on the question.

Pew Research Center's codebook for its 2021 study of what makes life meaningful counted "none" and "nothing" as legitimate answers¹. Only a reply that couldn't be read as an answer at all, such as "I don't know", was a non-answer.

The item response rate is the share of people asked who answered, not counting anyone routed past the question. In the US, the Office of Management and Budget's 2006 standards for federal surveys set a benchmark. For any item used in a report, they say to plan a nonresponse bias analysis if the expected item response rate is below 70%². If yours is lower, compare answerers with non-answerers on the closed questions you have.

That comparison matters because the people who write are not a random slice. In the same Pew study, certain groups, often the more affluent and more educated, were more likely to answer and gave more detail³. Longer answers mention more topics, so those groups can seem likelier to raise rare ones, such as pets. Pew controlled for topics per answer before reporting any group difference.

Read a random sample of answers, not the first rows in the file (often your earliest respondents), and draft codes from what people actually wrote. Pew's researchers tested their draft on a random sample of 150 responses³, reviewed where coders disagreed and refined it. After three rounds, it was clear enough for other coders to apply consistently.

Give every code four things: a short name, a rule for what belongs, a rule for what doesn't (and where it goes instead), and one real answer as an anchor. If you track a theme from last year, start from last year's codes and note any rule you change.

Code the full set, and let an answer take every code it earns. "Too expensive, and the app keeps logging me out" is both Cost and Reliability. Pew's codebook gave each phrase one category but expected most responses to mention multiple categories¹.

Code the topic apart from the tone. Pew told coders to treat "I don't have enough money" the same as "I have enough money", and flagged complaints with a separate code for difficulties or challenges¹.

1. 1

   Hold back what fits nothing

   Don't force a near miss into the closest code. Set it aside and keep going.

2. 2

   Read the pile on its own

   Once everything is coded, read only the leftovers. Themes too rare to show up in your drafting sample can surface here.

3. 3

   Add a code only if it's earned

   If several leftovers share one idea, write a new code with its own rules, then recheck the coded answers for it.

4. 4

   Report what remains

   Give the final count and its share of codeable answers, and say what they were about.

Hand a colleague a random sample with your codes hidden, and have them code it from the codebook alone. Communication researchers Matthew Lombard, Jennifer Snyder-Duch and Cheryl Campanella Bracken recommend a reliability sample of at least 50 units or 10% of the full sample⁴. They add that it rarely needs to be larger than 300.

Measure agreement with a statistic that discounts chance, such as Cohen's kappa, and compute it for each code. Plain percent agreement flatters you: if a code is rare, two coders who never use it agree on almost every answer. The same researchers call .80 or greater acceptable in most situations⁴, and say .70 may be appropriate in some exploratory studies.

Pew's coders started coding on their own only after scoring at least 0.7³ on Krippendorff's alpha, a similar measure. When two coders disagree, rewrite the rule that caused it, not just the answer, and recode.

Every percentage needs a base: the group it's a share of. Open-ended items offer three honest ones, and they can differ by more than 10 points. Name the base in the same sentence as the claim.

Because one answer can carry several codes, the shares can add up to more than 100%. Say so, and give the average number of codes per answer. Set a minimum base before you look, such as 30 answers, and below it report the count instead of a percentage.

Choose quotes because they plainly fit the code, not because they're vivid, and print each next to its code's count. Pew states that its quotes were purposefully, not randomly, selected to illustrate key points³ and may not represent all responses. Say how you chose yours.

Then carry the counts, bases and quotes into a survey results report that keeps each group's base beside its numbers.

Open-ended answers catch what a list of options would miss. After the 2008 US presidential election, Pew asked voters which issue mattered most. When given a list, 58% chose the economy; only 35% named it unprompted⁵, and 43% of those asked the open version gave an answer the list didn't include.

Pew notes that researchers sometimes pilot open-ended questions to find the most common answers⁵, then offer them as choices in a closed-ended version. Your biggest codes are candidates for next year's answer options, and the leftovers show what your questions don't yet ask. Draft them in a survey questionnaire design template that ties each question to the analysis it feeds.

For a faster first pass, use the tool at the top of this page. River drafts the codebook on a random sample, codes every answer, counts the leftovers and marks a check sample for a second coder.