Survey Results Report From Your Data

Put the answers first and the detail at the back. AAPOR, the American Association for Public Opinion Research, notes that a complete report usually carries a detailed methodological appendix that shorter summaries can refer to¹.

Give senior readers a one-page executive summary if the report runs long. Code free-text answers into counted themes before you report them, as this guide to analyzing open-ended survey responses explains.

Open the method with the kind of sample you have, because it decides what you can claim about precision. AAPOR's disclosure standards ask researchers to state whether the sample was chosen by probability methods or came from opt-in, volunteer or other sources¹. They also ask for the dates, the number of responses and the weighting.

• A random sample from a full list, such as 4,000 of your customers: report a margin of error.
• Everyone invited, as in most staff surveys: the response rate and who stayed silent matter more than any margin.
• An opt-in panel or an open link: report the bases. AAPOR's code allows a measure of precision only with a detailed description of the model behind it¹.

Then give the response rate and check who's missing. Guidelines for US federal surveys from the Office of Management and Budget (OMB) call for an analysis of nonresponse bias when fewer than 80% respond². A simple version compares respondents with everyone invited on facts you hold, such as region or length of service.

The base is the number of people behind a percentage. Print it beside every figure, in tables and on charts: 60% of 1,000 people and 60% of 15 are different findings. AAPOR's code also commits researchers to sharing the unweighted sample size behind each subgroup estimate¹ on request.

The margin of error shows how far from the true figure a result can reasonably fall by chance alone. In a Pew Research Center example, a simple random sample of 1,067 people has a margin of error of ±3 points, and a subgroup of about 160 has ±8 points³.

Weighting makes margins wider, an increase statisticians call the design effect. Pew warns that pollsters who ignore it are claiming more precision than their survey actually warrants³. Divide your responses by the design effect to get the effective sample size, and work out margins from that. From the weights alone, that size is the square of their total divided by the sum of their squares.

Fix the smallest group you will report before you see any results. The same OMB guidelines say to decide when an estimate's error is too large to publish before you produce any estimates². The US National Center for Health Statistics suppresses a percentage when fewer than 30 people stand behind it⁴, or when the effective sample size is under 30.

Keep suppressed groups in your tables, marked with the reason, so readers can tell a small group from a missing one. Next time, forecast each group's size before the survey goes out, as the survey questionnaire design template does.

Before you write that one group differs from another, test the gap. The OMB guidelines say to make comparison tests before stating that two estimates differ², and when a test fails, to conclude that the data don't support a difference.

The margin on a gap is wider than either number's margin. For two separate groups, square each margin, add them and take the square root. Don't judge by whether error bars overlap: the US Census Bureau warns that this method will not always provide an accurate result⁵.

Ranking words such as highest, lowest, most and least also claim a difference, so test the top two (or bottom two) groups before using one. Changes since your last survey need the same test. In one Pew example, a 3-point shift in a candidate's lead between two polls had a margin of error of ±8 points³.

Make each heading the finding itself ("Most customers would accept a 5% rise", not "Pricing"). Put the number, its base and the comparison in one sentence, so anyone who quotes it keeps the context. Report what the data shows, not why it happened: a survey can show that two things go together, but rarely that one caused the other.

When a change sits inside its margin, say so plainly. The UK Government Analysis Function suggests phrases like "broadly stable" when a movement isn't statistically significant⁶. It also warns that "exercise caution" on its own doesn't help readers, so tell them what they can and can't conclude.

Give every chart a headline title that states its message and a subtitle saying what the data is and when it was collected. The Analysis Function's chart guidance says the numerical axis should never be broken on a bar chart⁷: start it at zero, because readers compare bar lengths. Label the base under each bar, and use bars rather than a pie when slices are close in size.

For a faster start, use the tool at the top of this page. River builds the tables from your data, works out each group's margin and tests every comparison before writing the findings.