Analyzing Open-Ended Survey Comments: 5 Steps From Codebook to Action

Analyzing the open-ended comments in an employee survey means finding the themes that keep coming up and the issues that are new, then reading them alongside scores and what’s happening in the organization to decide what to improve. Below are five steps, from setting the question your analysis needs to answer to naming who owns each action. Use AI to surface themes, code comments, and summarize them; make the important interpretations and decisions about action yourself, checking them against the original comments and your organization’s context.

Say the survey closes, you read thousands of lines of comments, and you still end up at “communication is the problem, again.” What should change next time? The starting point is to go one step past collecting comments that use the same word. Drawing on Braun and Clarke’s 2006 paper, which explains how to identify and refine recurring themes in responses, here is a sequence for turning what employees say into action. [1]

1. Name the Decision Before You Read

Before you start, write down in one sentence the decision these results need to inform. Are you choosing this year’s two culture priorities, trying to understand what’s going on in a division where turnover has risen, or gauging reactions to a new performance review system? That question tells you which comments to read closely and how finely to break them down.

If you’re looking at reactions to a new review system, for instance, you might split the broad theme of “compensation” into pay levels, evaluation criteria, and how results are explained. To keep the analysis from ending as an exhaustive list of everything you read, also note what each theme tells you about the question you started with. Anything you can’t answer yet becomes a question for follow-up.

If you’re still designing the survey, apply the same thinking to your open-ended questions. Instead of “Please share any other comments,” ask about a real experience: “Describe one moment from the past three months when work was hardest for you.” If this is your company’s first survey, limit it to two or three open-ended questions and decide up front which decision each answer will inform.

As AI gets better at analyzing open-ended comments, people’s work doesn’t so much shrink as move. AI handles the middle of the process quickly, finding recurring themes and refining the codebook, and the studies below show it performing close to human analysts when it sorts comments into set categories. When it comes to building new themes or reading between the lines, though, results still vary widely. So people’s time should shift to the two ends: deciding at the start what decision you’re reading for, and judging the results against your organization’s realities at the finish. The more of the middle you hand to AI, the more firmly people need to own the start and the finish. Otherwise you just reach “communication is the problem, again” faster.

2. Use AI to Find Themes and Refine Your Coding Criteria

There are two ways to find themes. You can start from your existing survey dimensions or your analysis question, or you can read the responses and let new meanings emerge. Thematic analysis as Braun and Clarke describe it covers both approaches. You don’t need a finished, hand-built codebook before you can begin. [1]

Today’s LLMs can propose themes and their definitions, group similar comments, and separate experiences that differ. Start by asking AI to list recurring themes along with the responses that support each one. Have it set aside comments that don’t fit any existing category, too, so a predefined framework doesn’t make you miss new issues.

A 2026 study in PLOS Digital Health compared how models including GPT-5 and Claude 4 Sonnet analyzed healthcare focus group transcripts with how human analysts did. When the task was coding against predefined codes, the two models matched expert consensus about as well as the human analysts, or slightly better. When the task was generating new themes, results varied more from model to model. The findings show that AI can take on much of the work of qualitative analysis, and also that, depending on the task, its results still need checking. [2]

There is also recent work closer to open-ended survey data. A preprint released in August 2026 compared how five human coders and GPT-5.4 analyzed 903 responses from a survey of doctoral students in Europe. The researchers observed agreement between the humans and the model in both coding and theme generation, and concluded that, for this dataset, the model could come close to human analysis. It is still a preprint, and agreement varied widely depending on the question analyzed. [6]

If you’ll compare these themes across survey rounds, or several people will use them, write down the coding rules. A table that lists each theme’s definition, what it includes and excludes, and a sample quote is called a codebook. AI can draft one, and you can revise it as you read more responses. If you have only a small number of responses, it’s fine to start by reading them all and documenting just the themes you need.

Within a “communication” bucket, for example, you can separate hearing the reasons behind decisions too late, information not flowing between teams, and finding it hard to disagree with a manager. Refine the boundaries between themes by asking what behavior would need to change. If one response describes several experiences, tag it with several themes, and consider a new theme for comments the existing categories don’t explain.

Check AI’s coding against the original comments. Start with borderline responses, new themes, and comments that could call for a serious response, then read some of the responses grouped under each theme to see whether their meaning holds together. If you change a rule, reapply it to the responses you’ve already coded. Rather than using the same number of reviewers and responses every time, you can scale the review to the volume of data and how much rides on the judgment.

3. Read Intensity and Target, Not Just Counts

Say you’re reviewing your team’s comments: 30 are about the cafeteria menu, and 8 say people worry they’ll be penalized in their performance review if they raise a problem. Alongside the counts, look at what experience each comment describes and what kind of response it calls for. In the report, too, you can list each theme’s count side by side with what needs a closer look.

As you read, try recording four things: intensity, target, specificity, and suggestion. Is the person describing an annoyance, or saying they’re thinking of leaving? Is the comment aimed at a leader’s behavior or at a policy? Then look at whether it describes a concrete experience and what change the person wants. With those four noted, you can write more specific follow-up questions, even within a single theme.

Even among responses coded as “dissatisfaction with pay,” for example, read comments about pay levels separately from comments saying people don’t understand the evaluation criteria. The first are a starting point for examining pay levels and budget; the second, for examining how the criteria are explained and the process behind them. Even if you’ve tagged comments as positive or negative, go back to the original text before deciding on actions, and check what employees wrote about and what they experienced.

4. Read Comments With Scores and Check Team Context

If workload comes up repeatedly in a division where engagement scores fell, for example, you can write down the link between workload and the drop as a hypothesis to explore. Telta’s Oct. 11, 2026 article likewise suggests looking more closely at behaviors and context when you read scores and comments. The idea is to keep the experiences confirmed in employees’ responses separate from the judgment that those experiences affected the score. See our article on interpreting culture survey results.

When you compare two departments with similar scores, look at the situations people describe in their comments as well. If only one of them keeps mentioning how its leader makes decisions, prepare questions to find out which decisions people had in mind. Record where scores and themes show up together, then explore that connection further in follow-up conversations with employees.

When you decide how to report by team, set a minimum number of respondents and rules for quoting comments at the same time. For example, you might consider folding teams of fewer than five into their parent division. In the comments themselves, look for combinations of job level, project names, and specific incidents that could let someone guess who wrote them, and weigh both meaning and anonymity when you choose which sentences to share.

To pick quotes for the report, go back to the responses grouped under each theme. Rather than starting with the harshest or most eye-catching lines, look for sentences that show why those responses were grouped under that theme. Next to each quote, summarize what came up repeatedly in that theme and any differing views, so readers can see how a single sentence relates to the responses as a whole.

5. Set Hypotheses to Test and Assign Owners

Now it’s time to turn your themes into next steps. If you’re dealing with comments saying people don’t hear the reasons behind decisions in advance, start by writing down which situations to check. Has announcing decisions only after they’re made become a habit? Do explanations get lost as decisions from higher up are passed down? Look into it, and decide what behavior to change based on what you find.

To turn this into a plan, you might, for example, commit to this behavior: “Before finalizing a decision that affects the team, briefly share the reasoning.” Make the leader of that team the owner, and schedule a short follow-up pulse survey two months later that asks the same question again. Then look back, in order, at the hypothesis you started with, what was actually tried, and what changes employees noticed.

Build in a date for sharing results back with employees, too: explain which comments you read, what you’ll change first this quarter, and when you’ll check again. Telta’s Oct. 12, 2026 article covers deciding who will interpret the results, how much time they’ll have, and who in the business owns the follow-through. For more on who should own each action and where follow-up checks are needed, see “Culture Assessments: The Report Is In, So Why Has Nothing Changed?”

After the results briefing, the conclusion is often, “Communication got the most mentions, so let’s start with company-wide communication training.” But as step 2 showed, the communication comments lump together teams that hear the reasons for decisions too late, divisions where information is stuck between departments, and groups where it’s hard to disagree. One training program is unlikely to change things for any of them. Meanwhile, a handful of comments saying “If you raise a problem, you’ll pay for it” are a signal to address first, however few there are. Decide what to tackle first not by count but by weighing each comment’s intensity, its target, and where in the organization it came from and under what circumstances.

Interpret With AI, Then Decide What to Act On

You can hand AI everything from proposing themes and drafting the codebook to coding, summarizing, and finding representative responses. Linking each result to the original comments behind it makes the interpretation easier to check. When themes are merged or split, ask AI to explain why, and look at the minority views and alternative interpretations the summary left out.

When you move from analysis to action, define the behaviors the company can change and the decisions that need to be made. Treat AI’s suggested priorities as input, but weigh them against how serious the comments are, the organization’s circumstances, and who is responsible for what. It’s best to give comments reporting harassment or unfair treatment their own owner and process, separate from the theme counts.

When you wrap up an open-ended analysis, leave behind more than responses grouped by theme: list the questions to dig into, the behaviors to change, the owners, and when you’ll look again. Put that list next to the analysis question you started with, and you can separate what these results let you decide now from what needs follow-up conversations. In the next survey, read the responses against the same criteria to see what has changed.

Telta’s Survey Intelligence groups open-ended responses from culture and leadership surveys by theme and delivers them in reports that let you read the story behind the scores. To see what the company-wide and team-level reports include, talk to our team. [3]

‍

Related Reading

• Culture Survey Results: Why “Communication Is the Problem” Changes Nothing
• Culture Assessments: The Report Is In, So Why Has Nothing Changed?

References

1. Braun, V. & Clarke, V. · Using thematic analysis in psychology · 2006. Qualitative Research in Psychology, 3(2), 77–101. — Defines thematic analysis and approaches to finding themes in data. Based on the abstract in public search results and a public excerpt in a university repository.
2. Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts · PLOS Digital Health · Apr. 3, 2026. — Compares LLMs with human analysts on deductive and inductive analysis of healthcare focus group data. Coding metrics and theme-generation results are cited separately.
3. Telta · Survey Intelligence product page (in Korean) · No publication date; accessed Nov. 1, 2026. — Theme-based summaries of open-ended responses and company-wide and team-level reports.
4. Telta · Culture Survey Results: Why “Communication Is the Problem” Changes Nothing · Oct. 11, 2026. — How to read scores together with context and test hypotheses about causes.
5. Telta · Culture Assessments: The Report Is In, So Why Has Nothing Changed? · Oct. 12, 2026. — Recommendations on interpreting results, line owners, and follow-up checks.
6. Bergmann et al. · LLMs for Survey Text Analysis — A Performance Comparison Between Humans and GPT-5 on Inductive Content Analysis · Aug. 23, 2026. Preprint. — Compares human coders with GPT-5.4 on coding and theme generation for 903 open-ended responses.