How-To Guide12 min read
How to Track Customer Sentiment Over Time for a Feature or Theme
Last updated: · By Matt Teixeira, Founder and CEO
On this page
To track customer sentiment over time for a specific feature or theme, tag every mention of it to one fixed theme and label each mention positive, neutral, or negative. Each week, chart two numbers side by side: how many accounts raised the theme, and what share of them were negative. Compare every week with an eight-week baseline and a log of releases, and ignore any move smaller than your noise band.
Never read the share without the count: it can double because happy customers stopped writing in. Use NPS and CSAT as a cross-check, since they measure the whole relationship or one support interaction. Deck, Enterpret, Thematic, and Qualtrics Text iQ chart sentiment by theme; under a few hundred comments a month, a spreadsheet and an LLM are enough.
Key Takeaways
- Read sentiment and volume together. Rising negative share on falling volume often means happy customers went quiet.
- Count accounts, not mentions. A customer who writes in six times is one data point.
- Use text sentiment for the theme and scores for the relationship, linked by a theme NPS gap.
- Know your noise band: a weekly share swings about 20 points by chance at 25 accounts, and 10 at 100.
- Re-baseline after any survey, bot, or classifier change. Tickets skew negative, so a shift from surveys to tickets looks like a decline.
Step 1: Define a Theme That Can Move Both Ways
Sentiment tracking only works on a theme that can collect praise as well as complaints. Name it after a feature or product area, such as "Dashboards." A theme named after a problem, such as "Dashboards load slowly," is negative by definition, so its sentiment line stays flat however bad things get.
Track sentiment on feature themes and volume on problem themes. Before you count, write a one-sentence scope, two examples that belong elsewhere, and the problem subthemes inside, such as "Slow load." Subthemes tell you why the line moved.
Freeze the definition for at least a quarter. If you split or merge the theme, relabel the past eight weeks (how to build a customer feedback taxonomy covers how).
Step 2: Tag Every Source the Same Way
Use one theme list and one classification prompt or tool across tickets, survey and NPS comments, call notes, and Slack. Label sentiment per mention, not per message. "The new editor is great, but exports keep failing" is positive for the editor and negative for exports; scoring the ticket as mixed helps neither.
Record four fields per mention: the date the customer said it (not the import date), account ID, source, and sentiment. Add a type, such as bug, pain point, or feature request, if your tool provides one.
No model is right on every comment; even Qualtrics says Text iQ "is the most powerful in aggregate" (Qualtrics). As a rule of thumb, hand-check 20 random labels a month; if more than three are wrong, fix the prompt before you trust the trend.
Step 3: Choose Between Scores and Text Sentiment
Make text sentiment on tagged mentions your main line. NPS asks whether someone would recommend your whole product, so one feature rarely moves it.
| Method | What it measures | Weak spot for one feature |
|---|---|---|
| NPS (0 to 10) | Loyalty to the whole product | Barely moves; shifts with when and where you ask |
| CSAT (1 to 5) | Satisfaction with one interaction | Rates the support conversation, not the feature |
| In-app feature rating | One feature, right after use | Low response; only active users answer |
| Text sentiment on tagged mentions | How customers feel when they raise the theme | Needs consistent tagging; shifts with channel mix |
Link scores to the theme with a theme NPS gap: the NPS of respondents whose comments mention the theme, minus everyone else's. In a made-up example, if dashboard mentioners score 10 and everyone else scores 38, the gap is minus 28 points.
Check the gap monthly. If it narrows after a fix, the fix reached the people who cared enough to say so. The Typeform guide shows the same idea per survey, splitting each theme by detractors and promoters. To find the themes behind your detractors, use the NPS breakdown playbook.
Step 4: Set a Baseline and Know Your Noise Band
Before you call any change, know how much the line moves on its own. Backfill eight weeks and record two averages: accounts per week and their negative share.
As a rule of thumb, a share built on n accounts can move by about 1 divided by the square root of n by chance alone, roughly a 95% margin of error.
| Accounts per week | Chance swing in negative share | How to read it |
|---|---|---|
| 25 | About 20 points | Use a rolling four-week window |
| 100 | About 10 points | Read weekly; ignore smaller moves |
| 400 | About 5 points | Read weekly, with room to split by segment |
NPS is noisier, because it subtracts one share from another. On 100 responses with 40% promoters and 20% detractors, the margin of error is about 15 points, so "NPS fell 8 points" is not news.
Call a shift real when it clears the noise band, or when it moves the same way three readings in a row.
Step 5: Chart Volume and Sentiment Together
Every week, put both on one chart: accounts as bars, negative share as a line. Either alone misleads. Add positive share if you can; it catches the feature customers have started to love.
| Accounts | Negative share | It usually means | Do this |
|---|---|---|---|
| Up | Up | A problem is spreading | Read this week's quotes and find the subtheme |
| Up | Flat or down | More customers use it, mostly happily | Find the minority complaint before it grows |
| Down | Up | Happy customers went quiet; the stuck ones remain | Count negative accounts, not just the share |
| Down | Down | It got fixed, or customers gave up | Check usage and churn before you celebrate |
Once the chart builds itself, a Monday review of five to ten themes takes about 15 minutes. The weekly signal brief playbook gives it a format, and how to tell which support ticket pain points are trending has the volume spike rules.
Step 6: Segment Before You Trust the Average
A blended sentiment line can fall without a single customer getting less happy. If a sign-up push brings in new accounts, and new accounts always struggle more with setup, the average drops on mix alone. Split a moving theme by plan or ARR band, account age, and source channel.
The channel split catches the most false alarms. Customers open tickets when something breaks and answer surveys when asked, so tickets run more negative. Pause an in-app survey and sentiment drops while the product stays the same.
Track sentiment per channel, or hold the channel set fixed. Stop at two or three cuts: ten themes across six segments is 60 lines, and a few will jump by chance every week.
Step 7: Annotate Releases and Collection Changes
Most sudden moves on a sentiment chart trace back to a date, so log the dates next to it. Write one line for:
- Product changes: releases and flag rollouts that touch the theme, pricing changes, and outages
- Collection changes: survey wording, trigger, or audience; a new support bot; a new model, prompt, or theme definition
These alter what you measure, not how customers feel. Moving NPS to an in-app prompt after a success moment can lift the score overnight, and a bot that answers easy questions leaves angrier tickets for your team. Start a fresh four-week baseline after any of them.
For a release, wait at least 14 days and compare equal windows; how to measure whether a release reduced customer complaints covers the full check.
Worked Example: A Dashboard Redesign, Week by Week
This example is illustrative, with made-up numbers. A B2B SaaS team tracks "Dashboards" across tickets, NPS comments, and call notes. The baseline is 60 accounts a week at 30% negative, so the noise band is about 13 points.
| Week | Accounts | Negative share | Positive share | Logged event |
|---|---|---|---|---|
| Baseline (8-week average) | 60 | 30% | 35% | None |
| Week 1 | 110 | 48% | 30% | Redesign ships to all accounts |
| Week 2 | 95 | 45% | 33% | Hotfix restores saved filters |
| Week 4 | 75 | 33% | 41% | None |
| Week 6 | 65 | 25% | 45% | None |
Week 1 is a real problem. Negative share is 18 points above baseline on 110 accounts, well outside the noise, and both lines rose. The rise is in tickets, and the subtheme is saved filters lost in the redesign.
The segment split shows who. Accounts with 25 or more seats run 70% negative against 32% for smaller ones, because big teams had built the most filters.
The recovery is real too. By week 4, negative share is back inside the noise band. By week 6, positive share has risen three readings in a row to 10 points above baseline.
Overall NPS moved from 41 to 39 on about 120 responses a month, inside its noise band of about 13 points. The score missed both the regression and the recovery; the theme line caught both.
Common Mistakes That Create False Signals
- Reporting a share without its count. "Negative share doubled" on four accounts is two unhappy customers.
- Letting one account drive the line. A single escalation can produce 15 tickets in a week. Count it once.
- Trusting a sudden jump in neutral. When neutral grows overnight, the classifier usually changed, not your customers.
- Dating mentions by import. A 90-day backfill dated by import lands in one week and looks like a spike.
Tools That Help
A spreadsheet handles this up to a few hundred mentions a month. Past that, consistent tagging across sources breaks first, which is what feedback platforms fix.
| Option | Sentiment over time | Limits | Cost |
|---|---|---|---|
| Spreadsheet plus an LLM | Your prompt per mention, a weekly pivot chart | You rerun exports and audit labels yourself | Free to low, plus your time |
| Survey tools, such as SurveyMonkey | Sentiment per open-text question | One survey at a time; no tickets or calls | Some paid plans |
| Qualtrics Text iQ | Topic sentiment compared across two periods | Widgets need Advanced Text | By quote |
| Thematic | Theme sentiment, impact on score, score changes | Works best with 2,000+ comments | From $25,000 a year |
| Enterpret | Sentiment per aspect, theme trends, anomaly detection | Built for high-volume enterprise teams | $120,000 a year on AWS Marketplace |
| Deck | Theme sentiment timeline, subtheme trends, rolling 30-day NPS | Three labels, no intensity score; NPS and subthemes need Starter | Free; Starter $99 a month |
- Survey tools: SurveyMonkey scores open-text answers per question on "some paid plans" (SurveyMonkey Help).
- Qualtrics: the Top Sentiment Change widget compares topic sentiment "across 2 time periods," for Advanced Text clients only (Qualtrics Support); pricing is on request (Qualtrics pricing). It fits when your feedback already lives in Qualtrics.
- Thematic: Foundation is $25,000 a year for up to 25,000 comments across 3 datasets and works best with at least 2,000 rows (Thematic pricing). Pick it to learn which themes move NPS or CSAT.
- Enterpret: its own guide says it scores sentiment per aspect and tracks themes over time (Enterpret) across "50+ sources" (Enterpret), with an adjustable anomaly cutoff (Enterpret Help Center). AWS Marketplace lists $120,000 for 12 months and up to 300,000 records (AWS Marketplace).
Where Deck Fits
Deck handles the tagging, charting, and segment splits from Steps 2, 5, and 6 across tickets, calls, surveys, NPS, and Slack.
- Per theme: each theme's Overview has a sentiment timeline plus sentiment and category breakdowns, which separate Pain Points and Bugs & Errors from Feature Requests (Deck themes docs). The Themes list filters by sentiment and date range.
- Subthemes: on Starter and up, each subtheme's trend is colored by impact: green when positive feedback rises or negative falls, red when things get worse (Deck subthemes docs). Trends compare the last 14 days with the 14 before (Deck MCP docs).
- NPS and segments: on Starter and up, NPS Score plots a rolling 30-day score every 7 days, with promoter, passive, and detractor trends and their top themes (Deck NPS docs). The Segment filter re-counts Themes and NPS for one group (Deck segments docs).
- After a release: a completed Project keeps attaching new matching feedback, which gives you the after-window (Deck Projects docs). The release impact check playbook runs the before-and-after comparison.
- Limitations: sentiment is three labels with no intensity score, and you can't pin release markers on Deck's charts, so keep your event log beside them. Trend views count insights, not unique accounts, aren't divided by feedback volume, and there are no custom alert thresholds. The theme timeline groups insights by month, using the ticket or interview date when there is one, but subtheme trends date them by when Deck processed them, so a large backfill can look like a subtheme spike.
- Pricing: Free is $0 with 200 AI credits and 1 integration. Starter is $99 a month ($79 billed annually) with 2,500 credits, roughly 830 tickets or NPS responses at Deck's planning baseline.
To see the sentiment timeline on your own themes, start with Deck's Free plan.
Frequently Asked Questions
Sources
- Qualtrics Support: Text iQ best practices
- Qualtrics Support: Widgets in Text iQ
- Qualtrics pricing
- SurveyMonkey Help: Sentiment analysis
- Thematic pricing
- Enterpret: The 6 best tools to track sentiment shifts over time in 2026
- Enterpret: Customer feedback integration
- Enterpret Help Center: Anomaly
- Enterpret on AWS Marketplace
- Deck themes docs
- Deck subthemes docs
- Deck MCP docs
- Deck NPS Score docs
- Deck customer segments docs
- Deck Projects docs
- Deck pricing
Sources checked November 2, 2026.