Sentiment Analysis: When CSAT Surveys Let You Down

It’s always hard to gauge how happy your customers really are.

Customer satisfaction has some big limitations.

It varies a considerable amount depending on the nature of each request. Firstly, customer satisfaction (CSAT) is easily distorted by simple issues that are solved very quickly. Secondly, you can’t be sure that your customers are rating what you want them to rate - are they rating the quality of the service or are they, instead, rating the overall experience with your product or with your company?

CSAT Surveys are a good indicator of how happy your customers are, but they really are only an indicator.

Firstly, customers don’t always return CSAT surveys. It is said that only 5-30% of surveys are returned. This could present 2 problems: 

❌ Not enough data: everyone knows that the more data you have, the more accurate your conclusions will be. When you have a small % of returned CSAT surveys, you are likely to be working with a less-than-accurate representation of your customers’ true feelings.

❌ The 5% rule: You should take into consideration that your 90% CSAT from surveys may be anywhere between 85% and 95%. But if you have a lower percentage of surveys returned, perhaps this deviation could be even higher.

Considering these two problems, is your current CSAT score just a 'vanity metric'?

Businesses that exclusively use customer satisfaction scores have a churn rate of 60-80%, even though they were satisfied or very satisfied in their last CSAT survey.

Fred Reichheld

CSAT should never be used to measure your customer’s sentiment by itself, so it’s commonly coupled with NPS (Net Promoter Score) or CES (Customer Effort Score). It gives a much better indication, but the problem remains.

Only 1 in 26 unhappy customers complain… and 91% of unhappy customers who are non-complainers simply leave.

You need a way of analyzing what every customer is saying, especially the ones that aren’t giving feedback.  

From vanity metric to actionable insights

Customer sentiment analysis will give you a holistic view of how happy your customers are.

✅ It tells you how happy your customers who don’t return surveys are

✅ It’s no extra effort for customers, so it doesn’t detract from the customer experience 

✅ It provides more representative data of your whole customer base than surveys

Of course, it’s not foolproof, but it can be incredibly useful for bringing you closer to your customers and understanding a) what they are very happy with; and b) where their frustrations are coming from. When used in conjunction with survey results and KPIs, customer sentiment scores can be very powerful.   

How does customer sentiment score work?  

Sentiment analysis essentially detects whether a customer is happy or unhappy after an interaction with your support team

Here's one way to do it.

Goal: to have a score for each most recent customer message in each ticket to find whether they are happy, unhappy or neutral. 

To achieve this, you will have to score each word on how positive or negative a customer is in their writing. And don’t forget emojis 😃 they can also be a great indicator of how happy your customers are. 

Adjectives and verbs are given more weight in the analysis because they are the most important factor in understanding a customer’s true feelings. Emojis are also a good measure of customer sentiment or customer satisfaction.  Sentiment analysis essentially detects whether a customer is delighted, happy, indifferent, unhappy or angry after an interaction with your support team. 

Let's look at 2 customer messages:

This is extremely frustrating. I've tried for hours to get a ride, while saying it should just be minutes away and getting denied forever. Worthless service 😕

Customer sentiment analysis - Bad

Now lets look at a more cheerful message:

Miuros is a great asset for our whole team, our agents love it 😁

Customer sentiment analysis - Good

Messages are considered as positive, negative, or neutral depending on syntax, semantics and the sentiment score of each word. Neutral messages are usually factual descriptions of the user experience while the customer perspective is made up of either positive or negative words.

This will allow you to have a view of all the negative interactions and break them down by contact reason, team, channel or any other dimension. By doing this, you can really pinpoint where your customers are dissatisfied and act.  

But it’s worth considering that....

Language is very complex, and highly subjective. 

In the UK, 'that’s sick!' is a really positive phrase… The same is true for ‘painfully good’, for example.

One of the drawbacks is that we won’t know the difference between ‘sick’ being used as a positive and ‘sick’ being used as a negative. In large messages, this won’t problem (the rest of the message will reflect a positive score despite us reading ‘sick’ as negative), but shorter messages with similar slang will make scores less accurate. 

So.... The reality is that it’s impossible to gauge a customer’s sentiment correctly every single time, because everyone has such a different way of expressing themselves. Having said that, even CSAT is an inaccurate indicator of customer happiness when used by itself. It’s all about using all the data points you have to bring you as close to your customers as possible. Sentiment score is a great weapon when used with CSAT, NPS or CES.  


How to get up and running with customer service sentiment analysis

Firstly, you will need some way to process and analyze your data. There two ways to do this:

  1. Manually: You can compile a list of words that you think express positive or negative sentiment and give them scores based on how powerful they are. You won’t catch every single word but if you have a strong list it could prove to be valuable. There are a few open source libraries that can show you how to score tickets and how to weigh certain words. This will be an arduous process but, when done right, can produce good results. Bear in mind, it might well involve work from developers.

  2. Machine Learning: This will require a lot of help from developers for sure! Read and code comments manually (e.g. using Python): assign scores to particular words or phrases that suggest a positive negative or neutral sentiment, and weigh them based on how strong the feeling is. This could well pay off in the long run, but it requires a lot of set-up work.

  3. There are analytical solutions for customer service that provide out-of-the-box customer sentiment analysis as part of their offering. This could be a good option if you could benefit from sentiment analysis but you don’t necessarily have the developer resources to build it yourself.

Find an analytical solution that provides customer sentiment analysis automatically.

Once you have the foundations set up, you need to start thinking about how to use these actionable insights. 

3 Practical Applications of Sentiment Analysis

Listening to customers that don’t return surveys

It’s a very effective thermometer that tells you how your customers are feeling. You can break this down per any dimension: what channels are customers having a hard time with? Which contact reason brings the most pain? What product brings the most negative responses? 

Getting to why your customers are unhappy

After you’ve taken their temperature, you can diagnose and prescribe treatment. For example, this product is bringing a lower sentiment score, so let’s alert the product team to move the improvements up on their roadmap. It may result in huge customer retention gains.

Tracking customer happiness

To continue the metaphor - next comes the rehab. How are you going to make sure there is continual improvement in the customer experience? Sentiment is a great indicator of how your customers are responding to changes in your products and services over time.  

Want to have similar content delivered right to your inbox? Sign up to our monthly newsletter