The Net Promoter has been hot recently. Simple to measure it is a loyalty score which is supposed to signal future sales and correlate to value creation. Defined, it is a simple sum of the number of people that promote your brand minus the number of brand detractors. Companies that use it, like it because it is straightforward and requires just one question to their customers - “Would you recommend us to a friends or colleague?”
Originally conceived by loyalty consultant Fred Reichheld and supported by Bain and Satmetrix it has whirlwinded into the minds of CEOs with companies such as GE and American Express using it and apparently on conference calls with analysts. Although it is intuitive that knowing recommenders and detractors has value, it does seem to ignore other factors for market success or value creation. What about slightly important issues such as context of recommendation, pricing, market timing, supply, competition, availability of substitutes, advertising etc. What is seems to suggest is that all of these factors cancel each other out or are set to zero?
Although I am sceptical of the value of the measure in isolation, I go along with the hypothesis that identifying recommenders and detractors is important (in our case finding this in an online context). However over at IPSOS-MORI, they are completely dismissive. Once they put the score to scientific scrutiny (with help from Vanderbilt University,Norwegian School of Management & Koc University) they showed the measure to be largely spurious and even less predictive of future value than the American Customer Satisfaction Index (ACSI) - this is especially controversial as Reichheld states their examination shows that ACSI has no correlation with firm growth…
Conclusions from “The Net Promoter Debate” - an interview with Tim Keiningham Senior VP, Ipsos loyalty (May 2007). Full version downloadable here.
1. “Simple put, we found no support for the assertions attributed to Net Promoter. Our research clearly shows that claims of Net Promoter’s superiority in predicting firm growth, or in predicting customers’ future loyalty behaviours are false. Based on the evidence we’ve compiled, it’s hard to imagine a scenario where Net Promoter would be classified as a superior metric”
2. “As complaints about the effectiveness of Net Promoter have begun to surface, supporters argue that detractors are missing the point; Net Promoter is simple and gets companies to focus on customers. That may well be true, but the research used to support Net Promoter focused exclusively on growth… in particular, Net Promoter’s superiority in linking to growth. Few managers would adopt a metric just because it is simple. Simple and wrong is simply wrong.”
OK, so this debate is hardly new news, with plenty of heated discussion around the topic, but it is scary if the conclusion is that a score presented at calls with analysts is proven to be baseless. In my opinion it should at least get a more thorough review regards predictiveness and although companies should measure who is recommending them (and why) and who detracts them (and why) it may be prudent not to rely too much yet on just one number…
October 21st, 2007 at 11:56 am
I agree we should always be careful with basing our judgment on just one number. But if you read the book (The ultimate question), this is exactly what Reichheld wants to avoid. Today companies (especially big ones) are run on numbers, and these number never take into account the sentiment of the customer. It’s just sales, conversion, retention, margin, etc.
Customer sentiment is the result of many things influencing customers such as: context of recommendation, pricing, market timing, supply, competition, availability of substitutes, advertising, etc. It’s for sure not perfect, but nothing is in human sciences.
The most important thing about net promoter score for me is that it’s a simple metric that can be measured objectively and integrated in company dashboards that are used on a daily basis. And as it’s backed up by a Harvard Business school book, Bain & Co and reputed academics the chance that companies start using it is considerable. The alternative is that customer sentiment is not taken into account at all …
So pragmatically speaking, it’s probably net promoter score or nothing.
October 21st, 2007 at 9:17 pm
Hi Bert,
Thanks a lot for your reply.
Just to clarify, we are big believers in the importance of customer sentiment and measuring/reporting it (disc: we have built algorithms that detect this). What I am worried about is if CEOs are reporting an NP number that is supposed to correlate with value creation (market capitalisation) and this becomes a kind of “gospel”. It probably needs more testing before this happens.
In cases like Dell, a number that shows recommenders minus detractors makes sense given serious customer service issues. They want to know if they are improving. In their case is there any proof that their recent customer service investment has changed market value of the company? I would argue that ACER and HP competition has more to do with their future market value than NP.
In conclusion, we should keep measuring sentiment of customers and reporting this, but we should be careful when saying this has a direct causal link with share price.
October 27th, 2007 at 2:11 pm
I would like to respond to Simon’s comments that NPS is “backed up by a Harvard Business school book, Bain & Co and reputed academics.” Yes, NPS is backed by Bain & Co., but the support of a consulting firm that sells the metric hardly qualifies as a serious scientific investigation.
And yes, the Ultimate Question book by Reichheld is published by Harvard Business School Press. However, I am unaware of any Harvard Business School professor who would go on record claiming that NPS links to growth, that it is superior to other metrics such as customer satisfaction, or that it is “the one number you need to grow” as is claimed (see NetPromoter.com). In fact, I am unaware of any noted academic who would make such a claim.
As a result, the argument that NPS is backed by “reputed academics” is ridiculous. I know of no serious scientific inquiry by any academic that would support this statement. The only study of which I am aware which Net Promoter users cite as scientific evidence was not even published in a scientific journal (Paul Marsden, Alain Samson, and Neville Upton, 2005, in Brand Strategy magazine). This paper does not link NPS to future growth, nor does it meet the standards of a scientific journal.
The only scientific research of which I am aware that has rigorously investigated Net Promoter universally finds that the claims attributed to Net Promoter are false. In fact, the claim of superiority can be disproven using Reichheld’s own data for his best case scenarios (versus a metric he claims was examined and found to have a 0.00 correlation to growth).
I would like to be clear that the overriding issue of my co-authors and me is the integrity of the research presented in our scientific journals. The Harvard Business Review is one of only 40 Financial Times journals, so the work published there has a great influence on management science. If someone publishes “research” that claims that other metrics don’t work, but the one I sell links to corporate growth, then that research had better stand up to scientific inquiry. The fact that it can be disproven using Reichheld’s own data would be laughable if it did not reflect incredibly bad research or incredibly bad ethics.
Should you wish to see a videotape of a presentation I made in Toronto on the scientific research I conducted with Professors Bruce Cooil (Vanderbilt University), Tor Wallin Andreassen (Norwegian School of Management, and Lerzan Aksoy (Koc University), please go to:
http://www.ipsosloyalty.com/knowledgecenter/event-070501.cfm
Finally, no one is suggesting not measuring customer attitudes, or their recommend intention…the argument is silly. It is not an either this or nothing proposition. What did managers do before December 2003 (when Reichheld introduced Net Promoter in the Harvard Business Review)…the answer is quite a lot.
Sincerely,
Tim Keiningham
November 1st, 2007 at 5:10 pm
Hehe, looks like we have an interesting debate here. Note that I’m not a subject matter specialist, and for sure not a seasoned research professional or academic, I’ve just been digging in the trenches of marketing reality for about 18 years. And while digging I have seen a lot happen.
My question is, who should I trust on this?
I read from Tim that Reichheld’s (+Bain + Satmetrix) research is not backed up by academics. Whether academics were involved or not is not the point. The question is, who can we trust? A reputed loyalty professional who want’s to sell consulting and a book, or a reputed research professional who has his own research to sell and who unfortunately did not write a cool management book but still wants to defend his business …
Don’t get me wrong, I have a lot of respect for both Mr. Reichheld and Mr. Keiningham, and I would trust both as they have an outstanding reputation. But I simply do not have the time to go check all the research myself so I can decide myself.
The solution? Can somebody set up a debate between the two opponents so they can discuss at length. Make it a (video)Podcast so there is no time limit at all. Alternatively Simon could run both gentlemen through the Attention software and let the crowds decide who’s the winner, but then I’m afraid we will run into the next debate :)
As for Tim’s last remark, I agree companies have been working on loyalty before Net Promoter score was introduced. But my experience tells me that there’s far too few companies who took loyalty seriously, and almost none who actually integrated it into their day-to-day dashboards. And today, when consumers have taken back their power to think, choose and decide from self centered companies, today is when the power of taking into account customer sentiment will make or break a company. Back in the previous century when the masses had to watch TV commercials by lack of an alternative or a skip-button, and when we had no means to express our sentiment about a brand on a worldwide scale, back in those days the impact of negative consumer sentiment was much less.
So cheer up loyalty professionals around the world, your time has come. Think up a catchy metric, write book about it and hope to become famous. Or better, go out and check out the internet (I mean like blogs, podcasts, myspace, facebook, you know, where real action is) and discover how your customers really think. Then use that knowledge to build better stuff that will sell itself without wasting your precious money on wasting my precious time.
November 2nd, 2007 at 12:32 am
Bert –
I want to make clear that my work for IPSOS is NEVER allowed to interfere with my scientific research. If I ever felt any pressure in this regard, I would resign immediately. Furthermore, my co-authors, Professors Bruce Cooil (Vanderbilt University), Tor Wallin Andreassen (Norwegian School of Management), and Lerzan Aksoy (Koc University), are some of the most highly regarded scientists in academia. Therefore, if Net Promoter had performed as claimed, then that is EXACTLY what we would have reported.
Our overriding issue is the honest reporting of research in our most prestigious journals. The Harvard Business Review is one of only forty (40) Financial Times scientific journals (which means the research reported there will have tremendous impact on the work of management scientists). Our obligation as scientific researchers DEMANDS that we call attention to research reported in these journals that defies logic, and points to potential bias. And although we would prefer otherwise, since it was our research that uncovered this issue, the responsibility of calling attention in this case falls on my co-authors and me.
The most troubling finding from our research is that we found very strong evidence of research bias in the research reported by Reichheld in support of Net Promoter. In particular, we were able to replicate a subset of Reichheld’s reported data for his best case scenarios and compare it to a metric he claimed was examined and found to have a 0.00 correlation to growth, the ACSI. Our findings clearly show that when using Reichheld’s own data, Net Promoter wasn’t superior to the ACSI. And it is important to note that Satmetrix has made clear that they have no data supporting Reichheld’s claim of superiority to the ACSI. Therefore, it is virtually impossible to imagine a scenario other than research bias on the part of Reichheld as the cause of this finding.
Sincerely,
Tim Keiningham
November 2nd, 2007 at 12:27 pm
Here’s what the research community misses:
It’s not about research, or whose research is better. It’s not about a methodology debate. Frankly, it doesn’t even matter if the number tracks anything at all.
This is all that matters: “Net Promoter is simple and gets companies to focus on customers.”
You should fight to earn more customer recommendation. You should fight to reduce unhappy customers. That is the secret to a great business.
Net Promoter does something that the research community never could. It gives companies something to rally around. It gives companies a vision and an action plan that works all the way from the executive suite to the loading dock.
That is priceless.
November 2nd, 2007 at 1:42 pm
Sounds pretty convincing to me … I look forward to what Mr. Reichheld and his team’s view is.
November 2nd, 2007 at 2:42 pm
Hi Bert, Andy is a co-founder of WOMMA and a staunch supporter of the NPS methodology, so is well placed to explain it’s virtues.
The clear point from everyone’s commentary is that it is NOT about whether companies should measure customer satisfaction or be worried about negative feedback OF COURSE this is important.
Main issue is the proclamation that better NPS correlates with growth (I assume it is discussed on analyst calls because this in turn connects to market value/share price)
To collect together arguments for and against.
TIM - Tim’s joint research shows no signs that NPS is connected to growth, he doesn’t say that understanding customer sentiment is irrelevant but there is just no correlation.
FRED REICHHELD - Fred Reichheld sees the connection between NPS and growth
“The correlations between NPS and growth stand up only when a set of competitors is defined correctly. Home Depot must be compared to Lowe’s, not to Victoria’s Secret. If Lowe’s had a higher NPS than Home Depot, that would be a meaningful piece of information for Home Depot’s decision makers, whereas a higher NPS at Victoria’s Secret would be meaningless.”
http://netpromoter.typepad.com/fred_reichheld/2006/09/nps_does_not_wo.html#more
GENERAL ELECTRIC: GE see a connection between NPS and growth. Annual Report 2005
“GE use this as an important metric see 2005 Annual Report “Lastly, we are using a simple metric called Net Promoter Score (NPS) to measure how customers view GE. NPS creates a view of customer loyalty. The absolute score is less important than the trend. We learn from both promoters and detractors. Most importantly, we have been able to associate NPS improvement with growth. NPS is simple and we can use it across the Company. Our ultimate goal is to use improvements in NPS as a measure in how leaders get compensated.”
Perhaps if there is a data set that shows a link, then it makes sense that this should be independently verified?
November 3rd, 2007 at 2:45 am
NPS does not ignore any of the factors for market success or value creation. It incorporates them all. All the important issues that a consumer experiences explicitly or implicitly (and that influence his/her decision) such as context of recommendation, pricing, market timing, supply, competition, availability of substitutes, advertising ….they are all taken into account when ultimately he or she – often in a blink - takes a “buy or not buy” decision and when a bit later he/she again will decide to either recommend or not recommend. Buying and recommendation afterwards are both the result of the same good or lousy job the brand has been doing until the 1st moment of truth.
And since recommendations to friends, family etc are becoming more and more decisive factors into any “buy or not buy decision”, chances are that the “likeliness to recommend now” , can be a good prelude/prediction to growth in the next quarter or year. Reichheld is probably right.
So NPS is not the measurement of something in isolation. It’s the measurement of how consumers perceive a long and complicated process that lead them to the first moment of truth (buying). Then to the second (using). And finally to the third moment ot truth(recommending – or not) : the final stage.
Should companies rely only on that number alone? Of course not. Like the Net Profit of a company is the result of many factors, the Netpromoterscore too is the result of many factors. When your score is awful of course you’ll have to do additional research at Ipsos or Attentio to know why you’re performing so badly. Like when you have lousy profits (or no profits at all) you’ll also have to do your homework at both the sales and cost side. Net Profit and Net Promoter Score give an indication. They are a prelude. And may have prediction force. NPS is also not related to market capitalization or share price … but to growth of market share in the following period/year. And growth of market share doesn’t mean even profit.
To conclude … we should also NOT confuse loyalty with recommendation. There is a difference between the financial bottomline and the recommendation bottomline. You can be a loyal and profitable client without ever having recommended the brand use. From a future growth of market share you’re useless. If you tell nobody, nobody will know. And when you die, a certain % of market share will be lost. For future growth actively recommending clients are more important than loyal clients who shut up.
PS. We recently did some research in China to measure the recommendation power among internet users. If you’re interested you can read it here.
http://blog.boondoggle.eu/2007/10/the-recommendat.html
http://blog.boondoggle.eu/2007/10/new-i-mergeboon.html
November 5th, 2007 at 3:35 pm
I wanted to follow-up on a comment Bert made: “The solution? Can somebody set up a debate between the two opponents so they can discuss at length. Make it a (video)Podcast so there is no time limit at all.” I am a co-chair on the Research & Metrics Council for the Word of Mouth Marketing Association and we have set up a panel for precisely this kind of discussion at the Research Symposium on 13 November. Rather than a debate between two opponents we broadened the panel to include some other voices. You can view the agenda here: http://www.womma.com/research3/agenda/ Confirmed speakers are Deborah Eastman from Satmetrix; Tim Keiningham from Ipsos; and Matt McGlinn at BzzAgent. We’ll also be having a speaker from a company who has used the net promoter approach in their company. Given the interest in this topic I’ve asked WOMMA to make the audio recording available on their website after the event, which they agreed to do.
November 5th, 2007 at 7:27 pm
Hi,
I would love to go and listen to that panel… Sounds like the perfect collection of people. I think its excellent that it will be made available afterwards.
On a connected note - what is the WOMMA stance of NPS and its correlation to growth?
November 9th, 2007 at 11:11 pm
Simon,
Your readers have done a good job of laying out the debate and we look forward to the discussion next week at WOMMA. Andy says it best “It’s not about whose research is better”, it’s about real companies seeing real results. Many of those can be seen at http://www.netpromoter.com.
As to the claims of research bias or that the subsequent research has proven Net Promoter flawed, this is simply not true. The Journal of Marketing paper clearly states that they have not been able to prove such bias, and it also shows that Net Promoter was at least equivalent to ACSI in correlation to growth. The lastest research in Managing Service Quality found that the recommend question was one of the top correlates to behavior (1,2 or 3) in retention and recommend behavior. Does a correlation of .01 or .02 really matter enough to complicate the measurement system?
You may be interested in Fred’s latest posting on netpromoter.com. http://netpromoter.typepad.com/fred_reichheld/2007/11/are-you-doing-n.html
It’s not about the score, it’s about driving an organizational discipline to create more promoters and reduce detractors.
One has to question, what is all the hoopla about?
November 12th, 2007 at 7:41 pm
Deborah —
With regard to the statement regarding research bias in the Journal of Marketing (as I told you earlier) the standards of a scientific journal are at the absolute highest … and the Journal of Marketing is the top scientific journal in all of management and economics (as measured by the citation index). Therefore, 99% certainty is not high enough to conclusively prove research bias on the part of Reichheld. Nonetheless, there is no plausible explanation offered by Reichheld as to why Net Promoter was not superior to the ACSI despite his claims in HBR, The Ultimate Question, and his presentations that their was no linkage between the ACSI and growth (actually stating there was a 0.00 correlation). I am unable to find anyone who can offer a plausible explanation for this.
Also, you are mistaken that our findings found the ACSI and Net Promoter equivalent. We stacked the deck in favor of Net Promoter, using the best case examples presented by Reichheld. What our findings suggest is that even using the best case examples for Net Promoter, NPS is not superior to a metric that was claimed to be examined and found to have a 0.00 correlation to growth.
Again, my issue is not with the use of Net Promoter, it is with the research reported by Reichheld. And I also do not like the findings of our research misrepresented.
– Tim Keiningham
December 22nd, 2007 at 9:16 am
I have been measuring the Net Promoter Score for my product line for 5 years. Over that that period, I a noticed a strong predictive value in the NPS on my market share. At the same time I have also seen a lot of false conclusions drawn from the NPS. Here are my 6 take always with regard to NPS:
1/
The NPS is like an AWACS. It gives an early warning when something is wrong/right.
2/
I noticed a strong correlation between NPS and market share “in the trending” for my business.
3/
I always interpret the NPS relative to that of my competitors. The value is in the delta.
4/
It’s not a good idea take up the NPS in the company KPI sheet and let peoples bonus depend on it. Doing so will invoke bad business decisions.
5/
It’s not a good idea to use the Net Promoter Score to manage an operational division that relies on predictability (e.g.. customer care, sales)
6/
Your NPS can look a whole lot different overnight if you are confronted with an unforeseen event. (bad press, crisis)
The NPS works well for people who base judgement partially on intuition. It doesn’t work for people who only rely on factual evidence for their decision making. And that is in essence the problem with this debate: every manager has to understand what works best for him or her. And the same rule applies for financial analysts. Some analysts might conclude that NPS works for them to predict future value of a company in an industry while others might consider it to be just hot air.
Why did 6 sigma do great for some companies while it didn’t for others?
Why was off shoring a success for company A and a disaster for company B?
Why do some mergers or acquisitions work while others fail?
Business is not rocket science because if it were, consultants wouldn’t be making all that money. Nor would hedge funds.
I use the NPS to make judgement calls about my business because it works fine for me. I respect people who say that NPS is not working for them but I get the CREEPS of people who try to convert NPS into a binary yes/no metric.
Here is a trick to proof that NPS is not a good metric for predicting value growth:
If you increase the price for your product or service, your NPS will drop but the value will go up. What happens the next year will be a mystery.
Now as far as The Ultimate Question is concerned, is there a correlation between a good NPS and business growth? Sure there is, but you need to take into account a lot of biasing things such as industry growth, competition, brand awareness, first movers advantage, disruptive technologies, cash position, public relations, pricing, channel power… So far I haven’t seen a scientific study that was cleaned from these biases and I don’t think I will.
Dann Rogge
February 6th, 2008 at 10:25 pm
[…] but has not impacted our focus on valuable customers. For a good post on the debate look herehttp://www.attentio.com/blog/2007/10/20/net-promoter-debate/ ,for a response from the co-authors, […]