Bayesian Reasoning and UX Feedback

Yeah, this is going to be a geeky one, sorry.

So let’s say you have a product you sell with a user experience you designed. It sells 10,000 units. After the product has been out for a year or so, you engage with your user community on whatever platform you use, and you solicit some feedback about some element of the UX. Just call this element the “Z Function” for shorthand. You don’t get a lot of engagement back, but 8 people tell you that they don’t like the Z Function and want you to change it in some way, while 2 people respond and say they like the Z Function just fine and you shouldn’t change it.

Should you assume that 80% of your users don’t like the Z Function, and thus you should change it?

Absolutely not.

But I see companies do this a lot. They make the changes to the Z Function, and suddenly their sales aren’t as good, their old users abandon them, etc. What went wrong? Weren’t they just responding to their customer base?

Let’s examine the various problems. You can’t assume that the responses you get are representative. For one, people are more likely to complain than to compliment – if everything is fine, they just use the product. Most people don’t actively engage back with every company whose products they use. You shouldn’t say “wow, out of everyone that responded, 80% had complaints.” You should say “wow, out of everyone who used our product, only 0.08% had complaints.” Now, maybe the old adage is true that for every one person that complains, there’s 99 who have the complaint but don’t voice it. But that’s still only 800 people, or 8%. Worth addressing, perhaps – but still very much the minority. And if you change the Z Function to make those 8% happy, you run the risk of making the 92% unhappy.

The best indicator of the strength of your product isn’t how many people do or don’t complain. It’s how many people buy it. Actions speak louder than words, and most people don’t bother to buy a product and tell you that they love it. Buying the product is the compliment. A receipt is a love letter. And that huge, silent majority of people buying your product but not talking much about it – they’re the ones you have to keep happy, not the tiny minority that complain. If you can make that tiny minority happy, great – but if you do it by changing something fundamental about your product, you’d better be sure that your silent majority is still going to buy.

I often see company announcements like “You spoke, and we listened. We received dozens of complaints about the Z Function of our product, so we’ve completely changed it.” Dozens of complaints… out of hundreds of thousands of units sold. And the change is for the worse, and chases 20% of their user base to a competitor. And they’re scratching their head wondering what went wrong.

This isn’t advice saying you should ignore complaints. It’s advice saying you should know your statistics, recognize you can’t please everyone, and don’t give all your grease to the squeaky wheel if it’s making the other 3 fall off.