Data vs. Anecdotes: Which to Trust?
Two quotes. Which one do you resonate with more? One of them was a manifestation of organizational dysfunction, while the other offers a great insight.
Statement 1: I trust your data, and my intuition.
Statement 2: when data and anecdotes disagree, the anecdotes are usually right.
I strongly prefer Statement 2. Statement 1 sounds reasonable, but in the specific context where I heard it, it was a letdown.
I first heard Statement 1 from PM leadership at Meta, on an internal product that was deeply analytical and statistical in nature. In that context, it landed poorly. The speaker did not have prior enterprise software experience, nor a background in analytics or statistics, and came from a consumer experience team. When a statement like this is directed at a group of domain experts building analytical systems, it subtly inverts the knowledge hierarchy. Remember: information is leverage and knowledge is power, we should entrust decisions to the person who actually possesses the relevant domain knowledge. Otherwise, “intuition” becomes a proxy for authority rather than insight. This was part of the reason I eventually left Meta.
Statement 2 comes from Bezos. In a talk about early Amazon, he described a situation where metrics showed customer wait times for support calls were under 60 seconds, yet customers consistently complained about long waits. Both appeared true. During a weekly business review, when a customer service lead defended the metric, Bezos picked up the phone and called customer support himself. The wait time was ten minutes.
Should we throw modern data-driven practices out the window and make decisions based on anecdotes or gut feel? No. The lesson is that when anecdotes disagree with your data, it is worth investigating why. Very often, the issue lies in the measurement itself: data collection gaps, the wrong slice of data, misleading aggregates, or metrics that fail to capture the actual user experience. When that happens, the data is not telling the story you think it is.