← All articles Build the Right Thing

5 User Research Mistakes That Produce Misleading Results

By the FabricLoop Team  ·  May 2026  ·  4 min read

There is a version of user research that is worse than doing no research at all: research that produces confident but wrong conclusions. When a team does no research, they know they're operating on assumptions. When they do bad research, they believe they have evidence — and they stop looking.

The five mistakes below are not exotic edge cases. They show up in most product teams, most of the time, in ways that are surprisingly hard to spot from the inside.

"The goal of user research is not to confirm your instincts. It's to replace your instincts with evidence — including evidence that you were wrong."

The five mistakes

Mistake
What it looks like in practice
Mistake 1 Asking about hypothetical future behavior
The team asks "Would you use a feature that did X?" Users say yes. The team builds it. Nobody uses it. People are optimistic about their future selves and poor judges of what they'll actually do. Always ask about past behavior: "Tell me about the last time you had to deal with X."
Mistake 2 Interviewing users who already love the product
Research panels filled with power users and advocates produce warm, positive, improvement-focused feedback. What they don't produce is insight into why people left, why people never converted, or what the product needs to reach a new segment. Talk to churned users and non-users too — they are uncomfortable but essential.
Mistake 3 Leading questions that confirm existing beliefs
"How helpful did you find the new dashboard?" is a leading question. "Tell me about the last time you used the dashboard" is not. Leading questions produce polite agreement, not honest assessment. The researcher's attachment to the hypothesis leaks into the question phrasing, and users — being socially aware — follow the signal.
Mistake 4 Stopping at "what" without asking "why"
A user says: "I'd want a way to filter the list." The researcher notes "users want filtering." But filtering is a solution, not a need. What is the user actually trying to do that filtering would help with? The answer to "why" is where the design insight lives — and it often reveals a better solution than the one the user proposed.
Mistake 5 Treating qualitative research as quantitative
"Three out of five users said they found it confusing" is not a statistic. It's a directional signal from five people. Qualitative research tells you what and why; it cannot tell you how many. Presenting it with numerical precision gives stakeholders false confidence and often leads to optimizing for the wrong thing at scale.

Why these mistakes are so persistent

Most research mistakes are motivated. The team already has a direction they want to go — a feature they want to build, a hypothesis they believe in, a stakeholder they want to satisfy. Research becomes a ritual to legitimise a decision that has already been made, not a genuine attempt to test it.

The solution is not to make researchers more rigorous in isolation. It's to separate the role of "deciding what to build" from "running the research that informs that decision." When the researcher is also the person who proposed the feature, confirmation bias is almost inevitable.

A useful norm Before any research session, write down what result would cause you to change your mind. If you can't name it, you're not doing research — you're doing theatre. The ability to state a falsifying result is the difference between genuine inquiry and confirmation bias with extra steps.
On sample size For usability testing, five to eight participants reveal the majority of issues — diminishing returns set in fast. For problem discovery interviews, twelve to fifteen is usually enough to reach saturation. For anything you want to generalise quantitatively to a large population, you need a proper survey with a representative sample — not ten interviews.
How FabricLoop supports honest research Keeping raw interview notes, synthesis, and decisions in the same thread makes it harder to quietly discard inconvenient findings. When the evidence is visible alongside the decision, the team can see whether the conclusion followed from what was learned — or preceded it.

10 things to take away from this article

  1. Bad research is worse than no research — it produces false confidence and stops teams from looking further.
  2. Never ask about hypothetical future behavior. Ask about specific past behavior instead.
  3. Research panels of advocates produce warm, improvement-focused feedback — not insight into why people leave or never convert.
  4. Talk to churned users and non-users. The feedback is uncomfortable and essential.
  5. Leading questions produce polite agreement, not honest assessment. Watch your phrasing carefully.
  6. When a user proposes a solution ("I want filtering"), always ask why. The answer usually reveals a better solution.
  7. Qualitative research tells you what and why. It cannot tell you how many. Don't present it with numerical precision.
  8. Most research mistakes are motivated — the decision has already been made, and research is being used to legitimise it.
  9. Separate the person who proposes features from the person who runs the research that evaluates them.
  10. Before any research session, write down what result would cause you to change your mind. If you can't, you're not doing research.