jordan beck, phd

UX researchers—like other applied researchers—face many constraints in their work: tight timelines, pressure to move faster, and limited resources. These constraints can lead to methodological shortcuts that compromise the quality of the insights delivered. That’s not good, because the quality of the insights affects the quality of the decisions they inform.

Run a prioritization study without the proper rigor and you risk surfacing priorities that aren’t real. Ask leading questions in an interview or recruit a sample that isn’t representative of your target users and you’ll generate insights that point strategists and product managers in the wrong direction.

The problem isn’t necessarily that researchers make these mistakes out of negligence, though that is sometimes the case. It’s that constraints risk framing methodological rigor as a nice-to-have that the team can’t afford. It’s more important to deliver quickly than it is to deliver quality. But that framing is misguided: rigor isn’t a luxury. It’s a requirement for good decision-making.

Decision-makers trust researchers’ expertise, and so when researchers compromise on rigor, they risk delivering insights that decision-makers nevertheless treat as valid. Such insights undermine lots of other work: product decisions, resource allocation, strategizing, roadmapping, and requirements writing. The list goes on.

This doesn’t mean every study needs to be bulletproof or that we can’t adapt research methods to real-world constraints. Of course we can. Perfect conditions don’t exist, and there’s always room for improvement.

This does mean we need to be clear about what our methods can and can’t tell us. We need to know when a methodology is fit for purpose and when it’s not. We need to be open to saying ’no’ to research requests to avoid delivering bad data packaged as good insights.

Decision quality rests on insight quality, which, in turn depends on methodological rigor. Methodological rigor, therefore, determines whether we’re helping or hurting the decisions being made, and it’s something we need to talk about (and prioritize) more.