This is part of a series about metrics that wasn’t planned as such. I’ve got a lot of grievances to air, and today I’m focused on metrics that mask bad experiences and others that perpetuate false or misleading narratives.
Take engagement metrics for AI chatbots. I imagine that there are product teams out there tracking the number of conversational turns (switches between human and AI agents in the convo) and time spent in chat as measures of engagement. More conversational turns and more time spent in chat is a good thing if engagement is the thing that matters most.
But without an understanding of context, multiple turns might mean someone struggled to get a straight answer. Similarly, total time in chat might mean an unsuccessful search for information that feels different because of the medium [feels like I’m chatting with another person] but doesn’t actually produce better outcomes for the person chatting. Engagement doesn’t equal value (see, for example, Facebook).
Or consider a metric I see on LinkedIn: daily job posting counts. So many new opportunities everyday! It’s a seeker’s market! Go get one! Rise and grind! The only one holding you back is you!! Ok but seriously..
The numbers are eye-catching, and they give the impression of excellent conditions for job seekers, which is an interesting counterweight to the much more voluminous posts about lay-offs or frozen or reduced head-counts as firms evaluate (measure?) AI-related productivity gains. Hey, there were 30 new roles in [field x] posted today! There were also 5k laid off from [company a], 2k laid off from [company b], … you get the idea. Context matters.
The daily job opening posts compete for attention, drive visibility for the posters, and generate newsletter subscriptions [some posters publish newsletters on the topic]. It’s also possible that they point people towards legit opportunities, but I’m not so sure. A little digging reveals that the eye-catching number is of course misleading.
They don’t distinguish between real openings and ghost jobs. They don’t separate full-time roles from short-term contracts. They don’t offer sponsorships for people who are qualified but not authorized to work in the country of record. And so on. It’s compelling to see 45 jobs posted in the last 24 hours! in my field, but of those 45, 10 are junior, 20 are mid-career, 5 are principal, 5 are re-posts, and 5 are on-site or hybrid with no relocation support.
The daily job count metric is what I’d call a “feel good metric” in that, at a glance, it can cause a spike in optimism. It sells hope. It sells opportunity. In builds myths. It builds personal brands for the people making the posts. And all those things together make it a bit nefarious. The posters have no skin in the game other than as an aggregator. They’re not incentivized to vet the quality of the job openings or the relevance of the posts for their audience. Some members of their audience even repost the job opening posts, driving attention and engagement to themselves.
The same pattern plays out across domains. Metrics that look good in dashboards but obscure whether experiences are valuable. Statistics that serve an agenda — driving traffic, supporting a preferred narrative, making a product look successful — while adding little to the lives they purport to improve.
We can do better, so let’s do better.
jordan beck, phd 