Data Driven Is Not the Same as Number Driven. And the Difference Is Costing You.

Data driven. It is on every strategy deck, every job description, every quarterly review. Say it with enough confidence and you will rarely be challenged. It has become the organisational equivalent of eating your vegetables: everyone claims to be doing it, and very few people are doing it in full.

Here is the problem. Somewhere along the way, data driven stopped meaning what it was supposed to mean. It got narrowed. Compressed. Reduced to a specific kind of data: numbers, dashboards, conversion rates, cohort analysis, NPS scores. Quantitative data. The kind that fits neatly into a chart and looks authoritative in a board presentation.

That is not data driven. That is number driven. And the distinction matters more than most organisations realise. Researchers at Chicago Booth have given it a name.

The Gap Between Data Driven and Number Driven Has a Name

Data driven means making decisions based on evidence. All of it. Not just the evidence that can be counted, but the evidence that can be heard: in customer conversations, in win/loss debriefs, in the thirty seconds at a conference booth when you pitch something and watch someone’s face go blank.

Qualitative research is data. Rich, direct, human data. The kind that tells you not what is happening but why. The kind that no dashboard has ever produced and no survey has ever captured cleanly. It has always been part of what data driven meant. A genuinely evidence-led approach needs both qualitative and quantitative inputs.

It just got crowded out by the tools that were easier to automate, easier to schedule, and easier to present to a room full of people who equate a chart with rigour.

The result is a generation of organisations that call themselves data driven while systematically ignoring the most honest data source available to them: the actual words of the people they are trying to serve. Quantification fixation is not a character flaw. It is a structural habit built by years of optimising for what is measurable rather than what is meaningful. And it has a real cost.

“Data driven does not mean number driven. It means evidence driven. And evidence comes in more than one form.”

Erika Kirgios and her colleagues at Chicago Booth call it quantification fixation: the tendency to overweight numerical evidence compared to qualitative evidence when making decisions. Their research shows that when only some information is quantified, it distorts choices significantly away from what decisions would look like if all evidence were weighted properly. In other words, the moment you put a number on some things and not others, the numbers win, regardless of whether they deserve to.

This is not a fringe observation. It is peer-reviewed, it is documented across industries, and it describes the operating reality of most modern organisations with uncomfortable precision. The data-driven culture most businesses are proud of is, in practice, a quantification-fixated culture. The numbers are not wrong. They are just incomplete. And decisions made on incomplete evidence are only as good as the evidence they are missing.

Under pressure, this distinction becomes expensive. Organisations default to action: pivot quickly, fail fast, move with urgency. These are not wrong instincts. But they carry a hidden assumption that almost nobody questions: that the next decision will be better than the one that just failed. Without understanding why the last one failed, that assumption is optimism dressed up as strategy.

You can keep failing fast, or you will go bust. You can keep pivoting, or you will be dizzy and disoriented. At some point, the velocity of iteration is not the problem. The quality of the insight driving each iteration is.

The good news, and the point of this article, is that the tools to close that gap now exist. Qualitative research is not lagging behind its quantitative counterpart. It is not a slower, softer, less rigorous alternative. With the right framework and the right AI-assisted workflow, it can be conducted, coded and synthesised at scale, continuously, by any team willing to build the habit. What was previously a bottleneck is now a workflow.

It is also worth noting something about how win/loss data is typically captured, because it illustrates the problem precisely. CRM disposition codes align with what buyers actually report only around 15% of the time. Sellers record a single consequence: “lost to competitor”, “price”, “no decision”. Buyers, when asked directly, report four to six decision drivers per deal, rich with context, nuance and the specific moment where confidence in a vendor collapsed.

Most organisations are making competitive and positioning decisions based on the 15% version. That is the cost of under-investing in qualitative primary research.

Four Different Kinds of Data — and What Each One Can Actually Do

The pattern is consistent. Every data type has genuine value and every data type has a ceiling. The one most consistently underinvested in, primary qualitative research, is the only one that directly answers the question every organisation is asking when the numbers go wrong.

Qualitative Research Is Not Behind. It Has Been Underdeployed.

Qualitative research has not been absent because it lacks rigour or value. Every experienced PMM, GTM leader or founder knows that a single honest customer conversation can reframe a quarter’s worth of strategy.

The problem has been practical: qualitative research at scale was genuinely hard. Planning, recruiting, conducting, transcribing, coding, synthesising, distributing: each step required time, skill and resource that most teams could not sustain continuously. So it got treated as a periodic exercise rather than a permanent capability.

That constraint no longer exists in the same form. AI has changed the economics of qualitative research fundamentally. The steps that consumed the most time — transcription, coding, synthesis, formatting for different audiences — can now be automated or accelerated to the point where a single practitioner can run a continuous interview programme, process the outputs in near real time, and distribute structured insight across the organisation within hours of the final conversation.

That is not a marginal improvement. It is a structural shift.

So the right framing is not that qualitative research needs to catch up. It is that the last remaining practical barrier to running it continuously has been dramatically reduced. What was a bottleneck is now a workflow. The question is whether your organisation is building it.

The first time you stand at a conference booth, sit across from a customer in an interview, or present to a room and watch faces go blank, you learn more in thirty seconds than in any briefing document you have ever read. The person in front of you either leans in or their eyes gloss over. That reaction is data. Immediate, unfiltered and impossible to get from a chart.

“The first time you stand at a booth, sit across from a customer, or present to a room and watch faces go blank: you learn more in thirty seconds than in any briefing document you have ever read. That is not anecdote. That is primary data.”

What qualitative primary research gives you that nothing else can replicate is the compression of the feedback loop. You test a framing, you see the reaction, you adjust. No waiting for survey responses to reach statistical significance. No waiting for the next cohort to mature. Immediate signal, immediate iteration.

Draft unapologetically and edit without mercy. Your first version of any message, pitch or product hypothesis is not the point. The version shaped by ten real conversations is. For a content and product marketing specialist, a real human reacting honestly in real time is the most efficient editing tool that exists.

The First Challenge: Getting People to Talk Honestly

There is an objection worth addressing before we get to the method. Getting willing, honest participants is hard. People are busy. They are guarded. And even when they agree to talk, there is a reasonable question about whether they will tell you the truth.

It is a fair objection and it deserves a straight answer.

The dynamic shifts entirely when the problem being discussed is acute. When someone is genuinely struggling with an unsolved problem, when the status quo is causing them real pain, they are not doing you a favour by talking to you. You are the one doing them a favour by listening. They want to articulate the problem because articulating it is part of processing it. They want someone to understand the nuance because the nuance is precisely what generic solutions keep missing.

And they are far more likely to be honest because they are not protecting a decision they are happy with; they are describing a situation they want out of.

This means the customers most worth talking to are not your happiest ones. They are the ones sitting on an acute, unresolved problem in your category: a churned customer, a deal you lost, a prospect still using a workaround, a customer who bought but has not yet seen the value they expected. These are the conversations that produce the most honest, the most specific and the most actionable insight.

“The best research participants are not the ones most willing to give you their time. They are the ones who need someone to finally understand their problem.”

Practically, this means framing your outreach around their problem rather than your research. Not “would you be willing to take part in a customer interview?” but “we are working to understand the specific challenges around [problem area] and your perspective would genuinely shape how we approach this.”

One is a favour request. The other is a problem-solving conversation they have a reason to join.

Jobs to Be Done: The Framework That Makes Conversations Actionable

Jobs to Be Done is one of the most cited and least correctly applied frameworks in modern product and marketing practice. The framework was originated by Tony Ulwick in 1990 and popularised by Clayton Christensen through his 2016 book Competing Against Luck. That gap, between knowing the name and knowing how to use it, is precisely what this section is about.

The core idea is worth stating without academic decoration: customers do not buy products, they hire them to do a job. The job is the unit of analysis. What progress was the customer trying to make? What was stopping them from making it with what they had before? What made them decide your product could do it better? And critically: what would make them fire you?

For Product Marketing Managers, JTBD reframes the entire research agenda. Persona documents tell you who your customers are. JTBD tells you what they hired you to do. NPS tells you how satisfied they are. JTBD tells you whether you are still doing the job they originally hired you for. Pricing data tells you what they paid. JTBD tells you what they were willing to change — processes, workflows, internal relationships — to make it work.

That last signal is the most honest measure of product-market fit available, because it is grounded in behaviour rather than opinion.

It is also worth addressing a common misconception: JTBD does not replace demographic or firmographic understanding. At scale, the two are complementary. When you can conduct, code and synthesise qualitative interviews continuously, patterns emerge that cut across traditional segments: certain jobs appear consistently in specific company sizes, verticals or growth stages in ways that enrich your targeting significantly.

You stop segmenting purely by who the customer is and start layering in what job they are hiring you to do. That combination is more powerful than either dimension alone.

“The language your customer uses to describe the job they hired you to do is almost always better than the language in your current messaging. Go and get it.”

The semi-structured interview: scaffolding, not a script

The semi-structured interview is the practical vehicle for JTBD research. Ten well-chosen questions give you enough structure to make outputs comparable and codeable across sessions, while leaving room for the unexpected answers that contain the most useful signal.

In my experience, the best insights almost always come from following a thread you did not anticipate, not from staying on script.

Your questions should move through three phases: context — what was happening before they started looking; trigger — what changed and made the status quo unacceptable; and decision — what they evaluated, who was involved, and what ultimately made them choose or not choose you.

Run this framework across five to ten interviews and patterns emerge faster than you expect. By the third conversation you will be editing your questions in real time. By the sixth you will have language you did not write but should have. By the tenth you will have a positioning argument built on evidence rather than assumption.

Win/Loss Analysis: Where JTBD Adds the Nuance That Changes Everything

Most win/loss programmes are underperforming because they stop at the surface. A deal was lost because of a missing feature. A deal was won because of integrations. The seller records the answer, adds it to the tracker, and moves on.

The problem is that these answers explain the outcome without explaining the decision.

As noted earlier, CRM loss reasons align with buyer reality only around 15% of the time. Buyers report four to six decision drivers per deal. Sellers record one. JTBD win/loss closes that gap by going to the source: the buyer’s actual experience of the process, and asking the questions that surface what a dropdown never could.

Three categories worth analysing separately

Most teams analyse wins and losses. The third category, no-decision outcomes, is where the most revealing conversations often happen and the most overlooked. A deal that stalled or died without selecting any vendor tells you that nobody in your category made a compelling enough case for change. That is not a competitive problem. That is a value communication problem, and it sits much earlier in the conversation than most teams realise.

As a practical working mix, I would aim for roughly 40% won deals, 45% lost deals and 15% no-decision outcomes. Won deals reveal what genuinely differentiates you. Lost deals expose competitive vulnerabilities and execution gaps. No-decision deals surface the situations where your category itself failed to justify action, which is often the most important strategic signal of all.

The feature gap that is not really a feature gap

When a customer says they chose a competitor because of a specific feature, the instinct is to add it to the roadmap. Sometimes that is the right call. But a JTBD conversation will often reveal something more uncomfortable: the feature existed, or something close to it did, but the customer never believed it could do the job they needed done. The gap was not in the product. It was in how the product was understood.

That is a messaging problem, a demonstration problem, or a trust problem. Adding features will not solve any of them. JTBD win/loss gets you to that distinction. Standard win/loss does not.

The cost objection that is not really about cost

Similarly, “we went with a competitor because of price” is almost never the complete story. Price becomes the deciding factor when the perceived value of the job being done does not justify the cost in the buyer’s mind. That is a value articulation problem, not a pricing problem.

The question to ask is not “was our price too high?” but “what would the job have needed to be worth for the price not to have been a barrier?” That answer tells you exactly where your value case broke down.

Timing and pushing past the first answer

Interview within seven to fourteen days of the decision. Close enough that the conversation is vivid and specific. Far enough that the buyer is past any awkwardness about the outcome and willing to be candid.

In the interview itself, the discipline is consistent: push past the first answer. “We went with a competitor” is not an insight. “We went with a competitor because we were not confident you could handle our data volumes and nobody addressed that concern directly until it was too late” is an insight.

Ask what they were most uncertain about. Ask what the competitor said or showed that resolved a doubt yours did not. The discomfort in those answers is exactly where your most important work lives.

The pattern across multiple interviews

Individual win/loss interviews are interesting. Ten to fifteen of them, coded and compared, begin to surface meaningful recurring patterns. When the same trigger appears across five conversations, or the same unaddressed anxiety surfaces in three losses with different competitors, you are no longer looking at a single anecdote. You are looking at a positioning problem or a product gap with a growing body of evidence behind it: the kind of evidence that changes what gets built and how it gets sold.

“Lose a deal because of price five times in a row and you have a pricing problem. Lose it because a customer did not believe you could do the job five times in a row and you have a positioning problem. Those require completely different responses, and most organisations never make the distinction.”

The Scale Solution: How AI Makes Continuous Qualitative Research Finally Possible

Most of the practical challenges historically associated with qualitative research at scale now have workable solutions. Getting willing participants is addressed by problem acuteness and by framing outreach correctly, as covered above. The remaining challenges — the time required to conduct, code, synthesise and distribute insight — can now be reduced dramatically with AI.

This is a genuine shift in what is possible. A research sprint that previously required weeks of analysis to produce a findings document can now produce a structured, distributed insight report within hours of the final interview. The workflow looks like this.

The Connect stage is the one most organisations skip, and the one that makes everything else compound. Individual interviews inform individual decisions. A structured, searchable library of coded interviews informs strategy.

When a new market is being considered, filter by segment and see what jobs and triggers exist there. When a product feature is being debated, pull every interview where that job was mentioned. When the dashboard shows an unexpected churn spike, search for every conversation where the relevant anxiety surfaced months earlier.

The qualitative research does not expire. It accumulates.

And it connects: the same intelligence hub that houses your PESTEL macro signals and your competitive positioning should house your JTBD interview outputs. The macro picture and the human picture, in the same place, informing the same decisions.

That is what being genuinely data driven actually looks like: not just fast data, but the right data, at the right depth, maintained continuously and understood in full.

The Why Behind the What

The next time a dashboard shows something moving in the wrong direction and everyone in the room starts reaching for hypotheses, the question worth asking is a simple one: when did we last speak to a customer about this?

Not a survey. Not an NPS follow-up. A real conversation, with a real person, about the job they were trying to get done and whether you are still doing it well enough.

That conversation is not a supplement to being data driven. It is part of what data driven has always meant. The tools to do it continuously and at scale now exist. The only remaining question is whether your organisation is willing to expand its definition of evidence beyond what fits on a slide.

Start with five interviews. Use the ten questions above as your scaffolding. Record everything, code it with AI, and look for the first pattern. You will find it faster than you expect, and it will be more useful than anything you could have arrived at by waiting for statistical significance.

The first interview will be adequate. The tenth will be sharp. The fiftieth will change how your organisation understands its customers.

The insight is there. The framework is there. The tools are there.

That is the job.