There are few things worse than wasted time. In a business world where less than 3 of 10 startups reaches an exit, and many product launches fail to reach their potential, we need to make better, more informed decisions. And we can’t do this by just collecting more metrics. We need customer stories.
But how many calls are enough? Win/Loss and Churn interviews is not quantitative data; we don’t need thousands of data points to draw conclusions or produce insights. But neither will a few interviews help address a major investment decision. Indeed, I’m sure most of us can recall a product or feature release that was developed on the basis of anecdotes from a single lost deal.
So if you’re thinking about doing Win/Loss interviews, or Churn/Renewal interviews, you need to know: how many calls are enough? With dozens of client projects under our belts at Eigenworks, and thousands of buyer interviews, we can set this record straight.
The primary concept to understand for Win/Loss and Churn/Retention Analysis is “Saturation”, which we will define shortly, and then cover two starting steps: 1) how many conversations allow you to reach saturation, and 2) how to choose which buyers, potential buyers, users, or customers, to include in your study. If you don’t nail the sample size, you run the risk having a collection of win/loss interviews too small to be useful, or too large to be economical. And choosing people to interview is equally important to ensure you’re getting insights relevant to your research goals. Only by being smart with your time and effort can you get the value from a Win/Loss project that you need.
So let’s talk about sample size and reaching saturation.
What is saturation? There’s a long-winded academic answer to this question – the short version is that it’s the point at which talking to more people is no longer useful. It’s the point at which you start hearing the same things over and over again: when a 15th interviewee says the same thing fourteen others have, when no new information emerges. Saturation is the point at which everything starts sounding the same.
How do you identify saturation? You could contrast and compare every interview as it comes in, but that’s far too time consuming, and runs the risk of sequence bias if, for example, your first five interviewees all say the same thing. Sequence bias looks like saturation so you stop at five interviews, not knowing that your next ten might have all said something different. The solution is what we sometimes call ‘the magic number’ – that is, the ideal number for sample sizes for qualitative research that’s been hashed-out by scientists. Academics have studied this at length, and debate different approaches such as “phenomenology” and “grounded theory;” but again, we need a short answer here. We need a number.
So: How many Win/Loss calls are enough? Around 20-30.
A sample size of 20-30 gives you the best trade-off between time and clarity. You get enough interviews that you start to see saturation among your targets, but not so many interviews that you waste valuable time overcompensating for sequence bias. (Not every analysis absolutely must be the full 20-30 set – there are times where smaller data sets better balance efficiency over time - but we’ll cover these another day.)
The amazing thing about this range of participants is that it holds for large and small populations. Many SaaS customers ask about doing interviews at scale, and our experience with this is the same. Perhaps in more homogeneous populations, you can err closer to 20 participants, and with large populations, closer to 30, but we’re in the ballpark. Companies with hundreds of thousands or more customers likely have a larger number of market segments, but for each segment, we can still get saturation on themes by talking with 30 people.
How do you choose customers to interview?
Depending on your business, choosing people to interview may be easy or difficult. The more customers, or the more transactions you close per month, the harder it can be, and the more careful you need to be in making your selection. If you have a smaller number of transactions, even finding 20 interviewees may be difficult.
Regardless, you need to choose them. This step is all about segmenting your audience, customer base, or market. You could look at deal size, potential enterprise value, geographical location, or any number of other ways of slicing and dicing your base of prospects.
We are often asked for guidance on this, and we offer two rules of thumb:
- Pick the segment with the largest risk, or the segment that represents the largest untapped opportunity
- Choose a segment with enough things in common that you truly are getting 20-30 representative conversations in the segment.
If you’ve got a host of buyer segments that could use win/loss analysis, focus on the extreme edges: that product that never took off against its competitors, or that vertical that can’t seem to get enough of your services. You will also need to pick the deal types to focus on; we recommend never doing just wins or just losses – do a balance of both to get a good handle on what is going right and wrong. The deal type, then, is between standard sales (win/loss) and churn (the latter of which may not be a problem for those of you not in the SaaS world).
Finally, don’t try to spread yourself too thinly. Some prospective clients come to us and ask to spread 20 interviews over 6 different market segments, which would yield only a small handful of interviews in each segment. If these segments are truly unique, we end up unable to say anything definitive about any segment: we’re back to anecdotes.
If you do have many segments that need addressing, I always recommend starting by focusing on one segment, or at most two, and then expanding the study later on. You’ll likely get economies of scale once you study one segment in detail and can do a sort of “diff” when you move to other segments. I coined a term for this: The Rotating Lens, and this approach is very powerful for companies with many product lines to look at. Come to think of it, I should write about that sometime soon, too.
I hope this addresses the question of how many calls is ‘enough.’ Where do you go from here? Start by writing that risk and opportunity assessment – this will be your business case for justifying the research ahead of you. Demonstrate need: show that Win/Loss analysis is not just valuable in theory, but that it has concrete applications to your company’s problems. If you’re a junior, it’s also time to start hunting for a patron - someone who is willing to support your initiative at the higher level. You know that you need Win/Loss Analysis – so make your case to someone who will listen and recognize its value.
It’s time to stop thinking about a Win/Loss or Churn project and time to start actually doing something about. So get started – you’ve got about 20-30 phone calls to make.