The Vanity Cohort
A board nodded at 94% net retention. That cohort was eleven customers. On 87 customers the real number was 71%. Sample size is the tell.
Our Q2 cohort had retained at 94% net. The growth team led the board update with it. Our slide read Retention, Q2 Cohort: 94%. The board nodded. The CEO repeated the number twice.
That Q2 cohort was eleven customers. Three had expanded, one had churned. That 94% was the average of a sample so small that any one customer moved our headline by nine points. The same math on our Q4 cohort, eighty-seven customers, produced 71%.
The board never saw the Q4 number. It saw our 94%.
This is one of the most common reporting failures in growth-stage cohort work. Small cohorts produce volatile numbers. The volatility cuts both ways. Sometimes the small cohort looks great, sometimes it looks terrible. Whoever picks which cohort to feature is quietly choosing the direction of that volatility. Most teams feature the flattering one. We did.
Sample size is the tell. Written honestly, our slide reads Q2 cohort: 94% net, n=11. Given the n, a board asks the next question on its own: what is the Q4 cohort doing? Given only the headline, the board banks it. That is the whole gap between a number that informs and a number that flatters. We had watched the same omission bury a bad quarter in a board update a year earlier.
The deeper habit is to put every recent cohort on one chart. The shape of six months of retention tells you more than any single cohort ever will. If our Q2 is 94% and Q4 is 71%, the company has a structural problem that the newer cohorts are surfacing. The pretty number is the outlier, not the signal. It is the same trap as measuring retention on the wrong cohort: the arithmetic is right and the conclusion is wrong.
The cohort that matters is the most recent one with enough customers to mean something, usually thirty or more. The cohort with the prettiest number is rarely that one.