The Product Progress Fallacy
Why progress is rarely linear, and what to do about it
In this post we’re going to look at how to think about progress: why releasing features is insufficient (and can even be detrimental), and three principles for maximizing progress in the context of uncertainty.
How we think progress happens
If you and your team are following an Agile methodology, then you should be shipping working software on a regular basis. It’s tempting to equate shipping with success: if each shipped piece of software is adding value, shipping regularly is an improvement over waterfall-style development because that value is delivered sooner to customers.
The result is we tend to think of progress in this way:
But as any team who measures the impact of their work knows, not everything that is shipped actually creates value. It’s possible to ship a lot and have no impact. Worse, it’s also possible to ship a lot and have a net negative impact!
Setting value as the goal
The idea that it’s possible for our hard work to harm our customers’ experience is uncomfortable, but acknowledging this actually helps us approach product work more effectively. Let’s explore why.
First, everything we build is based on a theory about what our customers need and how to solve that need. Shipping is an exercise in experimentation1.
Second, our experiments often fail. The failure rate for startups is somewhere around 90%. Even if the failure rate for features in existing products is lower, it’s well above zero. And if you don’t take the time to pull those failed features back out of the product, that’s a lot of dead weight.
So, shipping working software shouldn’t be the end goal. Instead, the goal should be to release software quickly to get validation on its value, and iterate from there.
This sounds simple, but I find it's actually quite hard to operate as if these facts are true. The process of defining work and getting it shipped requires a lot of thought, care, and effort, such that shipping software truly does feel like something worth celebrating! But defining success as value delivered rather than software shipped means acknowledging that failure will frequently be the outcome. Rather than ignoring those failures, we must face them, and then push past them.
A better way to think about progress
If the linear model of progress shown above is a fallacy, then how should we think about progress? My belief is that progress looks more like this:
This line looks chaotic and unpredictable, and to an extent that’s true. But when you zoom in, there are some patterns that I think are helpful for thinking about progress and managing to maximize it.
Principle 1: assume a tax
We started off on a depressing note, so let’s get down into the valley before we try and dig ourselves out. The first thing to acknowledge is that everything you build has a price: complexity (not to mention the monetary cost of your time). Think about some of your favorite products. I am willing to bet that the majority have a common theme: simplicity. They do a small number of things, and do them better than anyone.
Contrary to our belief that if we just add this one feature, our fortunes will turn, typically, less is more. Poor teams are hoarders - they try to add more and more, never willing to give things up. The price they pay is in increasing clutter and complexity, resulting in a bad user experience (let alone the tax paid by the team on managing this complexity). And complexity compounds: the more you add, the steeper the tax.
Great teams are essentialists - they cut out anything that isn’t delivering big on its promise, and they do so quickly and decisively. As we’ve discussed before, there is an opportunity cost to everything; by over-committing to failing ideas, you’re using time that could be spent exploring the next idea2.
Principle 2: squeeze all the juice
If we think about principle 1 as minimizing downside, this second principle is about maximizing upside. I have found that winning ideas frequently breed more winning ideas. This is because those ideas are underpinned by some insight or behavior that can be applied more broadly. For example, one big win we had at Pack Health was switching from a model of assigning our coaches to new members based on which coach had the most capacity to one that was based on which coach was free at that member’s preferred time (regardless of whether they had the most capacity). This shift led us to a number of other wins centered around making the member’s first appointment as easy as possible (by increasing the diversity of time slots available, thoughtfully reminding the member of their appointment, and so on).
I think of this as squeezing all the juice out of your key insights. When you solve a problem, it’s tempting to assume it's solved fully and move on to the next pressing thing you have to do. But by doing this you pass up on a large portion of the benefit you can gain from the insights you uncovered in your pursuit of solving that problem. Like digging in a mine after you find the first nugget of gold, now’s the time to dig deeper until you start to see diminishing returns.
Principle 3: use all your senses
The product progress fallacy shows us that adding value is rarely linear, and can’t be measured simply by your team’s velocity. Measuring value properly is hard, and without that measurement, you can’t know if you’re making progress, nor if your initiatives are working. You also can’t learn from the mistakes that you’ve made, or unwind them (see Principle 1).
Without intentional effort and investment in tools and infrastructure to measure your outcomes, it’s too tempting to default to tangible but flawed proxies for success, like the amount you ship. So how can we develop a good signal for whether or not we are truly delivering value?
First, use all your ‘senses’ - including qualitative as well as quantitative data. For example, concept interviews and usability tests can give you an early warning of features that might miss the mark before you’ve made an investment in code that is harder to discard, whereas experimentation such as A/B tests helps tease out causal relationships (was it X that moved this metric, or something else?)
Second, use an organizing framework to explain what you are measuring, why, and how it fits into the bigger picture (e.g. growth loops, your customer journey). This helps keep everyone focused and provides a method for interpreting the results and insights your team is gathering. For example, a growth model can be used as a basis to discuss the levers that can be pulled to accelerate growth, and where trade-offs might occur (such as a tactic that increases acquisition but decreases conversion).
Third, find good input metrics that you can use as proxies for the outcomes you really care about (which are typically lagging indicators)3. This ensures your team has a strong enough signal to feel like they have the agency to influence things. Outcomes like retention can feel like they are everyone’s responsibility and therefore nobody’s focus. Continuing the Pack Health example above, one key metric for us was the proportion of weekly calls that were completed without a no-show. “First time” completed calls maintained a member’s momentum, and were specific enough to prompt ideas for improvements (such as call reminders, mid-week activities that helped members prep for their call, and easy advance reschedules).
Making progress in an uncertain world
We’ve talked about ways to make true progress in the context of uncertainty. But given this uncertainty, is it possible to make predictable progress? I think this depends on perspective. If your goal is to achieve predictability for each initiative you run, then you’re going to struggle. But achieving predictable progress over the long term is possible. The key is focusing on winning the war, not the battle.
If most initiatives fail, then we need to find ways to maximize our shots on goal, be rigorous about evaluating what we put out into the world, and clean up after ourselves when we miss the mark. This includes planning for failure and making sure you can revert your changes if necessary.
If true insights can pay off in multiple ways, then we need to think of ourselves as birds searching for an updraft. When we find that updraft - that insight - we need to ride it as high as we can. Just grinding to produce as much output as possible won’t get us nearly as far.
And if, moment to moment, the way is uncertain, then vision becomes very important. Rather than trying to map a path from A to B, we need to stay focused on our desired destination, get good at navigation, and remain flexible on how to get there.
What techniques have you found helpful in making sure you’re staying focused on delivering value for your users? Have you found yourself falling prey to the progress fallacy? I’d love to hear from you!
🙏 Many thanks to Yael Schwartzman and Michael Harrison for their thoughtful feedback and ideas on an earlier draft of this post. 🙏
For some teams, this might feel far from the current reality of delivery deadlines and feature roadmaps. Janna Bastow makes the point that for most other functions in the business, experimentation is an assumed and inherent part of the work, whether that’s a sales team’s weighted pipeline, or a marketing team’s experimentation around ads.
Michael shared a great analogy for this with me: poker. Professional poker players only play 15-25% of hands, while amateurs play 50+ percent. You have to fold your failing opportunities quickly in order to remain profitable; the same is true in product. Weaker product teams persevere with failing ideas because they fear locking in sunk costs and feeling like their efforts were a waste.
For a great case study on how to identify the right input metric, see Working Backwards: Insights, Stories, and Secrets from Inside Amazon by Colin Bryar and Bill Carr. The key point: input metrics take iteration to get right; start with something and then adapt as you identify the limitations of the metric you’ve chosen.