Lost confidence

source

Confidence games

Many prioritization frameworks include a measure of confidence¹⁠—how sure we are that we can execute, at more or less the predicted effort, resulting in more or less the predicted impact. This seems rational: if two projects generate equal value for equal effort, but we’re confident we can execute the first and unsure about the second, we should select the first.

¹ Or a measure of risk. The reader will decide whether risk is equal to 1—confidence.

This is not, however, how confidence scores are used. If it were, the process would look like this:

1. Score projects somehow.
2. If there’s one clear winner, do it.
3. If there’s a tie, pick the one we are more confident in.

That’s not the worst idea. But popular frameworks like RICE include “confidence” in step one:

Or RPS:

Which means, for example, the following two scenarios are deemed equally strong:

1. A small incremental feature, that we’re sure we can execute.
2. A large feature, with large impact, that carries some risk.

This equality is false. Especially when you remember that we’re typically confident in small projects and unconfident in larger projects, as well we should be.² But that systematically skews the prioritization away from delivering as much value as possible, which I would argue is the opposite of what a prioritization framework ought to do.

² If you disagree, consider that the entire motivation of the Agile movement was that we should always have low confidence that large projects will be successful, despite our best techniques of planning, analysis, and estimation. And consider this theory of Rocks, Pebbles, and Sand.

I don’t believe your confidence score anyway. First, because it’s ill-defined. What does “30%” mean? What it should mean, is you track your confidence scores across all projects, and then later measure how accurate you were, and then determine how accurate your “confidence” scores were with mathematical precision. But you don’t do that, do you? And if you only ship a few major features per year, you don’t have enough data to know, even in hindsight.

The second reason I don’t believe you is because we all know that projects are nearly always late, and often have less impact, less quickly, than we wanted. No matter how confident we were. Indeed, everything we choose to do, we have at least “pretty good confidence in,” or we wouldn’t do it at all! So what weight should we place on “confidence?”

Hofstadter’s Law
It always takes longer than you expect, even when you take Hofstadter’s Law into account.

To prove this, find any experienced Product Manager and ask: “Can you recall a feature you were certain would be well-received, but wasn’t?” Perhaps they had evidence from customer conversations, explicit requests, or purchase commitments⁠—yet after building it, almost no one used it, including those who promised they would. Their eyes will roll as they share multiple stories. This doesn’t make them a bad PM. Everyone has these stories. The best PMs have techniques to mitigate this problem,³ but none will claim they can eliminate it entirely.

³ Some techniques to improve prediction include asking customers to describe exactly how they would use a feature in their normal workflow. Often people genuinely think they would use something, but when forced to walk through it step-by-step, they realize, “Oh wait, this would require me to rewrite this code, we probably wouldn’t do that.” Or, “I’d need to export it into another system⁠⁠—actually, never mind.”

Similarly, ask content creators about their most successful work. Often it’s something they hastily produced⁠—a trivial piece they almost didn’t publish because it seemed uninsightful or trite⁠—yet it generated more views and engagement than anything else that year. Conversely, pieces they spent dozens of hours crafting, work they’re genuinely proud of and consider their best, generate minimal interest:

source

We can summarize the relationship between our confidence and actual results in a handy two-by-two table:

So, if “confidence” is too nebulous to define, and we shouldn’t trust ourselves with it anyway, what should we do?

What to use in place of confidence and risk

The answer lies in the realm of uncertainty, rather than of probability.

Probability presumes you know the underlying distribution, enabling mathematical predictions about future events. You can predict that flipping a fair coin 100 times is highly likely to result in between 40 and 60 heads, because you know the underlying distribution. If predicting whether a feature will create defensible differentiation were like coin-flipping, you could use probabilities.⁴

Almost nothing in a startup is like that. Outcomes cannot be assigned meaningful probabilities because things like startup success, strategy, and features are unprecedented, or too complex to model accurately, or we have no precision on the input variables. This is the domain of uncertainty.⁵

⁴ If you’re tempted to claim that Bayesian methods could still work, remember that you need numeric priors and conditional probabilities, both of which we established above are unknowable and ill-defined.

⁵ Often called “Knightian Uncertainty” after economist Frank Knight in his 1921 work “Risk, Uncertainty, and Profit.”

In this domain, we ask: What actions are wise regardless of the probability distribution?

I’ve previously written about embracing uncertainty in overall product strategy. Below, I’ll address a more specific question: How should we prioritize individual items in an uncertain world?

Here are some techniques.

True always

What is always true under any circumstance? This is Bezos’s principle of focusing on long-term constants.⁶ For instance, users universally appreciate fast, responsive software. They value web apps that feel native, with background synchronization and instant interactions, that work well on all their devices. Even in the worst case, where they don’t care about speed, they still wouldn’t say that speed was a negative; at best (in web apps like Notion, Miro, Gmail, and Google Docs), performance becomes a key differentiator that customers explicitly value.

⁶ Bezos frequently said of Amazon’s strategy: When you have something that you know is true, even over the long term, you can afford to put a lot of energy into it. His examples include customers wanting lower prices, faster shipping, and fast, fair customer service.

Not all features enjoy universal appeal. Rather than attempting precise numerical breakdowns of potential user interest, identify the features where essentially all customers will either value them, or at least enjoy them. Sometimes this certainty exists because it is mandatory, even if mundane. Enterprise requirements like SOC 2 compliance aren’t exciting, but they’re undeniably valuable when selling to the Enterprise. This certainty compensates for the lack of differentiation.

The caveat: your most innovative, differentiated ideas rarely fall into this “absolutely certain” category. While certainties are valuable, they’re unlikely to be your strategic differentiators. This tension is natural⁠—great products require both reliable improvements and innovative leaps of faith.

source

Quick discovery, quick recovery

I’ve been a long-time advocate of systematically interviewing potential customers to validate ideas before you start building. Still, I have to admit that this falls into the “confidence” trap. You never really know until you build. (You can, however, invalidate before you build, saving you months if not years of wasted time; therefore this is still the right place to begin.)

The typical solution is to build an SLC (my upgrade to the concept of the MVP)⁠—a simple but completed product that generates genuine feedback. Experience, rather than prediction. For existing products, that means maintaining a balanced portfolio between guaranteed wins and innovative bets, applying different validation methods to each.

For example, consider implementing “dummy features”⁠—buttons that, when clicked, reveal: “This feature isn’t built yet. Tell us how you’d use it.” This simple test provides real signals: a count of interested users and potential interview candidates who’ve demonstrated interest through action rather than words. They can provide insights before you build the feature.

This approach generates 100x better signal than surveys asking hypothetical questions. People easily say “yes” to survey questions about whether they’d use some feature, but taking an action⁠—even clicking a button⁠—requires genuine interest. Observed behavior beats stated intentions.

Focus instead on customer impact

Replace confidence with impact. I define impact in two distinct ways:

Majority rule

When the majority of users regularly use a feature, it’s undeniably important⁠—likely a key reason people adopt and retain your software.

Passionate advocates

Features that create passionate advocates among a smaller subset of users⁠—people who say they bought your product only because of this one thing. These “magnificent delighters” won’t appeal universally, but they inspire deep loyalty in specific segments. Like a piece of music that’s someone’s all-time favorite, even if most people merely acknowledge it’s decent.

These are what determine purchase decisions. Your product rarely satisfies every customer need perfectly, but when users absolutely love certain aspects, they’ll tolerate shortcomings elsewhere. We see this with beautifully designed software⁠—users accept missing functionality or limited platform support because the design experience itself is so compelling. There are many other reasons for a customer to love you despite your failings.

I quantify impact with this definition: A high-impact feature is either (1) regularly used by at least 51% of customers, or (2) cited by at least 15% of customers in their top three reasons for choosing or retaining your product.

This sets a high bar, but innovative, risky features demand a high bar. If you’re undertaking projects that might exceed timelines or have uncertain outcomes, the magnitude of the potential reward must justify that risk.

Invest in leverage

There are some aspects of the business or product where small, incremental changes yield large results. It sounds too good to be true, but there are mathematical or structural areas where it is almost always true.

Here I’ll promote my new book Hidden Multipliers, which details eleven such areas of leverage, where small changes create outsized growth with your current budget and team.

There. End of advertisement.

Maximize optionality

If we don’t know how the future will unfold, we can make choices that maximize the options we have when we get there. More than flexibility, more than avoiding lock-in, building systems that are almost always ready to handle anything that arises.

Some examples:

• Keeping costs low enables all kinds of pricing and packaging while still being profitable, allowing for testing today and resilience in the future.
• Selecting well-established, actively-developed, cross-platform libraries and frameworks for building user interfaces, so you’re able to handle any evolution in platforms and devices.
• Plug-in systems, so that both you and your community can build things that you cannot imagine today.
• API-first architecture to isolate teams, and connect your own front-end tools, back-end systems, and customer integrations.
• Wrappers around vendor services, so that you can swap out vendors if one becomes unstable, or too expensive, or lags behind others.

Future optionality requires additional work today. For example, adding vendor-wrapping APIs doesn’t add any value today. Those techniques are wise for mature companies where stability and predictability are more important than releasing a feature a month earlier, but might be the wrong choice for early-stage companies that must rely on their velocity to win against incumbents.

Portfolio of bets

Portfolios reduce variability at the expense of reducing maximum upside. That is, you’re unlikely to have zero wins (so your downside isn’t too bad), but wins have to make up for the losses, so even the occasional massive win isn’t nearly as massive as it would have been. The old joke is that the best investment portfolio would have been to buy Amazon at IPO and hold forever. Sure, but if you applied that advice to some other IPOs that year, you’d have $0. A portfolio of stocks means you’ll never go to zero, but your maximum growth will be far less than the best stock in the portfolio.

Mathematical sidebar

Why do portfolios work regardless of the underlying probability distributions? The Central Limit Theorem makes this precise: Suppose you draw a sample of N items selected at random from some population, where the population has a stable but unknown probability distribution. That sample will have some mean. Now do it again with a new sample of N items, and you get a new sample mean. Keep repeating that, and plot a histogram of those sample means. That histogram is always Gaussian⁠—a normal distribution⁠—with a mean equal to the population’s mean and a variance of the population’s variance. So, total portfolio results are normally-distributed regardless of the underlying probability distribution, and we expect results near that underlying mean, which means not zero, but also not near the maximum value of any one element in the underlying data set.

Even further, the The Lindeberg⁠—Lévy Central Limit Theorem shows that the same is true even when each sample is drawn from a different underlying probability distribution. This holds only under certain conditions (independence, finite variance, and no single variable dominates all others). Arguably these conditions fail with distributions common in startup environments, e.g. some Power Laws have infinite variance.

Portfolios work when you value being predictable, but they don’t work when you want outlier results. An example of the latter is a venture or angel portfolio, where 65% of those portfolios lose money and only 10% generate returns high enough to justify the risk and illiquidity. When hunting outliers, you need all-in investments, not portfolios.⁷

⁷ Mathematically, the reason for this breakage is that the underlying distribution of startup returns is a Power Law that violates the Lindeberg criteria mentioned above.

Therefore, if the goal of your prioritization exercise is to find features that will be strong differentiators in the market and strong growth drivers, a portfolio is the wrong tool. On the other hand, if you’re prioritizing a bunch of smaller things, where you want incremental but reliable results, a portfolio will get you those results. No need to argue about confidence.

Asymmetric bets

The natural counterpart to a portfolio. Portfolios work when you want reliability; asymmetric bets work when you want outliers. You’re not predicting whether the bet pays off⁠—we’ve established you can’t⁠—you’re taking bets with limited and quantifiable downside, yet with much larger upside.

If the worst case is “we lose two months” and the best case is “we’re completely differentiated,” the probability barely matters. You only need the downside to be survivable, and the upside large enough that one win pays for ten losses. The exact probabilities are unknowable; the shape of each bet is not.

Most asymmetric bets at the strategic level arrive pre-shaped⁠—a new market with compounding referrals, a moat that gets deeper the longer you dig. At the feature level, you usually have to construct the asymmetry yourself, because the default shape of a software project is a slow drift from “two weeks” to “two months” to “we’ve spent so long we have to finish it.” The mechanism is a budget you write down before you start: two engineers, three weeks, then we stop and look. The time-box is the bounded downside.

Another thing you can do is replace “confidence” with “how asymmetric.” RICE asks you to estimate a probability you’ve already conceded you can’t know. Asymmetry asks you to estimate two things you actually can: the worst case in time and money, and the best case in customer value. Both are concrete. Both you can argue about in a meeting and converge on a number you’d defend, at least if you restrict yourself only to powers-of-ten for all numbers, as you should anyway.

Stop pretending you can quantify confidence, or even define it.

Instead, use techniques that work whenever the future is unpredictable.

Because it always is.

Printed from: A Smart Bear
https://longform.asmartbear.com/confidence/
© 2007-2026 Jason Cohen      @asmartbear