Why ‘Experimentation at Scale’ Often Fails, According to Janna Bastow

In a recent LinkedIn post, Janna Bastow discusses a common pitfall that hinders successful “experimentation at scale” within organizations. According to Bastow, the primary reason for failure is a lack of organizational agreement on what constitutes a safe area for experimentation.

Bastow highlights a conversation with a ProdPad partner who described the intricate web of dependencies in modern business, particularly when customer promises are involved. “One small change can ripple across stores, supply chain, partners, and ops,” Bastow relays from the partner’s experience, illustrating how interconnected systems limit unbridled testing.

The Discipline of Experimentation

This interconnectedness, Bastow argues, necessitates a disciplined approach to experimentation. It’s not feasible to A/B test every aspect of a business, nor is it always possible to move at high speed across all fronts. Bastow emphasizes the need for clarity regarding which areas offer cheap learning opportunities and which involve significant costs or risks.

“That reality forces discipline. You can’t A/B test everything. You can’t move fast everywhere. You have to be explicit about where learning is cheap and where it’s expensive.”

Bastow points out that successful teams, in her view, are not necessarily less ambitious but are more precise in their strategy. They distinguish between decisions that are reversible and those that are irreversible. This allows them to protect the core customer experience while simultaneously creating dedicated spaces for testing on the periphery.

Moving Beyond ‘Experimentation Theater’

A key takeaway from Bastow’s post is her critique of viewing experimentation as a mere virtue signal. She contrasts this with a more pragmatic perspective where experimentation is treated as a carefully applied tool. This distinction is crucial, as Bastow frames it, in differentiating genuine learning from what she terms “experimentation theater.”

Separating Reversible from Irreversible Decisions

Bastow elaborates on how effective teams manage this balance. They don’t shy away from ambition but instead channel it through strategic clarity. This involves a conscious effort to identify and safeguard critical operational elements while fostering an environment where innovation can be explored safely.

“The teams that get this right aren’t less ambitious. They’re clearer. They separate reversible decisions from irreversible ones. They protect the core experience while creating space to test at the edges.”

The core message from Bastow is that effective experimentation at scale is not about the volume of tests but about the intelligence and safety with which they are conducted. By understanding the true cost and impact of changes, organizations can foster a culture of learning without jeopardizing essential operations.

“What I liked about this conversation was the lack of romanticism. Experimentation wasn’t treated as a virtue signal. It was treated as a tool, applied carefully.”

Bastow concludes by posing a critical question to her audience: “How does your team decide what’s safe to test and what absolutely isn’t?” This prompts further reflection on the practical implementation of strategic experimentation within business contexts.