Vistaly’s AI Overhaul: Teresa Torres Details Rebuilding Product for Agentic Workflows

In a recent LinkedIn post, Teresa Torres discusses the significant undertaking by Vistaly, a company she partners with, to rebuild its entire product around AI-powered workflows. The post, which is a promotion for an episode of her podcast “Just Now Possible,” details the transition from V1, a continuous discovery canvas, to V2, a ground-up rewrite focused on leveraging AI to synthesize customer interview data into opportunity solution trees.

Teresa Torres highlights the ambitious nature of this rebuild, noting that Vistaly chose to start from scratch rather than layer AI features onto their existing platform. She explains the core functionality of V2:

“upload customer interviews, get a snapshot of each, and let an agentic workflow draft and update your tree.”

The article delves into the challenges encountered during this transition, particularly the issues that arose when AI was applied to the complex task of opportunity mapping. Torres elaborates on the difficulties Vistaly faced, including user feedback on slow synthesis processes and a critical problem they termed the “house of cards.” This metaphor illustrates how a single poorly framed opportunity at a lower level could negatively impact the entire structure of the solution tree above it.

The “House of Cards” Problem and AI Evaluation Hurdles

Teresa Torres emphasizes that the AI’s output was not always reliable, leading to the “house of cards” issue where errors cascaded through the opportunity tree. She describes the rigorous evaluation process undertaken to address user complaints, such as one user noting they would “have to come back and clean up this branch.” This involved experimenting with various AI evaluation methods.

AI as a Judge and Orchestration Fixes

One of the key challenges highlighted by Torres was the unreliability of using a large language model (LLM) as a judge for the AI’s output. She found that this approach was difficult to calibrate effectively. The breakthrough, according to Torres, came not from refining prompts for the LLM, but from implementing a fundamental change in the orchestration layer.

“Why the fix was a repair loop in the orchestration, not a better prompt—and how that eval became a production guardrail.”

This repair loop allowed the AI agent to understand and follow specific rules, enabling it to log semantic moves like merging, moving, or reframing opportunities, thereby producing more consistent and reliable change sets. This focus on the underlying workflow and rules, rather than just prompt engineering, proved crucial.

Data Quality and Synthetic Testing

Torres also touches upon the critical importance of data quality for AI systems. Beta testing revealed that the quality of input interviews varied significantly, with some users uploading sales demos, stakeholder meetings, or even LLM-generated fake transcripts as interview data. To combat this, Vistaly developed a sophisticated system for generating realistic synthetic transcripts to serve as test data.

Rebuilding at Speed

The post underscores the impressive speed at which Vistaly rebuilt nearly all of V1’s functionality in just two and a half months, a stark contrast to the three years it took to build the original version. This rapid iteration was essential for V2’s agentic workflow capabilities.

“V1 was a purpose-built canvas for continuous discovery. V2 is a ground-up rewrite: upload customer interviews, get a snapshot of each, and let an agentic workflow draft and update your tree.”

Teresa Torres’s coverage of Vistaly’s AI journey provides valuable insights into the practical challenges and innovative solutions involved in integrating advanced AI into product development, particularly in the complex domain of opportunity solution mapping.