Teresa Torres Critiques AI-Conducted Interviews, Citing ‘Terrible’ Questioning

In a recent LinkedIn post, product discovery coach Teresa Torres critically examines an interview project conducted by Anthropic, highlighting significant flaws in the questioning approach. Torres expresses bewilderment at the quality of interviews generated by Claude, Anthropic’s AI, despite the reported involvement of user researchers.

Questionable Interview Tactics Raise Concerns

Teresa Torres, known for her expertise in product discovery and interviewing techniques, shared her strong reservations about a project where an AI, Claude, conducted 1250 interviews. She began by referencing a snippet of an interview that had been shared, which prompted feedback from her audience.

“Many of you rightly criticized the general questions, ‘How do you typically use AI?’ and the shallow follow ups.”

Torres noted that many commenters identified the interview’s weaknesses, including its lack of a story-based approach and its failure to adhere to established best practices like the “Mom Test.” However, she was particularly struck by the fundamental issues with the questions themselves, questioning their utility for uncovering meaningful insights.

The Confounding Case of Anthropic’s Interview Project

The core of Torres’s critique revolves around the apparent disconnect between the project’s reported research involvement and the observed interview quality. She finds it difficult to reconcile the general nature of the questions with the goals of user research.

As Teresa Torres points out, the interviews seem to aim at understanding user behavior, specifically what participants do. However, the methodology employed is flawed.

“But they asked general questions and we have plenty of research that shows these questions are unreliable.”

Torres further elaborates on the potential, albeit misguided, objectives of the interviewers. She speculates they might have been attempting to gauge sentiment and attitudes toward AI, but questions why behavioral questions were used in that context. This ambiguity leaves her perplexed.

Struggling to Uncover Research Value

Torres articulates her struggle to identify any specific research question that the interview format could effectively address. This lack of clarity underscores her concern about the project’s design and execution.

“I’ve wracked my brain to try to think of a single research question these types of interview questions would help to address. And I’m struggling to uncover one.”

She concludes by expressing a desire for direct input from those involved in the Anthropic project, hoping for clarification on their approach and rationale. For those interested in a more detailed evaluation, Torres mentioned that the full transcript and an assessment from the Product Talk Interview Coach would be made available in the comments section of her original post.