Measuring AI’s Impact on Coding: 🌀 Luca Rossi’s Call for Better Metrics

In a recent LinkedIn post, 🌀 Luca Rossi highlights a critical challenge facing the advancement of AI in coding: the need for better measurement. Rossi argues that without reliable ways to assess AI’s contribution, the industry struggles to understand progress and distinguish genuine advancements from hype.

The Problem with Current AI Coding Metrics

Rossi points out that common metrics, such as lines of code (LOC) generated by AI or the percentage of code written by AI versus humans, are fundamentally flawed. These measures, he contends, do not provide meaningful insights into the actual value or efficiency of AI in the development process.

“As an industry, we’ve scored poorly on this so far. We keep framing stories around LOCs written by AI, or % of code written by AI vs humans, both of which obviously don’t mean anything useful.”

The lack of meaningful metrics, according to Rossi, hinders the ability to make informed decisions about AI adoption and improvement. He emphasizes that the focus needs to shift from superficial counts to indicators that truly reflect productivity and impact.

A Two-Pronged Approach to Measurement

To address this measurement gap, 🌀 Luca Rossi proposes a two-part solution. Firstly, he advocates for the continued use of established delivery metrics, such as those found in the DORA (DevOps Research and Assessment) framework. These metrics are valuable because they focus on outcomes, regardless of the tools used, providing an unbiased view of performance.

“Delivery metrics like DORA are still 100% valid. They don’t care whether you use AI, they just measure output. Let’s use them to track improvement in an unbiased way,” Rossi writes.

However, Rossi acknowledges that DORA metrics alone are insufficient because they are lagging indicators. To gain a more proactive understanding of AI’s role, he stresses the importance of developing leading indicators specifically designed to capture how AI is being utilized.

Introducing ‘Leverage’ as a Key Leading Indicator

For these crucial leading indicators, 🌀 Luca Rossi suggests focusing on the concept of ‘leverage.’ This metric aims to quantify the amount of output achieved per unit of human input when working with AI. Rossi illustrates this with a spectrum:

• Negative Leverage: Situations where developers spend more time managing or correcting AI than they would have spent coding the solution themselves.
• Maximum Leverage: Scenarios where AI can accomplish tasks efficiently from minimal prompts, such as a one-line command.

Rossi argues that focusing on leverage is essential to avoid perverse incentives. If the industry only valued correctness or the percentage of AI-generated code, developers might resort to overly detailed specifications or pseudo-code to artificially inflate AI’s contribution, which is counterproductive to the goal of minimizing human effort.

“The amount of input matters because if you only cared about correctness and % of AI-written code, you could score perfectly by e.g. writing incredibly detailed specs that approximate the actual implementation. Heck. you might even write pseudo-code!”

He further elaborates that the ideal scenario involves minimizing human work, making the prompt’s brevity a key indicator of success. This, in turn, necessitates the creation of more robust platforms that AI can build upon.

Building Platforms for AI Collaboration

Rossi concludes by linking the concept of leverage to the need for better AI platforms. He notes that to maximize the efficiency and effectiveness of AI in coding, the underlying infrastructure and tools must be designed to facilitate deeper AI capabilities. He references a recent discussion with Rob Zuber on this topic, suggesting that developing such platforms is a crucial next step for the industry.

“For that, in turn, we need to create more *platform* that AI can build on top of.”

By shifting the focus to measurable leverage and investing in supportive platforms, 🌀 Luca Rossi believes the industry can move beyond superficial metrics and truly harness the power of AI in software development.