AI Safety: Rahul Kumar Highlights the Critical Need for ‘No’ in AI Agents

In a recent LinkedIn post, Rahul Kumar delves into a fundamental question surrounding the rapid advancement of artificial intelligence: who controls AI’s actions? Kumar raises a crucial point about the current focus on making AI agents smarter, suggesting that a more pressing concern is establishing mechanisms for oversight and control. He highlights this as a critical consideration for the future of AI adoption and trust.

The Unasked Question: Who Tells AI ‘No’?

Kumar’s post pivots on a simple yet profound observation: while billions are invested in enhancing AI capabilities, the mechanisms for an AI agent to refuse an action are often overlooked. This realization, he explains, struck him while exploring Kastra, a company focused on AI infrastructure.

He elaborates on the current landscape of AI infrastructure, noting that most systems prioritize elements like authentication, monitoring, logging, and evaluations. While acknowledging their necessity, Kumar points out that these typically function reactively, documenting what has already occurred rather than preventing it.

“Most AI infrastructure today is built around authentication, monitoring, logging, and evaluations. Those are essential. But they usually tell you what happened after an action has already been taken.”

This reactive approach, according to Kumar, is insufficient for building trust in increasingly autonomous AI systems. He argues for a proactive stance, where AI actions are evaluated *before* they are executed.

Introducing Runtime Authorization for AI

Kumar introduces Kastra’s approach as a significant departure from the status quo. He describes it as a “runtime authorization layer” designed to vet every AI action in real-time.

As Rahul Kumar explains:

“It acts as a runtime authorization layer, evaluating every AI action before it executes. Whether an agent wants to access sensitive data, use a tool, or make a production change, policies decide if the action should be allowed in milliseconds.”

This pre-emptive evaluation, Kumar suggests, is a fundamental shift. It moves the focus from AI’s intelligence to its safety and reliability. He posits that the success of AI in the future will not be solely determined by the sophistication of its models but by the degree to which businesses can trust its operations.

The Future of AI: Trust Over Raw Intelligence

Kumar draws a parallel between the evolution of software security and the future of AI. Just as authentication became a foundational layer for software applications, he believes runtime authorization could become a standard for autonomous AI systems.

This perspective challenges the prevailing narrative that emphasizes AI’s raw capabilities. According to Kumar:

“We’re entering an era where AI won’t be judged only by how intelligent it is. It will be judged by how safely it operates.”

He concludes that the companies poised to lead in the AI era will be those that can demonstrate robust control and accountability within their AI systems. This focus on confidence, control, and accountability, rather than just enhanced capability, represents the true future of AI infrastructure, according to Kumar’s analysis.

Kumar also shared that Kastra has recently launched on Product Hunt, inviting interested parties to explore the product and provide feedback.