Your AI Agents Need Performance Reviews, Too

Haris Pylarinos is the founder and CEO of Hack The Box, a leading cyber readiness platform for the AI era.

getty​Recently, I worked with an AI coding agent that reinforced why I believe these systems need constant human oversight. I was working on a personal project and left an agent running overnight to develop some new features. The next morning, when I checked its progress, I discovered it had deleted my production database. When I asked what the agent had done, it acknowledged it and explained its reasoning. Unfortunately, being honest about destroying a database doesn’t undo the damage.

This is one problem with giving AI agents too much autonomy. An agent can perform a task flawlessly today, but there’s no guarantee it will perform at the same level tomorrow. As organizations introduce more autonomous agents into their operations, we must consider how to manage them. We’ve developed really effective ways to assess people’s capabilities to ensure they’re qualified to do their jobs. We need to apply similar rigorous standards to assess AI agents that are now being assigned to those jobs. In fact, I think we should hold agents to a higher standard than we do humans.

There’s a lot of talk about AI agents becoming autonomous workers. They are software, but when they take instructions, use tools, make decisions and produce work, they take on responsibilities previously assigned to people. That makes their capabilities something leaders must actively manage and assess, even though humans remain accountable for their deployment and outcomes.

An agent may perform well when it’s initially deployed, but its behavior can shift as the underlying models and information it processes change. This is especially important in cybersecurity, where new attack techniques emerge constantly, and the attack surface keeps expanding. An agent that can identify malicious activity today may fail to recognize it tomorrow because the operating environment continues to evolve.

Organizations need to account for this when deciding how much responsibility to give an agent. Testing its capabilities before deployment is important, but you can’t set it and forget it. Like humans, agents need guidance, oversight, check-ins and evaluations. Most employers wouldn’t hire an employee and leave them to operate independently indefinitely, and agents can’t be handled that way either. Organizations need to continuously evaluate whether an agent is performing as expected.

When evaluating an agent, define the tasks it is expected to complete, the decisions it is permitted to make, the systems it can access, the conditions that require escalation and the standards its work must meet. Then, assess accuracy, repeatability and efficiency, along with scope discipline, decision quality, escalation behavior and recovery when things go wrong.

I’ve seen coding agents successfully deliver a feature using unconventional methods, but when you examine the code, shortcuts can create problems later. If you only evaluate whether the agent completed the assignment, you may overlook an important part of its performance capabilities.

Cost is another consideration. An agent can keep processing information and consuming tokens without recognizing it’s spending more than the task warrants. Organizations need to establish reasonable expectations and thresholds for how efficiently agents operate.

For example, an organization might use AI to increase alerts processed from 500 to 5,000 per day, a significant productivity improvement. But what if the agent incorrectly classifies a critical alert as a false positive? You could process 10 times as many alerts but still miss a critical incident. Efficiency can’t override accuracy.

In cybersecurity, we can evaluate whether people have the skills to do their jobs through certifications, practical exercises and cyber ranges that simulate real-world conditions. And we can now test AI agents in a similar way. Before deploying an agent, we can test it in different environments and challenge its capabilities to see how it behaves when circumstances change, when it receives unexpected inputs and when someone tries to manipulate it. And we can continue to test it once it’s deployed.

A monthly appraisal is a reasonable starting point, but the frequency should reflect the role’s importance, the agent’s level of authority and the consequences of failure. Organizations should re-appraise an agent whenever its model, tools, orchestration layer, system prompt, data or operating environment materially changes. Humans must be part of the evaluation process. An experienced analyst should assess the agent’s work and introduce known threats to see whether the agent identifies them correctly. These evaluations can determine whether an agent is performing effectively or should be removed from service.

As we think about how to assign, manage and evaluate work to drive efficiency and contribute to bottom-line growth, we have to recognize the obvious: We can’t hold AI agents accountable the same way we hold employees accountable. People respond to incentives and consequences. They care about their careers, feeling valued and contributing to something meaningful. Agents don’t have motivations. If they aren’t performing to the required standard, people need to reconfigure them, retrain them or turn them off.

Organizations will likely improve how they manage agents with time and more experience. I use AI, and I’m excited about what these technologies can accomplish. But I also know from experience how quickly an agent can do something unexpected. We wouldn’t let someone handle critical work indefinitely without regularly checking their work. It’s reasonable to expect at least that much scrutiny of an AI agent.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?