Managing AI Agents at Scale: The Team Skills That Actually Matter

Running one AI agent is a tooling question. Running forty is a management question, and most of the management answers are not yet in place.

The shift is happening faster than the readiness. KPMG's Q2 2026 AI Pulse, surveying 204 US leaders at organisations with revenue over $1 billion, found agent deployment has levelled off — "on par with last quarter (53% compared to 55%)" — while something else doubled: "the percentage of organizations orchestrating multiple AI agents across workflows doubled from 9% to 18%."

Deployment plateaued. Orchestration doubled. The same organisations are pointing more agents at more workflows, and that is a different problem from the one they solved last year.

How Many Organisations Are Actually Running Agents at Scale?

Almost none, which is worth saying plainly against the volume of vendor claims.

Two things stand out. The gap between piloting and production is enormous, and trust fell as experience accumulated. Capgemini's own summary is direct: "Only 27% of organizations express trust in fully autonomous AI agents, from 43% 12 months ago."

There is also a shadow problem underneath the official numbers. The MIT NANDA project's State of AI in Business 2025 found that "while only 40% of companies say they purchased an official LLM subscription, workers from over 90% of the companies we surveyed reported regular use of personal AI tools for work tasks." (That report labels itself preliminary findings and is not peer-reviewed, and it is not hosted on an MIT domain — treat it as directional.) If sanctioned AI is already outnumbered by unsanctioned AI, sanctioned agents will be too.

What Changes When You Go From One Agent to Many?

Three things that worked fine at single-agent scale stop working.

Ownership goes ambiguous. With one agent, everybody knows whose it is. With forty, "who approved this agent's access to the customer database?" becomes a question nobody can answer quickly, which means nobody can answer it during an incident either.

Permissions accumulate and never contract. Each agent gets access granted at setup. Almost nothing revokes it. Individually reasonable grants compound into an aggregate reach nobody scoped.

Review stops being real. This is the quiet one. A reviewer checking three agent outputs a day is reviewing. A reviewer nominally accountable for three hundred is approving. The process looks identical on the org chart and is worthless in the second case.

NIST has formalised the distinction. Its COSAiS project, building SP 800-53 control overlays for AI systems, lists "Using AI Agent Systems (AI Agents) – Single Agent" and "Using AI Agent Systems (AI Agents) – Multi-Agent" as separate use cases with separate control sets. When a standards body splits a category, it is because the controls genuinely differ.

What Skills Does a Team Need to Manage AI Agents at Scale?

1. Ownership assignment

The clearest published statement of this comes from the UK's National Cyber Security Centre, in guidance co-authored with the cyber agencies of the US, Canada, Australia, and New Zealand. Their agentic AI guidance, published 15 May 2026, sets a five-part accountability list:

You should be clear about who owns an agentic system, who approves its access, who monitors its behaviour, who reviews incidents, and ultimately who can stop it.

And, critically, on timing and authority: "These responsibilities should be defined before the agent is connected to real systems or data and, crucially, responsible individuals should be empowered and incentivised to intervene if necessary."

That last clause is the one teams skip. Naming a person who is not empowered to stop the agent, or who is measured on throughput, produces a name in a document and no control in reality.

The NCSC also gives the cleanest deployment test available anywhere:

2. Permission discipline

The NCSC's practical steps read as a syllabus for whoever grants agent access:

• "apply least privilege — give agents only the minimum access they need, for the shortest time required"
• "avoid long-lived credentials — use temporary credentials where possible and revoke elevated access when tasks are complete"
• limit scope, use secure defaults, understand dependencies, monitor behaviour, threat-model the deployment, and plan for incidents

This maps onto the failure mode OWASP calls Excessive Agency (LLM06 in the Top 10 for LLM Applications 2025): "the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM, regardless of what is causing the LLM to malfunction." Its three root causes give a team its shared vocabulary — excessive functionality, excessive permissions, excessive autonomy — and OWASP's mitigation list is directly operational, including "Minimize extension permissions", "Require user approval" for high-impact actions, and "Complete mediation: Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not."

That last one deserves emphasis. Asking the model to enforce its own permissions is not a control.

The Cloud Security Alliance argues the underlying tooling is not ready either. Its Agentic AI Identity and Access Management paper, released August 2025, states that "traditional identity and access management (IAM) protocols, designed for static applications and human users, can't keep up", and proposes a framework accounting for "autonomy, ephemerality, and delegation patterns of AI agents". If your agent fleet is authenticating like a set of service accounts from 2015, that is a known gap rather than an oversight.

3. Oversight design

Here is the uncomfortable finding a manager needs to know before relying on "a human in the loop".

Ben Green surveyed 41 policies requiring human oversight of government algorithms and concluded, in The Flaws of Policies Requiring Human Oversight of Government Algorithms:

First, evidence suggests that people are unable to perform the desired oversight functions. Second, as a result of the first flaw, human oversight policies legitimize government uses of faulty and controversial algorithms without addressing the fundamental issues with these tools. Thus, rather than protect against the potential harms of algorithmic decision-making in government, human oversight policies provide a false sense of security.

The mechanism behind that is automation bias, and it is well quantified. A systematic review in JAMIA, screening 13,821 papers down to 74 studies, found that when a decision-support system gave wrong advice it "increased the risk of an incorrect decision being made by 26%", and that "in 6% of cases, clinicians over-rode their own correct decisions in favor of erroneous advice."

The conclusion is not to abandon oversight. It is that oversight is a design problem, not a checkbox. The EU AI Act encodes what a functioning version looks like: Article 26(2) requires deployers to "assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support", and Article 14(4) specifies that the overseer must be able to understand the system's limits, "remain aware of the possible tendency of automatically relying or over-relying on the output... (automation bias)", override the output, and halt the system safely.

Competence, authority, and support. Most oversight rotas supply none of the three.

4. Measurement

Completion rates tell you the agents ran. They tell you nothing about whether anyone is genuinely supervising them.

NIST's AI RMF Playbook names the right metric under MEASURE 2.8: "Measure and document human oversight of AI systems... Maintain statistics about downstream actions by end users and operators such as system overrides."

Override rate is the single most diagnostic number a team running agents can track, and it is diagnostic in both directions:

Two standards back the logging that makes this possible. NIST's MANAGE 4.1 requires "post-deployment AI system monitoring plans... including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management." And ETSI EN 304 223, the European Standard on baseline AI security, makes it normative: "System Operators shall log system and user actions to support security compliance, incident investigations, and vulnerability remediation."

Why Is This a Training Problem and Not Just a Tooling Problem?

Because every control above depends on a person recognising something.

Least privilege depends on someone noticing an over-broad grant. Override statistics depend on reviewers who actually override. Incident response depends on somebody identifying an agent as the cause. None of that is automatic, and none of it is intuitive to a team that has only ever used AI as a chat window.

The standards say so directly. ETSI EN 304 223 requires that "Organizations' cyber security training programme shall include AI security content" and that "AI security training shall be tailored to the specific roles and responsibilities of staff members." NIST's AI RMF subcategory GOVERN 2.2 requires that "The organization's personnel and partners receive AI risk management training to enable them to perform their duties and responsibilities", and its Playbook insists the training differ between technical roles and oversight roles.

Capgemini's data suggests the knowledge is not there yet: "only half claim sufficient knowledge of AI agent capabilities", and "only one-third clearly comprehend where AI agents should be preferred to Gen AI, AI/ML, or process automation." A team that cannot say when an agent is the right tool cannot govern a fleet of them.

There is a workforce dimension too. Capgemini found "61% of organizations report rising employee anxiety about the impact of AI agents on their employment prospects... Despite this, fewer organizations are prioritizing reskilling or workforce restructuring." Asking anxious, untrained people to supervise the thing they think is replacing them is not an oversight strategy.

A Starting Checklist

1. Inventory the agents, including the ones nobody registered. Personal-tool use outruns official deployment.
2. Name an owner, an approver, a monitor, an incident reviewer, and a stopper for each agent — before it touches production data.
3. Check the stopper is empowered and incentivised to actually stop it.
4. Audit permissions per agent against least privilege, and set expiry on every grant.
5. Replace long-lived credentials with temporary ones, and revoke elevated access when tasks complete.
6. Move authorization into the downstream systems, never leaving the model to decide what it is allowed to do.
7. Size review to reviewer capacity. If the volume exceeds what a person can genuinely check, the oversight is decorative.
8. Log system and user actions, and report override rates alongside completion rates.
9. Train by role, splitting technical operators from oversight roles as NIST's Playbook requires.
10. Apply the NCSC test to every agent before deployment: can you understand it, monitor it, and contain it?

Where to Go Next

• The individual skills that sit underneath all of this: what training is required to use agentic AI effectively.
• The hub guide for this cluster: agentic AI skills.
• The foundation everything else rests on: what is AI fluency.
• The EU compliance floor: EU AI Act Article 4.
• Building the capability across a whole organisation: kju for enterprise.

The organisations that manage agents well will not be the ones with the best orchestration platform. They will be the ones where somebody can say, in under a minute, who owns each agent and who can switch it off.