Why Your AI Center Of Excellence Should Get Smaller As AI Gets Bigger

Igor Rikalo is President at o9 Solutions.

gettyAI activity has become ubiquitous within large businesses as they deploy co-pilots, agents, forecasting models and dozens of pilots across many functions.

McKinsey’s 2026 State of the AI survey found that 88% of organizations regularly use AI in at least one business function and 44% now report scaling it across the enterprise, up from 38% a year earlier. However, the financial results have not kept pace. The survey finds that only 6% qualify as high performers, crediting AI with at least 5% of operating profit, a share that has not moved since 2025.​

Large companies aren’t lacking effective ideas for using AI. Rather, they lack a reliable way to transform those ideas into business value, and this gap is why the AI center of excellence (AI CoE) remains necessary. However, its purpose needs to change. ​

Initially, AI CoEs acted as a pool for scarce technical talent because it was difficult and expensive to find workers with the necessary skills. Today, these constraints have mostly disappeared, but what remains scarce is even harder to buy: knowing which decisions move the business, earning enough trust to let a system run and measuring what changed.​

A modern AI CoE should own this record and create the standards around it, whereas business teams own the work and results. When the AI CoE is tasked with assessing every use case, tool choice and risk review passes, the enterprise slows down.

When starting new projects, organizations usually begin by inventorying possible AI use cases. However, it’s more productive to start with the decision itself. For example, consider the difference between “build a supply-planning agent” and “reduce the risk of running out of stock while holding inventory and expediting costs within agreed limits.” The first option describes a technology project, whereas the second identifies the decision, the trade-off, the owner and the measure of success.

• Where does performance stand today?

• What evidence would justify a decision to scale, redesign or stop?

At one global manufacturer I work with, the AI CoE scores every proposal based on strategic value, data readiness, repeatability, effort and risk. When teams submit proposals that fall short, they receive feedback and return with stronger ideas. Then, the AI CoE manages these initiatives as a portfolio, tracking how long ideas take to reach production, how many reach scaled adoption and how benefits compare with the original business case.

An organization’s AI CoE should own the capabilities (architecture standards, model evaluation, data access policies, security, and agent permissions) every team depends on, while business teams own what they build on those foundations because they understand their own trade-offs and determine whether employees adopt a redesigned workflow. However, this split has one common failure. Business teams do not have deep engineering skills, so planning, procurement, finance and service could each build their own connection to the same ERP, and none of the four would pass security review. Instead, the AI CoE should build the connections to core systems, the testing tools, the approved prompt and policy libraries, and the cost dashboards, then let business teams take them. If teams use those components, the AI CoE is building the right things. If they route around them, it is not.​

Deloitte’s 2026 State of AI in the Enterprise report found that 23% of organizations use agents at least moderately today and 74% expect to do so within two years, while only 21% report a mature governance model. This means AI adoption will roughly triple while governance has barely moved.

Earlier AI models produced predictions, whereas today’s AI agents can interpret a goal, select steps, use tools and, in some cases, act on enterprise systems without a person in the loop. But greater authority among agents calls for tighter limits and clearer ownership.

Every enterprise should maintain an agent registry that records each agent’s purpose, business owner, data access, permitted actions, escalation rules and whether the agent is currently live or retired. Retirement deserves particular attention because agents can accumulate quietly and very few organizations ever switch them off. A registry can look like recentralization, but the ownership split holds as the AI CoE owns the format and business teams own the entries.

I’ve previously written about decision memory and inside an AI CoE, it’s an operating requirement. Every significant AI-supported decision should leave behind the situation, the assumptions, the recommendation, the action taken, any human override, the outcome and the cost. Leaders can review decisions and ask if the recommendation improved the outcome, which assumptions proved wrong and where human judgment added value. That record is a key differentiator because your decision history is tied to your products, your customers and your constraints. Your teams learn from it, and so do the agents trained on it.

As AI activity grows, the central team should become smaller relative to the whole, retaining responsibility for standards, portfolio visibility, governance, decision memory and shared capabilities, while execution moves toward the teams that own the decisions.

Three measures show whether that shift is happening: the share of AI work business teams deliver without central staffing, the share of new deployments that reuse existing components, agents or data products and the time from problem statement to production. When the first two rise and the third falls, the operating model is working.

Companies that treat the AI CoE as a control tower will keep producing pilots. Those that treat it as a source of standards, shared capability and institutional memory, while pushing execution and accountability into the business, are likely to turn AI activity into results worth reporting.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?