AI Still Has to Run Somewhere

As agents take on business work, choosing a model is only one part of choosing how the business will operate.

By Randy J Rouse

Watch the flagship video: AI Still Has to Run Somewhere | Who Owns the Result?

Video series: Where Does AI Run? · One Answer, Several Meters · AI on Your Data Platform · Local Does Not Mean Self-Operated · Kubernetes Is an Option · What Must Remain Close? · Someone Owns the Result

In conversations with customers, I spend a lot of time talking about data, security, AI and the platforms that connect them. I also see customers moving substantial data and analytics work into platforms such as Databricks and Snowflake.

Those conversations bring me back to a practical question: how do we put AI to work so it produces value for the business?

Choosing a model is part of it. So are trusted data, agents and the identities and permissions that govern what they can do. But there is another part of that conversation I haven't explored fully in this series.

Where does the work actually run, and who is responsible for operating it?

That question reaches beyond buying GPUs or choosing a public cloud. It also reaches into the data platforms companies already use. Some supply context to an agent. Some can run models, tools or the agent itself. Those are different capabilities, with different operating and consumption costs.

In Who Is Watching the Agents?, I explored the evidence we need when machines begin taking action. The next question is: how do we make that work reliable, affordable and repeatable?

AI still has to run somewhere.

Running AI also creates a recurring bill. Gartner's August 2026 forecast puts inference spending within AI-optimized cloud infrastructure services at $23.3 billion for 2026, ahead of training at $19 billion. That is a forecast for one infrastructure market, not a measure of all AI spending or proof of hybrid adoption. But it gives leaders a reason to focus on the cost of operating AI after the initial build. Gartner forecast

Before we choose a location, define the work

“We're putting our AI on Azure.”

“We're using OpenAI.”

“We're building on Databricks.”

“We're buying GPUs.”

Those statements describe different decisions. An organization can make several of them at once and still have unanswered questions about how its AI operates.

Imagine an agent helping resolve an invoice exception. It retrieves purchasing policy, compares the invoice with an order, asks for approval and updates the financial system.

There are several places in that workflow where something must run:

The agent's application logic and task state.
The model that interprets information and helps choose the next step.
The retrieval services and business data supplying context.
The systems that approve and execute the transaction.

They need secure connections. They do not necessarily need the same address.

The agent application could run near the enterprise's systems while calling a remotely hosted model. Another workload might need a model running locally. A data platform might supply both governed context and managed agent execution. Buying infrastructure for one component doesn't tell us where the others should run.

Before we ask where AI belongs, we need to ask which part of AI we mean.

Customers are solving different operating problems

Most of these accounts are published by suppliers. They illustrate customer operating choices and reported results, rather than independently verified performance or market adoption.

At Emirates Global Aluminium, response time helps determine where AI runs. Its digital manufacturing platform uses a hybrid Azure environment: public cloud plus Azure Local on site. Microsoft's customer account describes local execution for applications needing rapid responses, alongside cloud capacity. EGA's chief digital officer reports that visual inspection expanded from 1.5–2.5% of carbon anodes to all of them. This example involves industrial AI. Thomson Reuters shows what the question looks like when agents take on operational work. EGA's account, Microsoft's customer story

In an engineering account co-written by its employees and published by AWS, the company describes an agentic platform engineering hub using Amazon Bedrock AgentCore. The team wanted to reduce repetitive coordination and avoid building all the agent infrastructure itself. The system includes specialized agents and human validation for sensitive operations. Thomson Reuters and AWS account

That is a strong reason to consider managed services: engineering time has value, too.

For Indonesia's General Elections Commission, KPU, the problem was legal research. Nutanix's customer account describes fragmented legal information and laborious research. The solution uses Llama 3 with Nutanix Enterprise AI and Nutanix Kubernetes Platform, alongside infrastructure and data services. The assistant helps the internal legal team work with its document collection. KPU customer story

And at DenizBank and its technology subsidiary Intertech, Red Hat describes a need to standardize the environments used by roughly 120 data scientists. The work involved self-service, pipelines and model serving. The team needed more than compute. It needed an environment people could use consistently. Red Hat's DenizBank account

Together, these different workloads give leaders better questions than “Which AI platform should we buy?”

What constraint are we trying to remove? What responsibility are we willing to take on? Which parts should someone else operate?

AI can run where the data already lives

That brings me back to the data-stack question we've been exploring. If a business is already investing in governed data, it makes sense to ask how much AI work that platform can support.

Databricks documents a managed agent runtime that hosts agents on Databricks Apps, alongside model access, tools, state and governance. That goes beyond storing the data an agent retrieves. Databricks documentation

Snowflake's Cortex Agents combine analytical and search tools with custom tools that can call backend systems. Snowpark Container Services separately provides managed execution for containerized applications. Both let customers run software within Snowflake, with different responsibilities and limits. Cortex Agents, Snowpark Container Services

Microsoft Fabric's data agent has a more specific role: governed, read-only conversational access to data. Another agent can use that capability within a larger workflow, but answering a business question is different from executing the resulting transaction. Fabric documentation

Google BigQuery brings AI into analytics and documents connections to agents through MCP and Agent Development Kit tools. Google's broader architecture offers separate runtime choices, including managed agent hosting, Cloud Run and Google Kubernetes Engine. Choosing BigQuery does not, by itself, settle where the whole application runs. BigQuery AI, Google's architecture guidance

These are not four interchangeable promises to run everything. A customer-facing agent still needs the right response time, access controls, transaction boundaries and recovery behavior. The platform has to meet those requirements.

Nor does a managed platform make consumption disappear. Snowflake documents separate orchestration and tool costs. Fabric meters data-agent token use through capacity units, with generated queries charged separately to their query engines. One useful answer can activate several meters. Fabric consumption

Bringing AI to the data can simplify part of the architecture. It does not eliminate the operating or economic questions.

Where it runs and who operates it are separate decisions

Nutanix is connecting its VM and Kubernetes platforms with inference and agent governance. Red Hat positions OpenShift AI around operating AI across hybrid environments. Broadcom's VMware Private AI Cloud announcement brings inference, agent applications and traditional workloads into its private-cloud strategy. The announcements mix available capabilities with features still rolling out. Nutanix, Red Hat, Broadcom

The public-cloud providers offer local options, too. EGA's deployment is one example. AWS AI Factories offers another operating model: customers supply data-center space and power, while AWS deploys and manages dedicated AI infrastructure there. The distinction matters: on-premises location does not necessarily mean self-operated infrastructure. AWS announcement

Specialist AI clouds, often called neoclouds, add another option. CoreWeave, for example, offers serverless inference for a curated set of open models alongside its broader compute platform. A specialist provider may operate the model-serving layer while the customer's application and data remain elsewhere. Whether that arrangement works depends on the actual service, connections and responsibilities. CoreWeave's inference offering

A service provider may also operate private infrastructure for a customer. The agreement needs to specify where responsibility for infrastructure ends and responsibility for models, applications and business outcomes begins. Leaders should ask for a named operator and a tested escalation path across those boundaries.

Location, ownership and operation are separate choices.

A company can retain equipment without operating every layer itself. It can rent capacity without buying a complete AI application. It can use a managed model while retaining control of the workflow that invokes it.

The useful choice is how much responsibility to retain, where to retain it, and what to expect from the provider.

Why Kubernetes belongs in this conversation

Containers package software and its dependencies. Kubernetes coordinates those containers across machines, including agent applications, tools and model-serving services. Its documented GPU scheduling setup requires vendor drivers and device plugins. An agent calling a hosted model need not have local GPUs. Kubernetes GPU scheduling

VMs and containers can coexist: Kubernetes nodes can be virtual or physical machines. An agent can reach an existing application through an API without creating a reason to rewrite it. Kubernetes documentation

Google's architecture guidance distinguishes a managed agent runtime, Cloud Run and Google Kubernetes Engine. It notes that GKE's greater infrastructure control requires more expertise and operating effort than a serverless approach. The choice should fit what the team can maintain. Google's architecture guidance

Kubernetes is an option, not a requirement to operate your own cluster. Providers can take on much of that work, but upgrades, security, capacity and recovery still need an owner.

Moving a container image doesn't automatically move its data, permissions, managed services or recovery procedures.

A portable workload is only useful if someone can operate it on the other side.

Hybrid is a hypothesis to test

My hypothesis is that enterprise AI will often take a hybrid shape because enterprises have different requirements within the same business process.

That does not mean every organization needs multiple clouds or private inference. Nor does it mean AI must retrace a neat journey from on-premises computing to public cloud and back again. Cloud never followed one universal sequence.

The customer examples support choosing a location for a reason. They don't establish that autonomous hybrid agents have become the default enterprise architecture.

Many enterprises would rather buy the capability. Menlo Ventures' 2025 report, based on a survey of 495 U.S. enterprise AI decision-makers, reports that 76% of AI use cases are purchased rather than built internally. This is investor-produced research, not a neutral census, and buying an application does not tell us where its infrastructure runs. It is a useful reminder to ask whether building is necessary at all. Menlo research and methodology

And hybrid adds work.

Every boundary can introduce network dependencies, identity translation, additional monitoring and another team or supplier involved in recovery. A design that improves local control may make the overall workflow harder to diagnose. A cheaper inference endpoint may become expensive after integration, idle capacity and engineering effort are included.

If a managed platform meets the business requirements, adding another environment needs a reason. Flexibility earns its cost when it solves a problem or preserves an option the business expects to need.

The boardroom questions

I would start with a workload, then ask five questions.

What must remain close? Identify the data, people, machinery and systems the workflow depends on. Close can mean remaining within an approved legal or security boundary, as well as physical proximity. Personally identifiable information, financial records and protected health information can bring specific privacy, access and handling obligations. Determine which regulatory, contractual or data-residency requirements apply, then trace what crosses a boundary, including prompts, retrieved passages, memory and logs.

What does the business need when something fails? Define response time, availability, acceptable interruption and recovery. Test what happens when the model endpoint or the connection to the system of record is unavailable.

What is the cost of a useful result? Include inference, retries, data movement, licenses, capacity, support and engineering time. Compare options at the required quality and demand level. Token price alone won't settle the decision.

Who operates the whole workflow? Identify the owner who can coordinate infrastructure, model services, data access and the business application. Ask the provider where its responsibility ends. A healthy GPU service doesn't prove an invoice was processed correctly.

What would changing our mind require? Test the dependency on models, APIs, data formats, permissions and people. An exit plan should describe the work of moving, not simply say that movement is possible.

These questions connect directly to the narrative I've been exploring.

Value lives in the capability and outcome. Identity determines who may act. Trusted data supplies context. Observability provides evidence. Infrastructure and operations make the work repeatable.

The interface may become simpler while the responsibility underneath it grows.

Leaders don't need one answer for every AI workload. They need to explain the choice for each important one, and know when to revisit it.

AI still has to run somewhere. Someone still has to own the result.