What should an AI gateway control? Enterprise requirements and implementation paths

One AI product can survive with an API key in the model SDK and a usage logger beside it. The operating model changes as the product count grows. The same cross-cutting code appears in many repositories, every team stores keys differently, and no two products agree on cost tags or prompt and response logging. Each new security rule triggers another round of application deployments.

An AI gateway becomes necessary not on the day a company adopts a model, but when it must manage model calls as one operating domain. A proxy in front of provider APIs is only the start. Identity, policy, cost, routing, and audit evidence must pass through the same enforcement path before the gateway becomes an enterprise control plane.

Why direct calls break down at enterprise scale

Direct calls are efficient for experiments. In enterprise operations, every product recreates the same cross-cutting capabilities.

Centralizing keys does not finish the key-management problem. If an application can call a provider without the gateway, attackers and insiders can bypass every central policy. IAM, network egress, and provider organization policy must close the direct route together. The short-lived credentials and model and region restrictions in our Bedrock credential exposure and LLMjacking guide belong at this boundary.

At customer A, keys grew faster than projects

Customer A uses the public LLM APIs of OpenAI, Google Gemini, and Anthropic across internal AX projects. Each project initially obtained the provider keys it needed. As the project portfolio grew, the operating model reached its limit: create another key, record its owner, revoke keys for closed projects, and divide the provider bill. The organization could no longer manage keys at the speed teams could call models.

The customer is building a LiteLLM-based AI gateway to bring those AX projects onto one model-access path. The objective is larger than replacing several provider SDKs with one interface. Provider keys must leave applications, each project needs its own virtual key or workload identity, and the same control plane must operate model allowlists, usage limits, and logging fields.

Moving existing keys behind a gateway does not complete the integration. Inventory current keys and callers first, migrate applications to virtual keys, then reject direct calls through network and provider policy. Without that final step, the old key path remains beside the new gateway.

Write enterprise requirements as six contracts

Copying a product feature list into an RFP gives every candidate a similar page of checkmarks. A stronger contract defines what happens when a control fails and what evidence must remain.

The NIST Generative AI Profile recommends inventorying third parties that can access organizational content and continually assessing approved providers and both commercial and open-source components.¹ A gateway can enforce and record that supply chain. It cannot create the accountability structure that NIST expects.

The OWASP 2025 LLM risks show the same boundary. Gateway rate and token limits reduce Unbounded Consumption, while PII inspection and logging policy reduce Sensitive Information Disclosure. Prompt Injection, Improper Output Handling, and Excessive Agency also require application-level data permissions and tool design.²

A cost limit is a real-time brake, not the invoice

Gateway cost is usually an estimate derived from the model name, input and output tokens, and a price table. The provider bill also reflects provisioned throughput, cached tokens, batch discounts, region, and negotiated rates. Use the real-time limit as a circuit breaker against overspend, then reconcile it with provider billing data at financial close.

In the linked LiteLLM test, a key with a $0.5 budget was blocked after usage reached $0.607. The limit evaluated accumulated spend when the request arrived, so in-flight work produced the overage.³ Claude Apps Gateway likewise states that Spend Limits are list-price estimates rather than invoices. The requirement is not simply “has a budget feature.” It is the acceptable overage, refresh interval, concurrent-request behavior, and variance from the provider bill.⁴

Define the attribution grain before implementation. User-only attribution cannot explain background jobs. API-key-only attribution cannot separate many functions sharing one key. Showback and chargeback need at least organization, department, project, environment, application, user or workload, model, and logical request.

Bedrock credential theft is a gateway failure test

The incident in our Bedrock credential exposure and LLMjacking guide began with stolen AWS credentials that had InvokeModel permission. The attacker created no new resources and shifted model-API spend to the victim account. An AI gateway does not erase that attack automatically. If stolen credentials can call Bedrock directly, the traffic bypasses gateway authentication, budgets, and logs.

Reducing the blast radius starts with changing the access path.

The gateway runtime's IAM role becomes a high-value target. Limit that role to required models, regions, and actions, and separate credentials for development, staging, and production. Design for a gateway compromise by separating secret-read, policy-change, and invocation authority. Before release, test whether a Bedrock call that bypasses the gateway is rejected and whether revoking one virtual key leaves other projects running.

Judge observability by traceability, not captured content volume

When every AI product implements observability and billing independently, even “request count” loses a common meaning. One product counts user requests; another counts every retry and fallback. A gateway should separate the user-facing logical request from each actual provider attempt. That distinction puts success rate, cost, and fallback ratio into one interpretable flow.

OpenTelemetry's GenAI semantic conventions define shared attributes such as operation, provider, requested and response model, and input and output tokens. Prompts, responses, and tool arguments are sensitive and potentially large, so content capture is opt-in rather than the default.⁵ A safer operating baseline is metrics-only, with content enabled only for approved use cases and a limited retention window.

A semantic cache is both a cost feature and a data-isolation feature. If tenant, permission, model, system prompt, or policy version is absent from the cache key, one user can receive another tenant's answer or an answer produced under an old policy. Define exclusions first for sensitive questions, time-dependent answers, and tool-execution results.

A Claude-specific gateway and an enterprise AI gateway have different scopes

Claude Apps Gateway is not a bad product; it has a clearly bounded job. It fits organizations deploying Claude Code and Claude Desktop internally while enforcing OIDC sign-in, managed settings, user and group spend caps, and OTLP telemetry. With Bedrock upstream, provider credentials do not need to reside on developer machines.⁴

That scope is too narrow to define an enterprise AI gateway. Its center is Claude client policy, not a general service plane for OpenAI-compatible APIs, multiple model providers, centrally operated prompts and guardrails, and non-interactive workloads. Developers with separate AWS credentials may also call Bedrock directly. Deploying the gateway alone does not close the bypass.

An MCP gateway is another boundary. An AI gateway governs model-inference calls. An MCP gateway handles tool discovery, tool authorization, and protocol routing. Agentic systems may need both. Our MCP and AgentCore state and authorization boundary guide explains why treating them as one product obscures the design.

Compare four adoption paths with the same questions

AWS's Multi-Provider Generative AI Gateway Guidance places CloudFront, WAF, and ALB in front of LiteLLM, then adds RDS, Redis, Secrets Manager, and S3. It exposes Bedrock and external providers through an OpenAI-style interface.⁶ The important pattern is not the AWS resource list. It is the separation of ingress, stateless data plane, policy store, secret store, and evidence sink.

What to verify when evaluating Portkey

Portkey cannot be reduced to “SaaS weakens data sovereignty” or “the gateway is open source, therefore everything is self-hosted.” Its official enterprise documentation separates Portkey-managed SaaS, hybrid deployment with the data plane in the customer's VPC, and air-gapped deployment of all components.⁷

In hybrid mode, the gateway that processes prompts and responses runs in the customer's environment. The log store can run there or in Portkey Cloud. Portkey operates the control plane containing the management UI and configuration, while the data plane periodically synchronizes policies, keys, and provider configuration. A strict no-egress requirement points to air-gapped deployment rather than hybrid. Provider secrets can also remain in AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault while the control plane stores only references.⁸

Evaluate Portkey against the other candidates by checking four points.

• Does the company have several model providers and AI products and need one UI and API for team-level policy and cost?
• Does rapid access to SSO, RBAC, audit, versioned prompts and configuration, and guardrail operations matter more than owning the gateway engine?
• Can security requirements be matched to a commercially supported SaaS, hybrid, or air-gapped deployment?
• Will a named owner operate the gateway as a shared platform rather than another product?

A self-managed LiteLLM stack is more natural when the company must own the control plane, deeply modify the policy engine, and has a platform team prepared to run databases, caches, telemetry, and upgrades. If the actual scope is limited to internal Claude Code deployment, Claude Apps Gateway is the smaller and more precise answer.

If Portkey remains a candidate, its proof of concept should leave six points in the contract and test record rather than merely demonstrate its screens.

1. Storage location and egress for prompts, responses, traces, aggregate metrics, configuration, and secret references in each deployment mode.
2. Per-feature fail-open or fail-closed behavior when the control plane, cache, or log store is unavailable, plus the validity period of the last known good configuration.
3. Licensing boundaries for SSO, organization and workspace RBAC, audit logs, budgets, and guardrails.
4. Provider-price updates, custom pricing, billing reconciliation, and budget behavior for in-flight requests.
5. Export formats for configuration, prompts, and logs; API compatibility and termination procedure for a move to another gateway.
6. Image provenance, CVE response, upgrade and rollback, support hours, and SLA.

Roll out in the order of observe, enforce, and make default

Start by inventorying every provider call and key. Record owner, environment, data classification, provider, model, authentication method, monthly usage, and existing logging and guardrails rather than product name alone. Build the NIST-recommended list of approved providers and third parties at the same time.

Next, insert the gateway in observe mode. Compare it with direct-call results while measuring user and workload attribution, logical requests and provider attempts, cost-estimation variance, and added latency. First prove that operators can answer production questions without content logging.

Move identity and secrets before stricter policy. Remove provider root keys from applications, then apply short-lived access and tenant policy. Run budgets and guardrails in alert-only mode to measure false positives and overage before blocking. Do not enable a fallback merely because it returns an answer. The alternate model must pass a quality regression test.

Finally, close direct provider access in network and IAM policy and make the gateway the default path. Pass the following failure tests.

This sequence applies the observability, cost-limit, and rollback evidence in our six gates from AI PoC to production to a shared platform.

The goal is one accountability system, not one pipe

The enterprise AI gateway's goal is not a polished dashboard containing every AI request. The organization must explain who used which model under which policy, how much it cost, what was blocked on failure, and where traffic moved.

A self-managed LiteLLM build, Portkey Enterprise, and Claude Apps Gateway are different answers. Data sovereignty, control-plane ownership, provider diversity, operating capability, and deployment speed change the choice. If the company never tests direct-call bypass, billing reconciliation, log privacy, cache isolation, and fail-open or fail-closed behavior, centralization becomes a new single point of failure rather than control.

We design AI gateway identity, network, and high-availability architecture through Cloud & Infrastructure, and model policy, guardrails, evaluation, and departmental operating models through AX Consulting.

References