Skip to main content

AI Gateway

Stacklok Enterprise

The AI Gateway is a component of Stacklok Enterprise.

Learn more about Stacklok Enterprise.

The AI Gateway is a self-hosted control point between AI clients and large language model (LLM) providers. It authenticates each caller, routes the request to the provider that serves the model, and applies identity, budget, data protection, and audit policies on the way through.

Clients call each model through its provider's own API, such as OpenAI Chat Completions, the OpenAI Responses API, Anthropic Messages, or the Gemini API. The gateway passes the request through in that format.

The AI Gateway governs model access. The Connector Gateway governs MCP tool access.

What you can do​

  • Set cost budgets for users and groups.
  • Limit which models each user or group can call, and show each caller only the models they can use.
  • Screen requests for prompt injection, payment card data, and personally identifiable information (PII).
  • Record structured audit events and forward them to a security information and event management (SIEM) system.
  • Authenticate callers with your identity provider or with virtual API keys, while provider credentials stay in the gateway.
  • Export traces and metrics to an OpenTelemetry collector, attributed to the users and groups that sent the traffic.

Where to start​

Deploy the gateway​

The platform team installs the AI Gateway with the platform chart and connects it to the Enterprise Manager. If the gateway isn't running yet, start with Configure the AI Gateway. The rest of this section assumes an installed gateway.

Administer an installed gateway​

To make a model available to your users:

  1. Connect a model provider, confirm the model is priced, and send a test request.
  2. Set budgets for the users and groups that will call it.
  3. If model access is on, grant the model to those users or groups.

Then add prompt injection screening, PCI/PII controls, audit log forwarding, and telemetry as your policies require.

Connect a client​

Users find their deployment's endpoints and per-client setup guides in the console. To prepare access and point users there, see Roll out gateway clients. For the endpoints and credential headers clients use, see the endpoints reference.

How configuration works​

You configure the AI Gateway through Helm values under the ai-gateway-v2 key of the platform chart. A configuration or credential change takes effect through a pod rollout; see How configuration changes apply. Pods can become ready with a wrong budgets or provider setting, so send an authenticated test request after each rollout.

Examples on these pages show values relative to the platform chart's values file and use <NAMESPACE> for the namespace the platform is installed into, <RELEASE_NAME> for the Helm release name, and <CHART> for the platform chart reference you install from. In prose, value paths are relative to the ai-gateway-v2 key, except those that start with global. or another chart's key, such as enterprise-manager..

Legacy gateway​

The legacy AI Gateway is still documented for platforms that run it. A platform runs one of the two, except during a migration window. To check which one you run, see Identify your deployment.

See the legacy AI Gateway documentation. To move to the AI Gateway, see Migrate from the legacy AI Gateway.

Contents​