AI Gateway: Governed LLM Proxy for Enterprise AI

⏱ 20m

What is AI Gateway?

MuleSoft AI Gateway is a governed proxy layer for Large Language Model (LLM) traffic. Built on the same Omni Gateway infrastructure that powers MuleSoft’s API management platform, it gives platform teams a single control plane for every AI request flowing through the enterprise — regardless of which LLM provider or client application originates that request.

Client Applications (Finance Agent, Sales Bot, Dev Copilot...)
                │
                │  OpenAI-compatible API
                ▼
         MuleSoft AI Gateway  (Omni Gateway)
         ┌────────────────────────────────┐
         │  Token Rate Limiting           │
         │  Prompt Injection Protection   │
         │  Model-Based Routing           │
         │  Usage Telemetry               │
         └────────────────────────────────┘
                │                 │
                ▼                 ▼
          Google Gemini        OpenAI GPT-4o

AI Gateway acts as a centralized governance layer, bridging application developers and LLM providers. Because it exposes a single OpenAI-compatible endpoint, developers can use any OpenAI client library — the gateway handles all provider-specific translation transparently.

Why it matters

Without a governance layer, LLM usage inside an enterprise is ungoverned:

Teams embed API keys directly in application code, creating a security liability
There is no visibility into which team is spending how much on which model
Switching providers requires code changes in every application
No consistent place to enforce safety policies (prompt injection, content filtering)

With AI Gateway:

Developers never see LLM provider keys — they authenticate with a Client ID and Secret issued by your platform team
Token consumption is tracked per business group and client application — billing allocation is automatic
Model-based routing lets you send specific models to specific providers, with automatic fallback
Governance policies (rate limiting, prompt injection protection) apply at the gateway layer — one policy covers every app

What you’ll build

By the end of this workshop you will have a fully operational AI Gateway that:

Exposes a single OpenAI-compatible endpoint via your Omni Gateway consumer URL (e.g. https://<gateway-host>/llm-proxy/v1)
Routes requests to Google Gemini (Gemini 2.5 Flash) and OpenAI (GPT-4o) based on the model field in the request body
Falls back to Gemini 2.5 Flash when a primary route is unavailable
Enforces token rate limiting: 100,000 total tokens/min, 50,000 prompt tokens/min per client
Protects against prompt injection attacks
Tracks token usage by business group and client application in the Usage Dashboard

Prerequisites

Before starting, make sure you have:

A MuleSoft Platform account with API Manager access
A Managed Omni Gateway with status Connected — or the ability to deploy one
Workshop credentials from the Credentials page
Postman installed

Note: AI Gateway is an add-on capability — confirm your MuleSoft Platform subscription includes AI Gateway / AI Management before starting.

Architecture overview

Layer	Component	Responsibility
Client	Any OpenAI-compatible app	Sends standard OpenAI requests
Inbound	Omni Gateway listener	Receives and authenticates requests
Gateway	AI Gateway policies	Rate limiting, prompt injection, telemetry
Routing	Model-based routing engine	Selects outbound route by `model` field
Outbound	LLM provider connections	Translates and forwards to Gemini or OpenAI

Estimated time

Section	Duration
Prerequisites	2 min
Create LLM Proxy	7 min
Verify Policies & Get Credentials	4 min
Test the Gateway	3 min
Monitor Token Consumption	3 min
Summary	1 min
Total	20 min

Ready? Start with Prerequisites →