API compatibility & migration cost
Is it a drop-in replacement for an SDK you already use (e.g. OpenAI Chat Completions), or does it require a new client and request shape?
Resource
LLM gateway buyer's guide
A vendor-neutral framework for choosing an LLM gateway.
Gateway types, the criteria that actually differentiate vendors, the questions to ask before you commit, and a two-week plan to run the evaluation. Privian is one option in the category — this guide is written so it's useful even if you choose someone else.
What is an LLM gateway and what should you evaluate?
An LLM gateway is a layer between your application and one or more model providers that centralises authentication, model routing and observability, and — depending on the vendor — data controls such as credential handling and egress protection. Evaluate it on API compatibility, credential model, data retention defaults, provider coverage, failure behaviour, feature support and deployment model — not on integration speed alone.
Summary
The short version
What is the problem?
Teams calling multiple LLM providers directly end up with duplicated auth code, scattered API keys, no single place to enforce data controls, and no good answer when a security reviewer asks what leaves the company and where.
How is it solved?
Put one gateway in front of every provider call, and choose it against written criteria — compatibility, credentials, retention, egress controls, coverage, and failure modes — rather than by whichever SDK was easiest to wire up first.
When should this approach be used?
When you have more than one provider or team calling a provider, sensitive data in prompts, or an upcoming security review that needs one describable data path.
What are the limitations?
A gateway is infrastructure, not a compliance certification. It shifts where controls live; it does not remove the need to configure them correctly or to keep verifying vendor claims against current documentation.
Why buy one
Why teams end up buying a gateway
Almost nobody sets out to buy a gateway on day one. It becomes necessary once a few pressures accumulate at the same time.
- Multiple providers, each with its own SDK, auth model and quirks
- API keys scattered across services, repos and environment files
- No single point where outbound AI traffic can be observed or controlled
- An enterprise security review asking what data reaches which provider, and nobody having a confident answer
The trigger is rarely a single incident — it's the moment a security questionnaire or a second provider integration makes "ask each engineer" stop being a viable answer.
Taxonomy
Types of LLM gateway
"LLM gateway" and "AI gateway" cover several genuinely different products. Knowing which category you actually need narrows the shortlist fast.
Framework
Gateway categories and who they suit
- 01
Routing / cost-optimisation gateways
Optimise for provider abstraction, failover, caching, load balancing and cost attribution across models. Suits teams whose main pain is provider sprawl and spend visibility, and who don't handle highly sensitive prompt content.
- 02
Observability-first gateways
Centre on tracing, evaluation and logging of every request for debugging and quality monitoring. Suits teams optimising prompt quality and reliability over privacy or cost.
- 03
Privacy / egress-control gateways
Optimise for reducing what leaves the boundary — detecting and masking sensitive values before the provider call. Suits teams whose driver is a security review, regulated data, or customer PII in prompts.
- 04
Self-hosted proxies
Open-source or self-managed software you run on your own infrastructure. Suits teams with data-residency requirements or the operational capacity to run and patch another service themselves.
- 05
DIY in-house wrappers
A thin internal library or service built to solve today's specific need. Suits very small teams or early-stage products where a vendor decision isn't yet worth the overhead — until sprawl catches up.
These categories aren't mutually exclusive in practice — most vendors lean toward one but carry some capability from another. Use the taxonomy to identify your primary driver before comparing feature lists.
Evaluation
What actually differentiates gateways
The criteria below are the ones that tend to matter once a gateway is load-bearing in production — not the ones that show up first in a demo.
Credential model
Does the vendor hold your provider keys (BYOK, encrypted at rest, decrypted per request) or resell tokens under its own account? Each has different billing and trust implications.
Data retention & logging defaults
Are raw prompts and responses persisted by default? For how long? Can logging be scoped or disabled per field?
Egress data controls / PII handling
Does the gateway inspect or transform request content before it reaches the provider, or does it forward whatever the application sends unchanged?
Provider coverage & model addressing
Which providers and models are supported, and how are they addressed — a stable namespaced id, or provider-specific strings you have to remap yourself?
Latency & failure modes
What does the gateway add to request latency, and what happens on a provider outage — retry, failover, or a clean error your application can handle?
Streaming & advanced feature support
Does it support streaming responses, tool/function calling, JSON mode and multimodal input, or only basic non-streaming text completions?
Observability & cost attribution
Can you see usage and cost broken down by team, key or feature without the gateway needing to store prompt content to do it?
Self-host vs managed
Do you run the gateway yourself, accepting the operational burden, or is it a managed service you depend on for uptime?
Deployment region / data residency
Where does the gateway process (and, if applicable, log) requests, and does that satisfy your residency requirements?
Contractual & security-review posture
Does the vendor have a security/trust page, a DPA, and answers ready for a procurement questionnaire, or will you be first to ask?
Pricing model
Is pricing a markup on tokens, a flat platform fee, or usage-based on gateway requests — and does it stay predictable as volume grows?
Framework
How to weigh these if you can only pick a few
- 01
Start from your trigger, not the full list
If your driver is a security review, weight credential model, retention and egress controls heavily. If it's provider sprawl and cost, weight coverage and cost attribution instead.
- 02
Treat feature-support gaps as disqualifying only if you need them now
A missing capability like streaming is a blocker for a real-time chat UI and irrelevant for a batch summarisation job. Match the gap to your actual workload.
- 03
Verify retention and credential claims directly, don't infer them
These are the two areas most often stated vaguely in marketing copy. Ask for the specific behaviour in writing.
- 04
Weight self-host vs managed by your ops capacity, not ideology
Self-hosting is not automatically more secure if nobody on the team can keep it patched; managed is not automatically riskier if the vendor's posture is documented and verifiable.
Vendor questions
Questions to ask every vendor
Framework
A fixed question list, asked identically of every shortlisted vendor
- 01
Do you persist raw prompts and responses by default?
If yes, for how long, and can it be disabled or scoped?
- 02
Who holds the provider API keys?
Vendor-held pooled credentials, or BYOK with keys scoped to our account?
- 03
How are provider keys stored and decrypted?
Encrypted at rest, in-process decryption, or plaintext in a config store?
- 04
Which providers and models are supported today?
And how are new models added — automatically, or on a release cycle?
- 05
Is streaming supported?
If not, is it on a roadmap with a date, or indefinite?
- 06
Is tool/function calling supported?
Required for agentic workflows; a hard blocker if missing and needed now.
- 07
What happens on a provider outage or rate limit?
Automatic failover, a documented retry policy, or a raw error passed through?
- 08
What added latency does the gateway introduce?
Ask for a number, not an adjective.
- 09
Where is the gateway deployed, and where is any logged data stored?
Relevant to data-residency and cross-border transfer questions.
- 10
Do you have a DPA, security/trust page and subprocessor list?
If not yet, is one available on request before you sign?
- 11
How is pricing structured, and what happens at 10x current volume?
Token markup, flat fee, or per-request — ask for the model, not just a sample invoice.
- 12
What is explicitly out of scope today?
Every gateway in beta or early stages has gaps. A vendor who states them plainly is more trustworthy than one who doesn't answer directly.
Process
How to run a two-week evaluation
- 1
Days 1-2: Write requirements — Identify your trigger (cost, observability, privacy, residency) and write down which criteria above are must-have versus nice-to-have before you look at any vendor.
- 2
Days 3-4: Shortlist 2-3 vendors — Match vendors to your primary category from the taxonomy above rather than comparing feature checklists across categories.
- 3
Days 5-9: Run a real integration — Point a non-critical endpoint at each shortlisted gateway using your actual request shapes, not a hello-world example, and measure latency and failure behaviour directly.
- 4
Days 10-12: Ask the fixed question list — Send the vendor-question list to each vendor in writing and compare the answers side by side, not the sales pitch.
- 5
Days 13-14: Decide and document — Record why you chose (or ruled out) each option against your written requirements, so the decision can be revisited later without repeating the whole process.
Scope check
When you do NOT need a gateway yet
A gateway solves specific problems. If none of yours apply, adding one is unnecessary infrastructure.
- You use a single provider with no plan to add a second one soon
- Your prompts don't carry customer, employee or regulated data
- Volume is low enough that direct SDK calls from one service are easy to audit by hand
- You already run an internal proxy that meets your needs and isn't causing problems
Where Privian fits
Where Privian fits — and where it doesn't
In the taxonomy above, Privian is a privacy / egress-control gateway, not a routing/cost-optimisation gateway. It is one option in this category, and for some buyers a routing-first gateway is the better fit — that's a real trade-off, not a hedge.
Fits well when
- Your trigger is a security review or customer/employee data in prompts
- You want BYOK credentials rather than a vendor-held pooled key
- You're integrating an OpenAI Chat Completions-compatible, non-streaming workload
- You want one endpoint across providers, addressed as
provider/model
Doesn't fit well when
- You need streaming, tool calling, JSON mode or multimodal input today
- Your primary driver is multi-provider cost routing and failover, not data exposure
- You need the Responses API rather than Chat Completions
- You require certifications such as HIPAA, SOC 2 or PCI today — Privian makes no such claims at this time
See the current implementation on the LLM Gateway product page or how it's positioned against named alternatives on the compare hub.
Transparency
Current limitations of this guide and of Privian
Provenance
About vendor-specific and category claims
FAQ
Frequently asked questions
- What is an LLM gateway?
- A layer that sits between your application and one or more LLM providers, giving you one integration point for authentication, model routing, observability and — depending on the vendor — data controls, instead of wiring each provider's SDK directly into application code.
- Do I need an LLM gateway if I only use one provider?
- Often not yet. A gateway earns its keep when you have more than one provider, more than one team calling a provider, sensitive data in prompts, or a security review that needs a single point to describe. A single low-volume integration can usually wait.
- What is the difference between a routing gateway and a privacy gateway?
- A routing/cost-optimisation gateway optimises for provider abstraction, failover, caching and cost attribution and generally forwards request content unchanged. A privacy/egress-control gateway adds detection and masking of sensitive values in the request path before the provider sees them. Many teams eventually want both sets of capabilities.
- Should I self-host my LLM gateway or use a managed one?
- Self-hosting gives you full control of the data path and infrastructure but makes you responsible for uptime, upgrades and scaling. Managed gateways remove that operational load in exchange for trusting a vendor's infrastructure and BYOK/credential model. The right answer depends on your team's operational capacity and your data-residency constraints, not on which option sounds more secure in the abstract.
- What should I ask a gateway vendor about data retention?
- Ask exactly what is logged by default, whether raw prompts and responses are persisted, how long any logs are kept, whether logging can be disabled or scoped, and where those logs are stored. Get the answer in writing, not inferred from a marketing page.
- Is BYOK more secure than a vendor-held API key?
- BYOK (bring your own key) means the vendor's infrastructure holds your provider credentials encrypted and decrypts them only to route a request, while usage still bills to your own provider account. It reduces one class of exposure — the vendor becoming a persistent intermediary account — but it does not by itself determine logging, retention or egress behaviour, which are separate questions to verify independently.
- How long should an LLM gateway evaluation take?
- Most teams can reach a decision in about two weeks: a few days to shortlist against written requirements, a few days to run a real integration against a non-critical endpoint, and a few days to review answers to a fixed question list before committing.
- Does Privian fit every gateway use case?
- No. Privian is a privacy-first gateway focused on detecting and masking sensitive data before it reaches a provider, with BYOK credentials and an OpenAI-compatible non-streaming endpoint. It is not the right fit today for teams whose primary requirement is streaming, tool calling, JSON mode or multi-provider cost routing as the main value — a routing-first gateway may serve those needs better.
- What is the single biggest mistake teams make when choosing a gateway?
- Evaluating on integration speed alone and only discovering the retention, credential and egress model after the gateway is already load-bearing in production. Ask the data-handling questions before the trial, not after the migration.
Next steps
Where to go next
Where to go next
LLM Gateway
See Privian's implementation of a privacy-first gateway in detail.
Compare hub
Read named, vendor-by-vendor comparisons for the compare-first buyer.
Pricing
Review current plans before you build against any vendor.
Supported models
The current provider and model catalog behind the provider/model address form.
Enterprise review
Built for enterprise review
Trust assets procurement and security teams routinely request.
Plans & pricing
Pricing for privacy-first LLM gateway use
Privian is in beta. Pricing is published transparently — review plans before you build.