LLM Failover Is Easy. Agent Failover Is Not.
As enterprises move from simple LLM applications to AI agents, putting an AI gateway in front of multiple model providers feels like an obvious architecture.
Agent
|
v
AI Gateway
/ | \
/ | \
OpenAI Claude GeminiIf one model becomes unavailable, the gateway routes the request to another.
Simple.
But there is a problem:
An agent is not a stateless API request.
Once an AI system starts maintaining conversations, calling tools, retrieving data, making decisions and executing actions, model failover becomes a context and execution-state problem, not simply a routing problem.
For teams building agents behind Kong AI Gateway, understanding this distinction is important.
Model Resilience vs Agent Resilience
Traditional API gateways are very good at handling backend failures.
A simplified pattern looks like:
Request
|
v
Backend A
|
X 5xx / timeout
|
v
Backend BLLM gateways can apply a similar concept to models:
Agent
|
v
Kong AI Gateway
|
+----> Claude
| X
|
+----> GPTThis provides model resilience.
If Claude is unavailable, rate-limited or unhealthy, another model can potentially process the request.
But that does not automatically provide agent resilience.
An agent may already have:
- 30 conversation turns
- system and policy instructions
- retrieved documents
- long-term memory
- tool calls
- tool results
- workflow state
- actions already executed
- intermediate reasoning state
- provider-specific conversation state
The fallback model needs enough of this context to correctly continue the task.
And some of it may not even belong inside the LLM context window.
The Most Important Principle: The Agent Must Own Its Context
A dangerous architecture is one where the model provider effectively becomes the system of record for the agent.
For example:
Agent
|
v
Kong
|
v
Claude Session ABC123Imagine the application relies heavily on that Claude session to maintain the conversation.
Now Claude becomes unavailable.
Kong routes the next request to another provider:
Agent
|
v
Kong
|
+----X Claude Session ABC123
|
+------> GPTGPT knows nothing about ABC123.
The model has changed, but the agent's state hasn't moved with it.
This is why production agents should maintain a canonical agent context.
What Is Canonical Agent Context?
Canonical context is the authoritative, provider-independent representation of the agent's state.
Instead of:
Claude session = agent stateor:
OpenAI conversation = agent statethe architecture becomes:
Agent / Harness
+---------------------------+
| Canonical Agent Context |
+---------------------------+
| System instructions |
| Conversation history |
| User/session state |
| Relevant memory |
| Retrieved information |
| Tool calls |
| Tool results |
| Completed actions |
| Workflow state |
| Checkpoints |
+-------------+-------------+
|
v
Kong AI Gateway
|
+--------+--------+
| | |
v v v
Claude GPT GeminiThe models become replaceable inference engines.
The agent remains the system of record.
That distinction becomes extremely important when implementing multi-model routing and failover.
Context Portability Is More Than Conversation History
It is tempting to think:
"I'll just resend all the messages to the fallback model."
That works for simple chat applications.
For agents, context is much broader.
Consider an agent performing a customer onboarding workflow:
User
|
v
Agent
|
+--> LLM: determine next action
|
+--> CRM: create customer
|
+--> Identity system: create account
|
+--> Billing system: create subscription
|
+--> LLM: generate welcome responseThe important context isn't just:
User: Create an account for John.It also includes:
CRM customer created: YES
customer_id: 83721
Identity account created: YES
Billing subscription created: YES
Current workflow step:
Generate welcome messageThis is execution state.
And it should normally live in the agent/harness, not only inside an LLM conversation.
The Dangerous Case: Partial Execution
This is where traditional gateway failover semantics become particularly interesting for agentic systems.
Imagine:
Claude
|
v
Tool call:
create_subscription()
|
v
SUCCESS
|
v
Claude response
|
X timeoutFrom the gateway's perspective, the upstream request failed.
Traditional infrastructure logic might say:
Timeout
|
v
Retry using fallbackSo:
Kong
|
v
GPTGPT receives the conversation and determines that it should call:
create_subscription()again.
You may now have two subscriptions.
Replace create_subscription() with:
transfer_money()
delete_account()
deploy_production()
send_email()
place_order()and the problem becomes much more serious.
A model retry is not necessarily an agent retry.
Idempotency Becomes an AI Architecture Requirement
Agent systems therefore need to track actions independently from model execution.
For example:
Agent Execution Store
Action ID: action-78291
Tool: create_subscription
Status: COMPLETED
Result: subscription-12345When the fallback model attempts the same operation, the agent can recognize:
action-78291 = already completedand return the existing result instead of executing the operation again.
This leads to an important architecture principle:
The gateway can provide model failover, but the agent must provide execution recovery.
Context Window Differences Matter Too
Suppose your primary model supports the context required by the agent:
Agent context
████████████████████████████But the fallback model supports a smaller effective context window.
Now failover occurs:
Primary Model
Context: OK
X
Fallback Model
Context: TOO LARGESimply switching models isn't enough.
The agent may need to perform:
- context compaction
- conversation summarization
- memory retrieval
- selective tool-result inclusion
- RAG retrieval
- token budgeting
This means model routing should ideally understand model capabilities, not just model availability.
Context Preservation Does Not Mean Behaviour Preservation
There is another subtle problem.
Assume we successfully send exactly the same context to two models.
Same Context
|
+--------+--------+
| |
v v
Claude GPTDoes that guarantee the same result?
No.
Different models may behave differently because of differences in:
- reasoning capability
- instruction following
- tool selection
- structured output
- context interpretation
- safety behaviour
- tokenization
- model training
- tool-calling implementation
So:
Context portability does not guarantee behavioural portability.
This matters when designing fallback policies.
A smaller or cheaper model may technically accept the request but may not have the capabilities required to safely continue the workflow.
Model failover should therefore ideally be capability-aware.
Provider-Specific Context Is Another Risk
Different model providers also expose different concepts around conversations, tools, reasoning and caching.
An agent tightly coupled to one provider might accumulate state that looks conceptually like:
Provider conversation ID
Provider tool-call format
Provider reasoning blocks
Provider cache references
Provider-specific message typesFailing over to another provider may require translating these concepts.
This is another reason to keep the agent's internal representation provider-neutral.
Think of it as:
Canonical Agent State
|
Context Adapter
|
+-----------+-----------+
| | |
v v v
OpenAI Anthropic Gemini
Format Format FormatThe agent owns the meaning.
Provider adapters own the representation.
Where Kong Fits
This creates a useful separation of responsibilities.
AGENT / HARNESS
Context and Execution Layer
+----------------------------+
| Conversation state |
| Memory |
| RAG context |
| Workflow state |
| Tool state |
| Checkpoints |
| Idempotency |
| Execution recovery |
+-------------+--------------+
|
v
KONG AI GATEWAY
AI Governance Layer
+----------------------------+
| Authentication |
| Authorization |
| Model routing |
| Load balancing |
| Model failover |
| Rate limiting |
| Token controls |
| Observability |
| Security policies |
| Provider governance |
+-------------+--------------+
|
+--------+--------+
| | |
v v v
OpenAI Anthropic GeminiKong provides a control point between agents and models.
But the gateway shouldn't necessarily become the system of record for every agent's workflow state.
That belongs naturally in the agent execution layer.
Think About Agent Resilience as Three Problems
A useful architecture model is:
Agent Resilience
|
+------------+------------+
| | |
v v v
Model Context Execution
Resilience Portability Resilience
| | |
v v v
Gateway Agent Agent
Harness Harness1. Model Resilience
Can another model process requests when the preferred model is unavailable?
This is where an AI gateway can provide significant value through routing, load balancing and failover.
2. Context Portability
Can the agent reconstruct the context required by another model?
This requires provider-independent agent state and careful context management.
3. Execution Resilience
Can the agent safely continue after a partial failure without repeating actions or corrupting workflow state?
This requires checkpointing, idempotency and durable execution state.
You need all three for a truly resilient agent.
Context Management Is Also a Governance Problem
There is another reason Kong should care about context: the context itself is data.
An enterprise agent's context may contain:
User identity
Customer information
Conversation history
Retrieved corporate documents
Database results
Tool outputs
Credentials or tokens
Business decisions
Prompt instructions
MemoryNow introduce multi-provider failover:
Agent
|
v
Kong
|
+--> Provider A
|
+--> Provider B
|
+--> Provider CFailover can potentially change where that data goes.
For example, an organisation may allow:
Internal data → Model Abut prohibit:
Internal data → Model BSo routing cannot be based solely on:
Which model is healthy?It may also need to consider:
What data is in this context?
Which provider is allowed to process it?
Which geography is allowed?
Which model meets the required capability?
Which model satisfies the organisation's policy?At that point, AI Gateway becomes more than an LLM load balancer.
It becomes part of the AI governance control plane.
Observability Needs to Follow the Entire Agent Transaction
Traditional API observability might tell us:
Request
Model: Claude
Latency: 2.1s
Status: 504For an agent, that isn't enough.
We may need to understand:
Agent Run
|
+--> Claude
| |
| +--> tool_call: CRM
|
+--> CRM result
|
+--> Claude timeout
|
+--> Failover triggered
|
+--> GPT
|
+--> final responseWhen something goes wrong, the question isn't simply:
"Which API returned an error?"
It becomes:
"What happened during this agent execution?"
That requires correlation across the agent, gateway, models and tools.
The Architecture Principle
For teams building production agents with Kong, I would summarize the design principle like this:
Kong should govern where inference happens. The agent harness should govern what has happened.
The gateway provides:
Connectivity
Security
Governance
Routing
Model resilience
ObservabilityThe agent provides:
Context
Memory
Workflow
Tool state
Checkpoints
Execution semanticsTogether they provide a much stronger foundation for enterprise AI agents.
Final Thought
Multi-model failover sounds simple:
Claude fails
↓
Use GPTBut for agents, the real architecture looks more like:
Claude fails
↓
Determine execution state
↓
Restore canonical context
↓
Check completed actions
↓
Select compatible fallback model
↓
Apply governance policy
↓
Transform context
↓
Continue safelyThat's a very different problem.
As AI applications evolve into autonomous and semi-autonomous agents, LLM availability will only be one part of resilience.
The bigger challenge will be preserving context, maintaining execution integrity and enforcing governance as workloads move across models and providers.
The goal shouldn't simply be:
"Can I fail over to another LLM?"
The better question is:
"Can my agent safely continue when the LLM changes?"
That is the difference between LLM resilience and agent resilience.