Production AI Application Architecture — Boundaries That Contain Failure

An AI application looks like an ordinary service until the first time it is slow. Then you discover that a single request fanned out into a retrieval call, three tool invocations, and two model completions, that nothing had a timeout, and that your p99 is now measured in minutes.

This is a guide to where the boundaries go. Not which framework to use — the frameworks change every few months and the failure modes do not.

The request path

Almost every non-trivial AI application has the same shape underneath whatever library is wrapping it:

request
  │
  ├─ 1  authorise and rate-limit          (before you spend anything)
  ├─ 2  assemble context                  (retrieval, history, user state)
  ├─ 3  call the model                    (with a budget and a deadline)
  ├─ 4  execute tools the model chose     (each one authorised on its own)
  ├─ 5  loop 3-4 until done or capped     (the cap is not optional)
  └─ 6  return, with a trace attached

The single most useful architectural decision is to make step 2 and step 4 explicitly separate from step 3, with their own timeouts, their own error handling, and their own trace spans. When they are fused into one “call the AI” function, every failure looks the same from the outside and you cannot tell a retrieval timeout from a model outage.

Budgets belong in code

Three budgets, enforced by the runtime rather than requested in a prompt:

Wall clock. A deadline for the whole request, checked before each model call and each tool call. When it is exceeded you return what you have with a clear partial-result marker, rather than continuing to spend on a request nobody is waiting for any more.

Iterations. A hard cap on loop turns. Loops that fail to terminate are the normal failure of agentic systems, not an edge case. The cap should be low enough that hitting it is a signal — if your cap is 50 and typical tasks take 4 turns, the cap is not doing anything except making a runaway expensive.

Cost. Tokens consumed per request, tracked and capped. Without this, one malformed input that sends the model into a retry loop becomes a bill.

Timeouts, and what to do when they fire

Every external call needs a timeout shorter than the deadline of whatever is waiting on it. The harder question is what happens next, and the answer differs by step:

Step	On timeout	Why
Retrieval	Proceed without it, mark the response as unsourced	A degraded answer beats no answer, but only if the caller knows it is degraded
Model call	Retry once with backoff, then fail	Transient provider errors are common; a second failure is not transient
Tool call	Return the error to the model as a tool result	The model can often route around one failed tool; killing the request removes that option
Whole request	Return partial results with a marker	Something is better than a 504, provided it is labelled

The pattern that causes the most damage is retrying the whole request on a timeout. It doubles load on a system that is already struggling, and because the earlier steps succeeded the first time, most of the retry is wasted work.

Streaming changes the failure surface

Streaming improves perceived latency and complicates everything else. Once you have sent the first token you have committed to a 200 response, so an error mid-generation cannot become a 500.

You need a way to signal failure inside the stream, and the client needs to handle it. In practice that means a terminal event carrying a status, and a client that treats “stream ended without a terminal event” as a failure rather than as completion. Both halves are required — this is one of the most common sources of silently truncated answers reaching users.

Caching, honestly

Caching in AI applications is less useful than it first appears and more useful than teams assume once they look at the right layer.

• Exact-match response caching rarely hits. Natural language input is too varied, and the hits you do get are often on the requests that mattered least.
• Embedding caching hits constantly and is nearly free. The same documents get embedded repeatedly during development and reindexing.
• Retrieval caching helps when the corpus changes slowly and queries cluster. Key on the normalised query and invalidate on reindex, not on a timer.
• Prompt prefix caching, where the provider supports it, is usually the biggest win in both cost and latency, because system prompts and few-shot examples are large and identical across requests. Structure prompts so the stable part comes first.

Where the state lives

Three kinds of state, and they should not share a store:

Conversation history is append-mostly, read on nearly every request, and grows without bound if you let it. It needs an explicit truncation or summarisation policy decided at design time, because the alternative is that the policy gets decided for you by the context limit, at runtime, in production.

Retrieval corpus is read-heavy and rebuilt in batch. Its index is a derived artifact — you should be able to delete and rebuild it from source without losing anything. Treat any system where the vector index is the only copy of the data as an outage waiting to happen.

Application state — users, permissions, audit — is ordinary relational data with ordinary requirements. Resist the pull to put it in the vector store because the vector store is already there.

The tool boundary is a security boundary

When a model can call tools, the tool is an API endpoint whose client is non-deterministic and whose input is partly derived from untrusted content. Two rules follow:

Authorise inside the tool. The model choosing to call delete_account is not evidence that the caller is allowed to delete that account. The tool must check, using the identity of the human request, every time. Passing an identity token through the loop and having the tool verify it is the whole design.

Treat tool output as data. Content returned by a tool — a fetched page, a database row, a file — can contain text that reads like instructions. If that content is concatenated into the next prompt with no separation, you have built a system where a document can direct your agent. Keep tool results in clearly delimited result blocks and never let them alter the system instruction.

Observability that answers the actual question

The question during an incident is always “where did the time go” or “why was that answer wrong”. Both need per-request attribution, which means tracing rather than aggregate metrics.

At minimum, each request should record: retrieval latency and how many documents came back, the IDs of the chunks that entered the context, prompt and completion token counts, model and version, each tool call with its duration and outcome, iteration count, and the terminal reason.

The retrieved chunk IDs are the field teams most often skip and most often need. Without them you cannot answer “was the right document even retrieved”, which is the first question in every bad-answer investigation and the one that decides whether you are debugging data or debugging a model.

The OpenTelemetry generative AI conventions give a vocabulary for most of this, which is worth adopting even if you never export to a vendor that understands it.

Deciding what to build

Is the task a single question over a bounded corpus?
  └─ yes → retrieval + one model call. No loop. No agent.
Does it need to take actions in other systems?
  ├─ one action, known in advance  → call the model, then call the tool yourself
  └─ unknown sequence of actions   → agent loop, with caps and per-tool authorisation
Does it need to be right rather than plausible?
  └─ add an evaluation suite before adding capability, not after

The bias worth having is toward the simplest structure that fits. An agent loop is a big increase in failure surface, cost variance, and debugging difficulty. Plenty of systems shipped as agents are one retrieval call and one completion wearing a loop for fashion.

Before you call it production

• Every external call has a timeout, and every timeout has a defined behaviour
• The loop has a hard iteration cap, enforced in code
• Cost per request is measured and bounded
• Tools authorise using the human caller’s identity
• Tool output cannot alter system instructions
• Retrieved chunk IDs are logged on every request
• Streaming failures produce a terminal error event the client understands
• There is an evaluation suite, and it runs on every prompt or model change
• The vector index can be rebuilt from source