2 Agentic infrastructure

A model call is one operation inside a larger system. Serving, tool execution, state storage, and evaluation determine whether that operation becomes a useful, repeatable workflow.

The agent architecture chapter separates decisions from execution. Infrastructure makes that separation observable: every transition should have enough evidence to explain what happened and enough structure to recover when it fails.

Trace a single request

Begin with a request identifier that follows a task through its components. Record the model invocation, tool proposal, validation result, execution outcome, and final response as related events. A distributed trace represents work as spans with parent–child relationships; OpenTelemetry’s tracing concepts provide a standard vocabulary.

Useful measurements answer different questions:

• Latency: how long did the user wait, and which operation dominated?
• Consumption: how many model tokens and tool calls were used?
• Outcome: did the task satisfy a verifiable completion criterion?
• Failure: was the problem a model decision, an invalid tool argument, a dependency error, or an exhausted budget?

Logging every raw prompt and response can expose confidential inputs. Choose what to retain deliberately, and keep identifiers separate from secrets and access credentials.

Make state transitions explicit

A task may move through queued, running, waiting, completed, or failed. A retry should resume from a known state rather than guessing which side effect already occurred.

For example, retrying a read is different from retrying a payment. Before executing an operation with an external effect, decide how its identity will be recorded and how repeated delivery will be detected. An idempotency key can identify one intended operation across retries when the downstream service supports that contract.

task-event.json

{
  "task_id": "example-task-17",
  "step": 3,
  "operation": "lookup_document",
  "status": "completed",
  "duration_ms": 184,
  "attempt": 1
}

This event is a deliberately small example. A production schema should match the questions the system needs to answer, rather than accumulating fields because they might be useful someday.

Evaluate the complete workflow

A model benchmark and an application evaluation measure different systems. An application can fail even when each model response looks plausible: a tool may receive the wrong identifier, a retried action may run twice, or the task may stop before verification.

Build a small evaluation set around real task contracts. For a document lookup assistant, a test might require the correct document, a supported answer, and no unauthorized writes. Preserve the initial state and expected outcome so a future implementation can be compared under the same conditions.

Start with failures you can classify

When a run fails, record the first incorrect transition and the evidence needed to reproduce it. “The agent was wrong” is difficult to fix. “The executor accepted a document ID absent from the retrieved result set” identifies a concrete boundary to improve.

Keep the first deployment small

One well-instrumented worker and a bounded execution loop can teach more than a large orchestration platform with unclear task semantics. Add scheduling, parallelism, and durable storage when the workload requires them. Keep model, tool, and infrastructure changes distinguishable so an evaluation can explain which change affected the outcome.

Continue with a note on evaluating bounded agents for a compact worked example.