The agent that forgets: Stateful vs. stateless AI agents

A language model knows a great deal and remembers almost none of it. The information it learns during training is fixed in its weights, and any insights gained during a conversation are lost as soon as the request is completed. There is permanent knowledge and total amnesia, and for many production agent deployments, the middle layer remains thin or application-specific: the model can use context supplied at request time, but it does not reliably accumulate experiences from previous interactions.

That gap is why much of what ships as an agent is, in fact, a workflow. It plans, it calls tools, and loops toward an answer but does not retain information. If you run it twice on the same problem, it makes the same mistakes with confidence, because the second run has no idea the first one happened. We believe that agents need a durable state when the task extends beyond a single request. Before understanding how the state should work and how memory and grounding make it trustworthy in a follow-up blog, it is important to know what a state is and the problem it solves. Therefore, let’s define a few terms:

• State refers to the information an agent carries forward to continue a task: messages, tool results, user preferences, workflow status, approvals, or intermediate decisions.
• Memory is a selective form of a state retained across interactions or sessions: information the system chooses to retain because it may matter later.

Statelessness is an intentional design choice

Statelessness is a valid architecture and often the right choice. A stateless agent treats every request as self-contained. As a prompt is received, the model processes it and generates a response. The entire process retains nothing. Everything the agent needs to know must be included in the payload. With no memory pinned to any particular server, a request can go to whichever instance is free. This method avoids the need for sticky sessions, shared caches, or a database on the hot path, improving scalability. For extraction, classification, summarization, and one-shot tool calls, statelessness is the right design choice. However, issues arise when tasks require continuity beyond a single interaction.

The cost of using stateless agents

The initial cost is mechanical: since the agent remembers nothing, the client must resend the entire history with each request. As payloads grow and token spend increases, the conversation history eventually approaches the model’s context window. At that point someone has to decide what gets summarized or discarded. In a purely stateless design, that responsibility often falls to the client or application layer, even though context selection is one of the decisions most likely to affect answer quality.

The second cost is more significant than the token bill. Without a durable state layer managed by the agent platform, any necessary data consolidation must occur elsewhere. The application might maintain a transcript, a summary, structured account data, or a combination of these, but it is responsible for deciding what survives from one interaction to the next. As a result, the agent is unaware that the customer is on an enterprise contract with different SLA terms, that a proposed fix from the last quarter was unsuccessful, or that a policy change took place a few months ago. Although all this information exists somewhere in history, it does not translate into actionable knowledge for the agent. You feel the impact everywhere in the deployment:

• The agent starts over on every contact. A stateless support agent asks the same qualifying questions every single time because it has no idea whether it has met this person before. The context that would have prevented the escalation existed, but the agent had no way to maintain it.
• Corrections do not persist. When a subject-matter expert takes the time to correct the agent, it lasts exactly as long as the context window. The next user relitigates the same wrong assumption from scratch. In a purely stateless system, the same mistake can repeat unless the correction is captured somewhere outside the interaction loop: in a prompt, policy, retrieval source, eval, or a product workflow.
• Long-term work needs continuity. A migration, audit, onboarding program, or complex claim may span days or weeks; the system must keep a clear record of where the work stopped, what has happened so far, and what should happen next.
• Costs scale with conversation length rather than the value they provide. Because clients resend everything at each turn, expenses rise as the transcript grows, regardless of the answer's usefulness. This is a high cost at the enterprise level.

Moving state behind the API boundary

The structural fix is to have the agent manage its state directly rather than relying on the client to remember everything. Instead of sending the complete history each time, the client sends a new message with an identifier. The agent loads the appropriate state, processes the message, and updates the state afterward.

This is a simple example. Production systems also need concurrency control, retention policies, permissions, observability, and safeguards for what gets written back into state. But this application brings two main benefits. First, the client no longer handles oversized data. Second, the server controls what determines the answer quality. History can be reranked, summarized, or reorganized on every turn without the client knowing. This method also unlocks workflows that a stateless system can't handle. For example, an agent can pause during a slow tool call or while waiting for human approval and then resume later with everything still in place. The benefits now map directly back to the costs:

Payloads remain flat. The client sends a single message regardless of how long the conversation has run, and token usage no longer scales with transcript length.

The agent stops starting over. A stateful support agent opens with the account tier, the open ticket, the integration that broke last quarter, and the agreed-upon workaround. In support environments, that continuity affects the operational metrics teams work with: handle time, deflection rate, and escalation volume. The exact impact depends on the workflow, data quality, and guardrails around the agent.

Corrections are durable. When an expert corrects the agent, that correction is written into the state and outlives the context window, turning a one-time fix into institutional knowledge.

Personalization happens without retraining. The system adapts to an account, team, or regulatory context based on its current state rather than through fine-tuning. There is no need for a retraining cycle, model refresh, or waiting for the ML team.

Long-term projects are now feasible. Tasks that once took days or weeks now have an agent that can track progress.

The tradeoffs

Managing your infrastructure can also add up to the costs. A reliable system that can handle user requests efficiently is essential. One major challenge is that if a user's session is tied to a specific server but their next request is routed to a different one, it can cause problems. We call this localized amnesia, where the new server has no record of the user's session information. To address this, there are a couple of common solutions: you can either store session data in a central location, such as Redis or Postgres, or ensure that user requests always go to the same server—this is known as session affinity. However, these approaches come with their own complexities. This is the reality of scaling your infrastructure: as performance and user load increase, you'll need to consider trade-offs.

If you are building a summarizer or a one-shot classifier, stay stateless and avoid a database to solve a problem that doesn’t exist. But if you are building something that will be useful on day thirty, you must be stateful.

Building a trustworthy state

Moving state server-side is the necessary first step. Once an agent carries beliefs forward over weeks of interactions, someone has to check whether those beliefs are true. A stateful agent that learns from experience without verification slowly convinces itself of things nobody ever said and states them with growing confidence because it now holds them as established knowledge. That is why we think memory and grounding are the same problem viewed from two angles. Memory is what turns history into experience; grounding is the process of tying outputs or stored claims back to trusted sources. Getting the first without the second is how you build an agent that confidently remembers the wrong thing for six months.

At Vectara, much of our work is focused on the reliability layer around generation: retrieval from governed data, grounded responses with citations, hallucination detection and correction, and permission-aware pipelines. Those same requirements apply to agentic memory. If an agent stores something for later, the system should know where it came from, whether the user is allowed to use it, and how to check it before treating it as fact. Agentic memory is the natural extension of that work and an area we are actively developing. In the next post, we will get into how memory should actually work: how an agent decides what is worth keeping, how stored beliefs stay attributable to their sources, and how you keep a memory system from quietly hallucinating its own facts.

If you are looking to build agents that adapt to your unique data and policies, remember what is important over time, and remain trustworthy, contact us!