Agent Memory Explained: Short-Term, Long-Term and Context Management

Imagine hiring an assistant who forgets your name every time you blink. That is exactly what a language model does between calls. Nothing you said yesterday, or even ten minutes ago, survives unless somebody puts it back in front of the model. Today we learn how to give agents memory that actually works.

• Agents start every call blank
• Memory must be engineered

In the last lesson we built the ReAct loop, where an agent thinks, acts and observes in a cycle. That loop piles up text fast. In this lesson we cover short-term memory, long-term memory, procedural memory, how to score what is worth remembering, how to manage the context window, and finally Python code that plugs two memory systems into our agent.

• Short-term and long-term memory
• Scoring, compaction, privacy
• Python implementation

Here is the core idea. A model's weights are frozen at inference time. The only thing that changes from call to call is the text you send in. So the context window is the agent's entire working memory. If a fact is not in the window, then as far as the model is concerned, that fact does not exist.

• Model weights are frozen
• Only the prompt carries state

Every turn appends new tokens, and in an agent loop the growth is brutal, because each tool result gets appended too. Say your window is one hundred twenty eight thousand tokens. A few long web pages and a code file can eat half of it in minutes. Even before it is full, very long contexts cost more, run slower, and models can miss details buried in the middle.

• Every turn appends tokens
• Cost and latency grow
• Quality can degrade

So we organise memory into three families. Short-term memory lives inside the window: the conversation buffer, a sliding window, or a running summary. Long-term memory lives outside in a database and is retrieved on demand. Procedural memory is the how-to knowledge, such as system instructions and skill files. Let us take them in turn.

• Short-term: inside the window
• Long-term: outside, retrieved
• Procedural: instructions and skills

The simplest approach is the conversation buffer: keep every message and send them all each turn. It is lossless and trivial to implement. The problem is that cost grows with every turn, and because each call resends the whole history, the total tokens processed over a conversation grow roughly with the square of its length. Eventually you hit the wall.

• Keep every message verbatim
• Simple and lossless
• Grows without bound

The next option is a sliding window: keep only the last N messages, say the last ten. Cost per call is now bounded, which is great. The catch is brutal forgetting. If the user told you their budget in message two, it falls off the edge at message thirteen, and the agent will happily suggest something far too expensive.

• Keep only last N messages
• Fixed cost per call
• Forgets early facts

The best-known compromise is buffer with summary. Keep the recent messages verbatim, and compress everything older into a short summary written by the model itself. A summary of forty turns might take two hundred tokens instead of eight thousand. The cost is that summaries are lossy, and each re-summarization can blur details further, so the prompt for summarizing matters a lot.

• Compress old turns with an LLM
• Keep recent turns verbatim
• Lossy but compact

In practice the summary rolls. When the buffer passes a token threshold, you take the oldest messages plus the existing summary and ask the model to produce a new summary. Tell it explicitly to preserve names, numbers, decisions and open tasks, because those are what generic summaries tend to lose. Trigger it by counting tokens, not messages, since messages vary wildly in size.

• Fold old summary plus dropped turns
• Preserve names, numbers, decisions
• Trigger by token count

Short-term memory dies when the session ends. Long-term memory persists across sessions, stored outside the model and retrieved only when relevant. This is what lets an assistant remember that you are vegetarian next month. There are three common shapes for it: vector-store memories, key-value facts, and episodic logs.

• Persist across sessions
• Stored outside the window
• Retrieved when relevant

Vector-store memory reuses what you learned in the embeddings and RAG lessons. Each memory is a short sentence, such as user prefers window seats. You embed it, store the vector with metadata, and at query time embed the current message and fetch the nearest memories by cosine similarity. It is fuzzy and scales well, but it can return near-misses and has no notion of truth.

• Embed each memory snippet
• Retrieve by similarity
• Fuzzy, scales well

Key-value memory stores structured facts: name equals Priya, diet equals vegetarian, timezone equals Berlin. Lookup is exact, updating is trivial because you overwrite the key, and you can always inject the whole profile since it is tiny. The downside is that you must decide the schema, and free-form knowledge does not fit neatly into keys.

• Structured profile entries
• Exact lookup, easy to update
• Great for stable preferences

Episodic memory is a log of what happened: the task, the steps taken, the outcome, and a timestamp. An agent that failed to parse a certain API last week can look up that episode and avoid repeating the mistake. Over time you can distill many episodes into short lessons, which keeps the log useful instead of just huge.

• Record what happened and when
• Timestamped events
• Summarize into lessons

Procedural memory is knowing how, not knowing what. It is the system prompt, a project instructions file, or a library of skill files that describe how to do a task like writing a release note. Good designs keep only short descriptions in context and load a full skill file when the task calls for it. This keeps the window lean while the agent stays capable.

• Instructions and skills files
• Loaded when needed
• Edited by humans or the agent

Whatever the store, memory has four operations: write, retrieve, update and forget. Storage technology is the easy part. The hard part is the policy behind each operation: what is worth writing, how to find the right thing, what to do when facts change, and when to let go. Let us look at each.

• Four operations
• Each needs a policy
• Policies matter more than storage

What deserves to be written? Durable facts, stated preferences, decisions, corrections, and lessons from failures. What does not: small talk, one-off details, and raw transcripts. A good pattern is to run an extraction step after a conversation that outputs atomic, self-contained sentences. Atomic means each memory states one fact and makes sense without the surrounding chat.

• Durable, reusable, user-specific
• Not raw transcripts
• Extract atomic facts

To retrieve well, we score every candidate memory on three things. Recency, which decays with age. Importance, a number from zero to one, often rated by the model when the memory is written. And relevance, the cosine similarity between the memory and the current query. The final score is a weighted sum of these three terms. This idea was popularised by generative agent simulations.

• score = recency + importance + relevance
• Each term scaled 0 to 1
• Weights are tunable

Let us use numbers. The query is: suggest a dinner place in Berlin. Recency is zero point nine nine raised to the number of hours since the memory was last used. Memory A, user is vegetarian, is forty eight hours old, so recency is about zero point six two. Memory B, asked about weather, is twenty four hours old, giving zero point seven nine. Memory C, flight to Berlin Friday, is two hours old, giving zero point nine eight.

• recency = 0.99 ^ hours
• 48 hours gives 0.62
• 24 hours gives 0.79
• 2 hours gives 0.98

Now importance and relevance. A has importance zero point nine and relevance zero point eight two, so the total is about two point three four. B has importance zero point two and relevance zero point three, total one point two nine. C has importance zero point eight and relevance zero point seven, total two point four eight. So C ranks first, A second, and the weather memory loses clearly.

• A: 0.62 + 0.90 + 0.82 = 2.34
• B: 0.79 + 0.20 + 0.30 = 1.29
• C: 0.98 + 0.80 + 0.70 = 2.48

The weights are knobs. If you double the relevance weight, A becomes three point one six and C three point one eight, nearly tied, because both match the query well. Raise recency weight for fast-moving topics, raise relevance for lookup-style tasks. Also set a minimum score threshold, so that when nothing is relevant the agent retrieves nothing rather than injecting noise.

• Raise relevance weight for search-like tasks
• Raise recency for fast-changing domains
• Apply a score threshold

Updating is where many systems fail. When the user says they are now vegan, you must find the old vegetarian memory and replace it, not store both. On write, retrieve the nearest existing memories and ask the model whether the new fact adds, updates, or contradicts them. Forgetting means expiry dates, decaying unused memories, and honouring explicit delete requests from the user.

• Check for conflicts on write
• Supersede, don't duplicate
• Decay, expire, or delete

Memory is one part of a bigger discipline called context engineering: deciding exactly what goes into the window on each call. Think of the window as a budget. System prompt, tool definitions, memory, retrieved documents and history all compete for space, and every token should earn its place.

• Decide what enters the window
• Every token should earn its place
• Order and stability matter

Compaction is the main tool for long-running agents. Suppose the window is one hundred twenty eight thousand tokens and you trigger at eighty percent, which is one hundred two thousand four hundred. At that point you summarise the oldest sixty thousand tokens into about fifteen hundred, a forty to one reduction, and continue. Keep the goal, decisions made, files touched and open tasks.

• Trigger at 80 percent full
• Summarize old turns and tool results
• Keep goals, decisions, open tasks

Tool outputs are usually the biggest source of bloat. A web page might be twenty thousand tokens, yet the agent needs two sentences. Truncate or pre-process results: keep the first thousand tokens, extract only relevant parts, or save the full output to a file and return a short preview with a pointer. The agent can always ask for more later.

• Tool results are the biggest bloat
• Keep head, tail, and pointer
• Let agent fetch more on demand

How is memory retrieval different from RAG over documents? The machinery is nearly identical: embed, search, inject. The difference is the source and the lifecycle. RAG indexes shared, mostly static documents. Memory holds knowledge generated by interaction, specific to a user or agent, and it changes constantly, so it needs writes, updates and forgetting that a document index rarely needs.

• RAG: shared knowledge documents
• Memory: user and agent experience
• Same machinery, different source

One practical wrinkle is prompt caching. Many providers cache the processed prefix of your prompt, making repeated prefixes cheaper and faster. That rewards a stable layout: system prompt and tools first, then history, with new content appended at the end. If you rewrite the summary near the top or reshuffle retrieved memories every turn, you invalidate the cache. So compact in batches, and place volatile memories late.

• Stable prefix gets cached
• Append, don't edit earlier text
• Put volatile memory late

Now the code. We write a class called SummaryBufferMemory. It holds a summary string and a list of recent messages. After every add, it counts tokens. If the total passes a limit, it pops the oldest messages, asks the model to merge them with the existing summary, and stores the result. A method called render returns the summary as a system note followed by the recent messages.

• Class holds summary and recent list
• Token counter triggers compaction
• LLM call writes new summary

Next, a class called VectorMemory. Its remember method embeds a fact and stores the vector with metadata: user id, creation time and an importance rating. Its recall method embeds the query, computes cosine similarity against stored vectors for that user only, combines it with recency and importance using our weighted score, drops anything under the threshold, and returns the top k.

• remember: embed and store
• recall: score and return top k
• Metadata: user id, time, importance

Plugging this into our ReAct agent takes two hooks. Before the loop starts, call recall with the user's message and inject the results into the prompt under a heading like known facts about this user. After the loop finishes, run an extraction prompt over the conversation, write the new atomic facts, and let the update check handle conflicts. The tool loop itself does not change at all.

• Before the loop: recall and inject
• After the loop: extract and write
• Keep tool loop unchanged

Memory is personal data, so treat it that way. Store the minimum you need, avoid secrets such as passwords and keys, let users see and delete what is remembered, and comply with the regulations that apply to you. Staleness is the other risk: a remembered address or job title can be wrong today. Keep timestamps, prefer recent facts, and have the agent confirm old, high-stakes details.

• Store minimal personal data
• Let users view and delete
• Timestamp and verify old facts

The first classic mistake is storing everything. It feels safe, but a store full of trivia drowns the useful memories, retrieval returns junk, and your privacy exposure grows. Be selective at write time, give memories an importance rating, and prune low-value entries on a schedule.

• Noise drowns signal
• Retrieval quality drops
• Cost and privacy risk rise

The second mistake is contradictory memories: lives in Paris and lives in Berlin both retrieved, and the model guesses. Fix it with update-on-write and timestamps. The third and most serious is leaking across users. If retrieval is not filtered by user id, one customer's private facts can appear in another customer's answer. Enforce the filter in the storage layer, not in the prompt.

• Contradictory memories confuse the model
• Always filter by user id
• Namespace per user or tenant

Let us recap. The context window is the only working memory. Short-term memory uses buffers, sliding windows and summaries. Long-term memory uses vector stores, key-value facts and episodic logs, and procedural memory holds skills. Retrieve with recency, importance and relevance, manage the window with compaction and truncation, mind the cache, and protect privacy by isolating users.

• Context window is working memory
• Short-term, long-term, procedural
• Score, compact, protect

Next time we ask what happens when one agent is not enough. In multi-agent systems, several agents split a task, and suddenly memory and context become a question of what to share and what to keep private. When is it worth the extra complexity, and how do you build it well?

• When one agent is not enough
• How agents share context
• Subscribe for the next lesson