The 6 concepts of agent memory

Tap any concept to jump to its 5-card cluster.

• 1The Stateless API ProblemWhy every call starts from zero
• 2Three Memory LayersBuffer, vector store, episodic
• 3Layer 1: In-Process BufferToken-aware sliding window
• 4Layer 2: Semantic Vector StoreRecall facts by meaning
• 5Layer 3: Episodic / SummaryCompressed session memory
• 6Putting All Three TogetherOne build_context() call

Every call starts from zero

Each API call to your local Mistral endpoint is a fresh context window with no memory of any prior turn. No previously resolved entity, no user preference set five minutes ago. The model isn't broken — there's just no plumbing connecting calls to each other.

The naive fix — resend the whole conversation every time — is expensive and hits the context ceiling fast. The real fix is a tiered memory system where each layer stores a different class of information and retrieves only what's relevant to the current query.

The doctor with daily amnesia

BEFORE: Imagine a doctor who wakes up every morning with complete amnesia. Every appointment opens the same way: "Hello, I don't know who you are. What brings you in?"

PAIN: The doctor is technically skilled — they can diagnose and prescribe — but each visit starts from zero. They can't track a chronic condition, can't recall last month's dizziness, can't spot that three symptoms together form a familiar pattern.

MAPPING: That's your agent by default. The cure is memory plumbing: working notes for this visit (buffer), a searchable patient chart (vector store), and filed narrative summaries after each visit (episodic). Same skilled doctor — now with a memory.

Three ways stateless agents fail

1. Context overflowThe messages list grows past the model's window. Older turns get silently truncated — the agent "forgets" mid-conversation.
2. Lost facts across sessionsEverything lives in RAM. When the process restarts, all accumulated knowledge is gone. The next session starts cold.
3. Slow re-groundingWith no persistence, users re-explain background every session. Repeated re-grounding burns tokens and frustrates users.

Which failure you hit tells you which memory layer to add.

Match the failure to the fix

IF agent forgets earlier turns
   in the SAME conversation:
  → Layer 1 buffer overflowed

IF facts vanish after a restart:
  → add Layer 2 vector store
     (persists to disk)

IF long-running agent must "pick
   up where we left off":
  → add Layer 3 episodic memory

# fixable in <200 lines of plumbing

Misconceptions

Three layers, three jobs

A memory layer stores and retrieves information at a specific granularity and persistence level. The three layers differ in where data lives (RAM vs. disk vs. summarized text), how it's retrieved (by position vs. by meaning vs. by session), and how long it survives (this process vs. restarts vs. multi-day sessions).

Well-architected agents compose all three, not pick one: the buffer handles the immediate turn, the vector store handles facts from any past session, and episodic memory handles long-horizon narrative continuity.

Human memory, three speeds

BEFORE: Picture trying to run your whole life out of one notepad — every phone number, every conversation, every year, all on the same page.

PAIN: The page overflows instantly, and finding anything means re-reading the whole mess. One store can't be fast, durable, and compact all at once.

MAPPING: Humans split memory by role: working memory (what you're holding right now), long-term memory (searchable facts you can recall on demand), and episodic memory (the gist of past events). The three agent layers mirror exactly that — buffer, vector store, episodic — each optimized for its own job.

The three layers at a glance

1. BufferRecent turns, retrieved by position, gone on restart.
2. Vector storeFacts retrieved by semantic similarity, persisted to disk.
3. EpisodicLLM-compressed session summaries, prepended as context.

Which layer for which need

IF multi-turn coherence, follow-up
   questions in one session:
  → Layer 1 (buffer) — always on

IF specific facts must persist
   across sessions / restarts:
  → Layer 2 (vector store)

IF agent runs for days and needs
   "where we left off" continuity:
  → Layer 3 (episodic)

# they compose — not either/or

Misconceptions

A token-aware sliding window

The buffer is the in-process messages list — every raw implementation already has one implicitly. The key upgrade is making it token-aware: count tokens before appending and evict the oldest messages to stay within a budget, instead of letting the model silently truncate.

The context window is the total token budget for one call — system prompt, all messages, and the response combined. Exceeding it drops the oldest tokens invisibly. A token-aware buffer evicts predictably so you always know what the model sees.

A whiteboard, not a filing cabinet

BEFORE: Think of a small whiteboard where you jot the running notes of the meeting you're in right now.

PAIN: The board is finite. If you keep writing without ever wiping, you eventually scrawl over old notes at random — and you can't tell what got smudged out.

MAPPING: A token-aware buffer is a disciplined whiteboard: before it overflows, it deliberately wipes the oldest full exchanges to make room. You always know exactly what's still on the board — no accidental smudging, no half-erased turns.

Evict at write time, in pairs

1. Append the new messageAdd the turn to the list first.
2. Estimate tokensApproximate ~4 characters per token — good enough for budgeting without a real tokenizer.
3. Evict oldest pairsWhile over the message OR token limit, drop the oldest user+assistant pair together.
4. Return a safe listBecause eviction runs on write, get() always returns a list that fits the budget.

Pseudocode

FUNCTION add(role, content):
  messages.append({ role, content })

  # evict by message count
  WHILE count(messages) > max_messages:
    drop oldest PAIR (2 messages)

  # evict by token budget
  WHILE estimate_tokens(messages)
        > max_tokens:
    drop oldest PAIR

FUNCTION estimate_tokens(msgs):
  RETURN total_chars(msgs) / 4

# always evict user+assistant together
# so no half-turn ever confuses the model

Misconceptions

Recall facts by meaning

The vector store saves facts as embeddings — fixed-length lists of numbers that encode meaning — and retrieves them by semantic similarity, not exact text. Ask "delivery status?" and it finds the stored "order shipped" fact, because their embeddings point in nearly the same direction.

Crucially, it persists to disk and survives restarts. Embeddings run 100% locally via all-MiniLM-L6-v2 (an 80 MB CPU model, ~2ms each, no API key) and storage uses ChromaDB (local SQLite). No cloud, no cost per fact.

A librarian who knows meaning

BEFORE: Picture searching a filing cabinet that only matches exact words — you must remember the precise phrase you filed something under.

PAIN: You filed "dispatched yesterday" but search "shipping update" — and get nothing. Exact-match search fails the moment your wording drifts from the original.

MAPPING: A vector store is a librarian who understands meaning. You ask about "shipping update" and they hand you the "dispatched yesterday" note, because they grasp the two mean the same thing. You recall by intent, not exact keyword.

Save then recall

1. save(text, metadata)Embed the text locally, store the vector (dedup near-duplicates first).
2. recall(query, k)Embed the query, return the top-k closest facts by cosine similarity.
3. forget(id)Delete a specific memory by its ID.

Pseudocode

FUNCTION save(text, metadata):
  vec = embed_locally(text)   # MiniLM
  # skip near-duplicates
  IF best_similarity(vec) >= 0.95:
    RETURN existing_id
  store(id, vec, text, metadata)
  RETURN id

FUNCTION recall(query, k = 5):
  qvec = embed_locally(query)
  RETURN nearest(qvec, k)  # cosine

# persisted to disk (ChromaDB)
# survives process restarts

Misconceptions

Compress sessions into episodes

Episodic memory stores structured narrative summaries of past sessions, compressed by the LLM itself. After every N turns (or at session end), Mistral produces a compact episode: what was discussed, what was decided, what's still unresolved.

On the next session, the agent retrieves relevant past episodes and prepends them as a context prefix — so it "remembers" the arc of a long relationship without replaying the entire message history. A 50-message conversation becomes a ~200-token summary.

Meeting minutes, not transcripts

BEFORE: Imagine every team meeting were archived only as a full word-for-word transcript, hundreds of pages long.

PAIN: Before the next meeting, nobody re-reads 200 pages. The context is technically there, but it's too bulky to actually use, so everyone shows up cold.

MAPPING: Episodic memory is the minutes: a short, structured recap of decisions, open items, and key facts. You skim one page to re-enter the conversation instantly. The LLM writes those minutes for the agent after each session.

Compress, store, recall

1. Trigger compressionAfter N turns or at session end, hand the transcript to Mistral with a strong structured prompt.
2. Extract structureAsk for summary, entities, decisions, open questions, and key facts — explicit fields, not a vague blurb.
3. Store the episodeSave as a JSON file (id, timestamp, turn count). Strip any markdown fences before parsing.
4. Recall & prependNext session, match relevant episodes and prepend them as a system-context prefix.

Pseudocode

FUNCTION save_episode(messages):
  convo = join(messages)
  raw = llm(COMPRESSION_PROMPT + convo)

  raw = strip_markdown_fences(raw)
  TRY:
    ep = parse_json(raw)   # summary,
  # entities, decisions, open_questions
  EXCEPT parse_error:
    ep = { summary: raw }  # fallback

  write_file(new_id, ep)
  RETURN new_id

# next session: recall_relevant(query)
# then prepend as a context prefix

Misconceptions

One facade over three layers

The AgentMemory class composes all three layers behind two methods. build_context(query) assembles messages from every layer in priority order; save_turn(role, content) writes to the buffer and optionally to the vector store.

The agent loop never touches the individual layers — it just calls those two methods. All three-layer logic is encapsulated. Order matters: episodic prefix first (oldest, least specific), vector facts as a synthetic system message next, then buffer messages last so recent context always wins.

The briefing folder

BEFORE: Imagine an assistant who, before every meeting, had to personally raid three separate rooms — the whiteboard, the archive, the minutes cabinet — and collate everything by hand.

PAIN: That's slow, error-prone, and every part of the agent that needs context has to re-learn where all three rooms are. The plumbing leaks everywhere.

MAPPING: AgentMemory is a chief-of-staff who hands you one briefing folder: long-term background on top, relevant facts in the middle, the live conversation on the bottom. You ask once — build_context — and the folder is assembled, correctly ordered.

Assembly in priority order

1. Recall episodesMatch relevant past episodes; prepend as a system prefix.
2. Recall factsSemantic search the vector store; inject as a system message (not a fake turn).
3. Append bufferAdd recent turns last so the live conversation wins.

Pseudocode

FUNCTION build_context(query):
  messages = []

  # Layer 3: oldest context first
  eps = episodic.recall(query)
  IF eps: messages += system(eps)

  # Layer 2: relevant facts
  facts = vector.recall(query, k)
  IF facts: messages += system(facts)

  # Layer 1: recent turns win, go last
  messages += buffer.get()
  RETURN messages

FUNCTION save_turn(role, content):
  buffer.add(role, content)
  IF auto_save: vector.save(content)

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.