The 5 concepts of conversation management

Tap any concept to jump to its 5-card cluster.

• 1The Stateless RealityThe model has no memory
• 2History Management PatternsFull, sliding window, summarize
• 3Token Budget AllocationFour consumers, one window
• 4Message Pruning StrategiesCut filler, keep the facts
• 5Production Conversation ManagerStore, prune, summarize, persist

The model forgets between every call

Each API call to the model (Mistral via Ollama) is stateless: there is no session, no server-side storage, no memory of any kind. What feels like a continuous chat is an illusion your code creates.

To give the model “memory,” you resend the entire message history — system prompt plus every prior user and assistant turn — on every single request. Leave a message out and, to the model, it never happened.

The expert with perfect amnesia

BEFORE: Imagine hiring a world-class expert for advice, but every time you walk into the room they have zero memory of any previous conversation with you.

PAIN: Say “so what about option B?” and they stare blankly — no idea what option A was, who you are, or what problem you're solving. Every interaction starts from absolute zero.

MAPPING: That's exactly how the API works. Each call is a fresh room with a fresh expert. The only way to give “memory” is to hand over a written transcript of every prior exchange — your code replays the whole conversation from scratch each time.

What actually goes over the wire

1. Build the full arraysystem message + all prior user and assistant turns, in order.
2. Send everythingThe whole array goes in the request — nothing is stored server-side.
3. Get one replyThe model reads it all fresh and returns the next assistant message.
4. Append & repeatAdd the reply to your array; next turn resends the whole (bigger) thing.

Pseudocode

history = [ system_message ]

FUNCTION send(user_text):
  history.append({ user: user_text })

  # send the WHOLE array — no memory
  # exists on the server side
  reply = api.chat(messages = history)

  history.append({ assistant: reply })
  RETURN reply

# 'history' IS the memory. It lives in
# your code and grows every single turn.

Misconceptions

Three ways to carry the history

Because you resend history every turn, you need a strategy for what to send. Three patterns exist: full history (send everything), sliding window (keep only the last N messages), and summarization (compress old turns into a short summary, then drop the originals).

Each trades off differently. Full history is simplest but grows unbounded. A window is cheap but forgets early context. Summarization preserves meaning at bounded cost. Most production agents use a hybrid: a summary + recent turns + the system message.

Packing one strict carry-on bag

BEFORE: Imagine packing for a two-week trip, but the airline allows one carry-on with a strict 7 kg limit.

PAIN: Cram everything in and it's rejected at the gate. Pack only what fits right now and you might reach the formal dinner in hiking clothes because you dropped the dress shoes days ago.

MAPPING: The context window is that bag. Full history crams everything in (fine for short trips). Sliding window packs only the last few days (cheap, but you lose early essentials). Summarization writes a packing list of what you left behind plus today's essentials — the best balance for long journeys.

The three strategies

1. Full historySend all messages every time. Simplest code; best for short chats under ~20 turns.
2. Sliding window (N)Keep only the last N messages. Fixed cost, but early context is lost forever.
3. SummarizationCompress old turns into a summary, prepend it, drop the originals. Keeps meaning at bounded cost.
4. Hybrid (production)Summary of old turns + full recent turns + always-present system message.

Which strategy to use

IF chat is short (< ~20 turns):
  → full history (simplest)

IF only recent context matters
   AND you want fixed cost:
  → sliding window (last N)

IF early facts still matter
   in a long conversation:
  → summarize old + keep recent

# production default: hybrid —
# summary + recent turns + system

On local Mistral, summarization is an extra Ollama call — no external cost, but it adds latency.

Misconceptions

Four consumers share one fixed window

The context window is a fixed size (e.g. 32K tokens for Mistral via Ollama). Four things compete for that space: the system prompt, the conversation history, the current user message, and the reserved response.

History grows every turn, so the space left for a response shrinks over time. You must count input tokens before each call and prune when the remaining headroom drops too low — otherwise the model runs out of room to answer.

The fixed-size suitcase

BEFORE: Imagine a suitcase with exactly 32,000 units of space — not one more — and everything you need must fit inside.

PAIN: You pack a travel guide (system prompt), a photo album of memories (history), today's souvenirs (current message), and must leave room to bring things home (the response). Overstuff the album and there's no space left for souvenirs or the trip home.

MAPPING: The context window is that suitcase. The four consumers — system prompt, history, current message, and reserved response — all compete for the same fixed space, and the album (history) keeps growing.

Budgeting each request

1. System promptFixed 200–800 tokens — but paid on EVERY request, not just the first.
2. HistoryVariable and growing; the piece you actively manage with pruning.
3. Current messageVariable — this turn's user input.
4. Response headroomWhat remains after input. Keep enough space for the model to reply.

Pseudocode

# run BEFORE every API call
input = estimate_tokens(all_messages)
# rough rule: ~4 chars = 1 token

available = CONTEXT_LIMIT
             - system_tokens
             - history_tokens
             - current_msg_tokens

IF available < RESPONSE_HEADROOM:
  # not enough room to reply
  history = prune_to_budget(history)

SEND messages

Misconceptions

Cut the filler, keep the facts

Pruning is selectively removing messages to stay within token limits while preserving the information the model actually needs. It's not just “drop the oldest” — you score messages by importance.

Greetings and filler get cut first; key facts (account numbers, decisions, tool results) are preserved no matter how old. The critical rule: after removing any message, the remaining array must still alternate user/assistant, with a user message first after the system prompt.

The film editor's cut

BEFORE: Imagine you're a film editor with 6 hours of raw footage that must become a 90-minute movie — and key plot points are scattered throughout.

PAIN: Blindly cut the first 4.5 hours and you lose the character introductions. Keep everything and the audience drowns in filler scenes.

MAPPING: Pruning a conversation works the same way. You score each message by importance: greetings and filler are cut first; key facts are preserved no matter how old; and sometimes you replace a whole block of scenes with a narrator's summary.

Five pruning strategies

1. FIFODrop the oldest messages first. Simple, but may lose critical early context.
2. Importance scoringTag messages with metadata; drop low-importance ones first.
3. Semantic dedupRemove redundant exchanges that repeat the same meaning.
4. Role-based retentionAlways keep certain types: tool results, preferences, key decisions.
5. Summarize-and-replaceCondense a block into one summary before dropping the originals.

What to cut, what to keep

FOR each message, oldest first:

  IF it's greeting / filler:
    → cut it first

  IF it holds a key fact
     (id, decision, tool result):
    → keep, no matter how old

  IF a big old block remains:
    → summarize-and-replace it

# AFTER cutting: verify user/assistant
# still alternate, user-first. Broken
# alternation = API error.

Misconceptions

One class that owns the whole loop

A conversation manager is a class that sits between your app and the Ollama API. It bundles everything from the first four concepts into one reusable component so your app code just calls send().

It handles six responsibilities: message storage, token counting, automatic pruning, summarization, metadata tracking, and serialization to JSON so conversations survive restarts. Build it in layers — basic, then sliding window, then auto-summarizing and persistent.

The CEO's personal assistant

BEFORE: Imagine a CEO who takes every meeting with no assistant — carrying a growing stack of every note from every past meeting, verbatim and unfiltered.

PAIN: By month three the stack is so tall they spend more time flipping through old notes than engaging. Week-one decisions are buried under pages of “sounds good” and “let's circle back.”

MAPPING: The conversation manager is that assistant. It records every exchange, scores messages by importance, compresses old meetings into executive summaries, and ensures the model walks into each meeting with exactly the right briefing.

Build it in three layers

1. Basic managerStore messages, prepend the system prompt, call Ollama, track token usage.
2. Sliding windowOverride “get messages” to send only the last N — and fix alternation.
3. Smart managerAuto-summarize once tokens cross a threshold; keep recent turns verbatim.
4. Persistsave()/load() serialize full state to JSON so chats survive restarts.

Pseudocode

CLASS Manager:
  messages = []
  threshold = 8000

  FUNCTION send(user_text):
    messages.append({ user: user_text })
    reply = api.chat([system] + messages)
    messages.append({ assistant: reply })

    IF last_input_tokens > threshold:
      summarize_old()   # keep recent
    RETURN reply

  FUNCTION save(path):
    write_json(path, all_state)

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.