Keeping a long chat in the window: pruning by summarization, not deletion

Every chat assistant has a hard ceiling: the context window, the maximum number of tokens the model can read at once. The system prompt, the whole back-and-forth so far, and the new question all count against it. As a conversation grows, that running total grows with it, and sooner or later it bumps against the ceiling. When it does, something has to give.

The reflexive answer is to drop the oldest messages. It’s simple, it always works, and it’s almost always the wrong default. The better one: when the window fills up, summarize the part you’d otherwise throw away, splice the summary back in, and only delete when summarizing genuinely can’t help. I first built this loop years ago into a GPT-3.5-era chat wrapper of mine; the code has aged badly, but the pattern hasn’t. It has done the opposite: as of mid-2026, every major provider ships some version of it as a native feature. This post walks through the general shape, the token counting underneath it, and the two failure modes that bit me when I rolled my own.

When the window fills up

To know whether you’re about to overflow, you have to count tokens, and tokens are provider-specific. A word is not a token; punctuation, whitespace, and rare words all split differently depending on the tokenizer. Worse, each provider answers the “how many tokens is this?” question through a different mechanism:

• OpenAI publishes its tokenizers in tiktoken, so you count locally with no API call. Current models (GPT-4o through the GPT-5 family) use the o200k_base encoding; the older cl100k_base you’ll still find in most tutorials belongs to the GPT-3.5/GPT-4 generation. One catch: tiktoken’s model-name lookup lags new releases, so encoding_for_model() can raise KeyError on a model that’s been out for weeks (it happened with GPT-5.1). The defensive idiom is a fallback to the current encoding family:

import tiktoken

def get_encoding(model: str):
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        return tiktoken.get_encoding("o200k_base")

• Anthropic doesn’t publish a local tokenizer at all. Counting is a dedicated endpoint, client.messages.count_tokens(...), and counts are model-specific (Claude’s 4.7-generation tokenizer counts the same text differently than earlier models). Running Claude text through tiktoken undercounts substantially; don’t.

• Google used to be service-call-only too, but the current google-genai SDK gives you both: client.models.count_tokens(...) against the API, or a LocalTokenizer that downloads the vocabulary once and counts offline.

So the counting layer is a port in the hexagonal-architecture sense: a narrow, provider-specific adapter (count_tokens(messages) -> int) that the rest of the logic doesn’t need to know about. Everything above it, the actual reduction loop, is shared.

Deletion loses meaning

The standard fix is first-in-first-out: while you’re over budget, pop the oldest message and recount. It’s cheap and it always terminates, but look at what it costs you. The thing a user established forty turns ago (their name, the file they’re debugging, the constraint they stated once and never repeated) is exactly the oldest content, and it’s the first to go. The model doesn’t forget it; you deleted it. From the model’s side the early conversation simply never happened, and you get the maddening experience of an assistant that re-asks something you already answered.

Summarize the middle

The better move is to treat overflow as a problem of semantic preservation, not storage. Keep the system prompt and the most recent turns verbatim, and replace everything in between with a single summary message. Here’s the loop in provider-neutral Python; count is the adapter from above and summarize is one call to a small, cheap model:

KEEP_RECENT = 4  # newest messages that always stay verbatim

def fit(messages, budget, count, summarize):
    while count(messages) > budget:
        system, rest = messages[0], messages[1:]
        if len(rest) == 1:
            raise ContextOverflowError("newest message alone exceeds the budget")
        middle, recent = rest[:-KEEP_RECENT], rest[-KEEP_RECENT:]
        if len(middle) >= 2:
            summary = summarize(middle, max_output_tokens=budget // 4)
            candidate = [
                system,
                {"role": "user", "content": "Summary of the conversation so far: " + summary},
                *recent,
            ]
            if count(candidate) < count(messages):
                messages = candidate
                continue
        messages = [system, *rest[1:]]  # fall back: drop the oldest real message
    return messages

Three details carry the design:

The summary is capped, and the API enforces it. Passing budget // 4 as the summarizer’s max output tokens means the summary can never eat more than a quarter of the window, so folding twenty turns into it actually buys room. Ask for brevity in the prompt too; a hard cap that triggers mid-sentence truncates the summary instead of shortening it.

The fallback has to be reachable. My first implementation checked “can I summarize?” but never “did the summary help?”. When the summary didn’t shrink the conversation below budget, the loop just summarized the summary, forever, one paid API call per iteration. That’s the difference between the if count(candidate) < count(messages) guard above and an infinite loop wearing a while-statement. Summarize once; if it didn’t buy room, delete.

Steer the summarizer with the prompt, not the knobs. The old advice was “run the summarizer at temperature 0 for a faithful condensation”. On several current APIs there is no knob to turn: Anthropic removed temperature outright from Opus 4.7 onward, and OpenAI’s reasoning models restrict it. What you control is the instruction. Tell the summarizer to preserve names, decisions, constraints, and open questions, and to keep the user’s language.

Is the summary as good as the original? No. Summarization is lossy by definition, which is why the field has metrics for how lossy: ROUGE measures n-gram overlap against a reference summary, and BERTScore measures semantic similarity with contextual embeddings, which tracks human judgement better than surface overlap does. But “most of the meaning, compressed” beats “none of it, deleted” in almost every conversation that matters.

The industry converged on this

What used to be an application-layer trick is now shipped infrastructure, and it’s striking how consistently the vendors landed on the same two-tier shape.

Anthropic now offers server-side compaction as an API feature: when a conversation approaches a token threshold, the API itself summarizes the earlier context into a compaction block that you pass back on the next request. The same loop, moved behind the endpoint. Claude Code, their coding agent, auto-compacts long sessions the same way. And the deletion tier didn’t disappear either; it got smarter. Anthropic’s context editing clears stale tool results instead of summarizing them, on the observation that some content (a 2,000-line file listing from thirty turns ago) is better dropped than condensed.

OpenAI’s Agents SDK ships the pair explicitly: session memory supports both trimming (keep the last N turns, deterministic, free) and summarization (keep recent turns verbatim, compress everything older into synthetic messages), and their cookbook walks through the same tradeoff this post does. LangChain has carried the pattern the longest, from ConversationSummaryBufferMemory years ago to today’s SummarizationMiddleware.

You might expect million-token context windows to have killed the problem. They raised the ceiling, but the economics stayed: a chat API is stateless, so every turn re-sends the whole history, and input tokens cost money on every send. Compaction is how a long-running session stays affordable, which is exactly why the providers shipped it for their long-context models rather than instead of them.

How much context you keep

The payoff is easiest to see by tracking effective turns retained: how much of the original conversation still has its meaning represented in the window as the raw chat grows. The two figures below come from a toy model of the loop, not from measured summaries. The model runs a synthetic chat (each turn adds a 180-token user message and a 420-token reply) against an 8,000-token window, and assumes a summary keeps 70% of the meaning it folds in at 18% of the tokens, capped at a quarter of the window like the summary cap in the loop above. Three strategies run against it: FIFO truncation (drop whole oldest turns), per-message pruning (drop the oldest single message), and summarization-pruning.

Both deletion strategies do the same thing once the window fills: they plateau. The window can only hold so many recent turns, so everything earlier is gone, and the line goes flat no matter how long the chat runs. Summarization-pruning keeps climbing, because each old turn is folded into the summary at the assumed retention factor rather than dropped to zero. There’s a brief dip right at the crossover, where the first big summarization pass is lossier than just keeping the most recent turns verbatim. That dip isn’t an artifact of the 70% assumption (it shows up for any retention factor under about 0.9), and it’s exactly why the loop keeps deletion as a fallback rather than summarizing unconditionally.

The catch is cost. Each time the window overflows and a summary is regenerated, that’s an extra call to the summarizer model:

The retained-meaning gap widens steadily in summarization’s favour, but the count of extra summarizer calls climbs roughly linearly with the conversation. That’s the trade: a small, capped model call now and then, in exchange for an assistant that remembers the start of the conversation. For most chat applications that’s a bargain, and it’s the same small-model economics I saw when I benchmarked a tiered LLM safety gate: the cheap call earns its keep as long as you know what it can’t do.

Build it or buy it

If you’re on one provider and its managed option fits, use it: Anthropic’s compaction or the OpenAI Agents SDK session summarizer is less code and fewer bugs than anything you’ll write. The hand-rolled loop still earns its place when you sit across providers, when you need to control what the summary preserves, or when you want the deletion tier to be smarter than “oldest first”. Either way the shape is the same: count tokens through a provider adapter, summarize the middle with a capped call, keep the newest turns verbatim, and make sure deletion is actually reachable when the summary doesn’t help.

Pruning keeps a single long conversation coherent; remembering a user across conversations is a different problem, one I tackled when I gave an LLM agent memory.

References and further reading

• tiktoken: OpenAI’s BPE tokenizers for local counting; o200k_base is the current encoding family.
• Token counting, Claude API: Anthropic’s count_tokens endpoint; counts are model-specific.
• Counting tokens, Gemini API: count_tokens as a service call, plus the newer local tokenizer in the google-genai SDK.
• Compaction, Claude API: server-side summarize-the-overflow, the managed version of this post’s loop.
• Context editing, Claude API: the smarter deletion tier: clear stale tool results instead of summarizing them.
• Session memory, OpenAI Agents SDK cookbook: trimming vs summarization, with the same tradeoff analysis.
• Short-term memory, LangChain docs: the pattern as SummarizationMiddleware (formerly ConversationSummaryBufferMemory).
• Chin-Yew Lin, “ROUGE: A Package for Automatic Evaluation of Summaries”: n-gram-overlap metrics for summary quality.
• Zhang et al., “BERTScore: Evaluating Text Generation with BERT”: embedding-based similarity that correlates better with human judgement than surface overlap.