Context window

An LLM’s context window is the maximum token budget available for the material a model processes in one request or interaction. Depending on the API and model, that budget can include system instructions, developer instructions, conversation history, retrieved passages, tool descriptions and results, the current user message, and some or all of the generated output.

It is measured in tokens, not words or characters. Tokenization varies by model, language, whitespace, punctuation, and code. Count with the tokenizer or usage data for the exact model instead of applying one universal words-to-tokens conversion.

Capacity is not reliable recall

If a model accepts a document, that does not prove it will use every part equally well. The study Lost in the Middle found that moving relevant information within a long input could materially change performance, with information in the middle often harder to use than information near the beginning or end for the evaluated tasks and models.

This does not mean every model always ignores the middle. It means advertised capacity and task-specific effectiveness are different measurements. Test your model, prompt, document shape, and question distribution.

The underlying Transformer architecture uses attention to compute contextual sequence representations, as introduced in Attention Is All You Need. Product implementations differ, and larger contexts can involve architectural and serving techniques beyond the original paper. For application design, the durable lesson is that input position, relevance, and competition can affect results.

What consumes the window

A request may look short in the interface while carrying substantial hidden or generated material:

• the application’s system and policy instructions;
• schemas and descriptions for callable tools;
• prior conversation turns;
• search or retrieval results;
• images or other inputs represented through model-specific accounting;
• the new question;
• reserved output capacity.

Suppose a model accepts a context budget of C tokens. A planning estimate is:

available input = C - reserved output - system - tools - safety margin

Use actual usage data because provider accounting differs. Leave a safety margin for variable user inputs and tool results. Decide whether the API rejects an oversized request, truncates part of it, or requires the application to shorten it.

A context-budget worksheet

Record a high-percentile estimate rather than only the average:

The numbers here are illustrative, not recommendations. Fill the table from your own traffic and preserve enough headroom for unusual but valid requests.

Better context is usually selected context

Sending everything feels safe because nothing is deliberately omitted. In practice, irrelevant material can introduce conflicting instructions, stale facts, privacy exposure, and extra cost. Prefer a selection policy:

1. Keep stable control instructions concise.
2. Include only tools the current step may call.
3. Retrieve evidence for the present question rather than pasting a whole corpus.
4. Preserve source identifiers, dates, and version metadata.
5. Select conversation turns that establish active goals and constraints.
6. Summarize history only when the summary itself can be checked or safely corrected.

For a legal contract question, selecting the relevant clause plus its definitions and exceptions is usually more defensible than adding fifty unrelated pages. For code assistance, the active file, interfaces, error, and neighboring tests may matter more than the entire repository.

A position-sensitivity test

Create a synthetic but realistic input containing several independent sections. Put a unique answer-bearing fact in one section, such as release_code = ORCHID-417. Ask the same question while moving that section:

• near the beginning;
• around one quarter;
• in the middle;
• around three quarters;
• near the end.

Repeat with paraphrased questions and distractor facts. Record exact-match recovery, unsupported answers, latency, and token use. Then run a semantic task using real documents; unique-code retrieval alone does not measure synthesis.

If performance varies, duplicating the fact at both ends only hides the problem. Improve retrieval, reduce irrelevant text, label evidence, and place the task and output contract clearly. Re-test after any model or prompt change.

Context window, memory, and training are different

• Context is material available during the current computation.
• Conversation memory is application-managed information saved and later reintroduced.
• Retrieval fetches external records for the current request.
• Training changes model parameters through an optimization process.

A chat product may feel persistent because it stores a preference and includes it later. That does not mean the current conversation permanently retrained the base model. Ask what is stored, where, for how long, and how it is used.

Practical rules

• Count tokens using the target model or its reported usage.
• Budget the output as well as the input.
• Treat the maximum as a hard capacity, not a quality promise.
• Test information at different positions and input lengths.
• Retrieve and rerank decisive evidence instead of filling unused space.
• Preserve citations and version metadata when facts matter.
• Monitor truncation, request rejection, latency, and cost.
• Re-evaluate when the model, tokenizer, prompt, or retrieval policy changes.

A context window tells you how much material can enter the interaction. Only task-specific evaluation tells you how much of that material the system can use reliably.