Context Window

A context window is the maximum number of tokens a language model can process in a single input, including both the prompt and generated output.

In large language models, the context window defines the span of text the model can attend to when generating a response. It is measured in tokens, which are units of text such as words, subwords, or characters. The context window includes the user’s input prompt and any text the model generates, up to a fixed limit set by the model’s architecture. For example, a model with a 4,096-token context window can process a prompt of 2,000 tokens and generate up to 2,096 tokens in response.

The context window size directly affects the model’s ability to maintain coherence over long passages. A larger context window allows the model to consider more preceding text, which is critical for tasks like document summarization, multi-turn conversations, or analyzing lengthy code files. However, increasing the context window requires more computational resources, as the attention mechanism in transformers scales quadratically with sequence length. This trade-off has driven research into efficient attention mechanisms, such as sparse attention or sliding windows, to extend context length without proportional cost increases.

Practical implementations vary widely. Early models like GPT-2 had context windows of 1,024 tokens, while modern models such as GPT-4 support up to 128,000 tokens, and some specialized models claim even larger limits. The context window is a key specification for users, as exceeding it truncates input or output, potentially losing important information. Developers must design prompts and applications to stay within this limit, often using techniques like chunking or summarization to handle longer texts.

Why it matters

The context window determines how much information a model can consider at once, directly impacting its usefulness for real-world tasks. A small window forces users to condense inputs, risking loss of nuance, while a large window enables handling of entire documents or extended conversations. This specification influences model selection for applications like legal document analysis, long-form content generation, and interactive chatbots, where maintaining context over many turns is essential.

Related terms

• Token / Tokenization
• Large Language Model (LLM)

FAQ

How does it work?

The context window is implemented through the model’s attention mechanism, which calculates relationships between all tokens within the window. During processing, the model assigns attention weights to each token pair, allowing it to reference earlier tokens when generating later ones. The window size is a fixed architectural parameter, often determined by the maximum sequence length the model was trained on.

What happens when input exceeds the context window?

When input exceeds the context window, the model typically truncates the beginning of the prompt or fails to process the excess tokens. Some models may raise an error, while others silently drop tokens beyond the limit. This can cause loss of context, especially in long conversations or documents, where early information becomes inaccessible.

How does context window size differ between models?

Context window sizes vary significantly across models, from 2,048 tokens in older models to 128,000 tokens in recent ones like GPT-4 Turbo or Claude 3. Larger windows require more memory and computation, so they are often reserved for high-end models. Specialized models, such as those for code or long documents, may prioritize larger windows, while smaller models trade off context for efficiency.