Context Windows: What They Cost and How to Use Them Well

A large language model's (LLM)^[1] context window is its short-term memory. It defines the amount of text, measured in tokens, that the model can see and process at one time when generating a response. This includes both the user's input and any previous conversation history. For years, small context windows were a significant constraint, but recent models now boast windows of hundreds of thousands or even a million tokens, enough to fit entire books.

This expansion appears to solve many problems, allowing models to analyze long documents, maintain extended conversations, and perform complex reasoning without specialized training. This power is not free. Larger context windows introduce substantial costs in money, latency, and computational resources. They also have surprising failure modes, like degraded accuracy on information presented in the middle of a long prompt. For engineers and technical leaders, building effective and efficient systems requires moving beyond the simple assumption that a bigger window is always better. It demands a clear understanding of the underlying mechanics and a deliberate strategy for when to use a large context and when to employ other patterns.

The Mechanism: How Context Windows Actually Work

At its core, an LLM is a function that takes a sequence of text as input and predicts the next most likely piece of text. The input sequence is the context. The process of using this context to generate an output involves several distinct steps rooted in the Transformer architecture^[2], first introduced in the 2017 paper "Attention Is All You Need."

Tokenization

First, the input text is not processed as raw characters. It's broken down into a sequence of tokens using a tokenizer. A token is a common sequence of characters, which can be a whole word or a piece of one. For English text, one token represents roughly four characters or three-quarters of a word. Most modern models use a method like Byte Pair Encoding (BPE)^[3], which creates a vocabulary of these sub-word units. For example, the word unhappiness might be tokenized into un, happi, and ness. The size of the context window is defined by the maximum number of these tokens the model can handle, like the 128,000 tokens for OpenAI's GPT-4 Turbo^[4].

Embeddings and Positional Encoding

Once tokenized, each token is converted into a numerical vector called an embedding^[5]. This vector represents the token's meaning in a high-dimensional space. However, the original Transformer architecture is inherently position-agnostic; it sees the input as an unordered bag of tokens. Without information about sequence order, sentences like "dog bites man" and "man bites dog" would be indistinguishable.

To solve this, the model adds a second vector, called a positional encoding, to each token's embedding. This encoding vector doesn't represent the token's meaning but its absolute position in the sequence. The model is pre-trained with a specific maximum position, which fundamentally sets the hard limit for the context window. If a model was trained to understand positions up to 8,192, it cannot natively handle a token at position 8,193 without specialized techniques.

The Self-Attention Mechanism

The real work happens in the self-attention layers. For each token in the sequence, the attention mechanism calculates a set of scores that represent how relevant every other token in the context is to the current token. A token with a high attention score has a strong influence on the model's interpretation and processing of the current token. This allows the model to form connections between distant words, like linking a pronoun late in a document to the noun it refers to at the beginning.

This is not a sequential process. The model doesn't "read" from start to finish. Instead, it computes these relationships for all tokens in parallel, creating a matrix of attention scores across the entire context. This parallel processing is what makes Transformers so effective on modern hardware like GPUs.

The Quadratic Cost

The power of self-attention comes at a steep price. To calculate the attention for every token, the model must compare it with every other token. This results in a computational complexity that scales quadratically with the sequence length (n). This is often expressed as O(n²)^[6]. If you double the context window from 4,000 to 8,000 tokens, the computation required for the attention layers roughly quadruples. The memory required to store the attention matrix also scales quadratically. This scaling behavior is the primary technical and economic barrier to infinitely large context windows.

A Brief History of Window Expansion

The context window sizes common today are a very recent development. For years, the quadratic scaling of attention kept them small, forcing developers to use complex workarounds.

Early influential models operated with what now seem like tiny windows. OpenAI's GPT-2, released in 2019^[7], had a context window of 1,024 tokens. Its successor, GPT-3, introduced in 2020^[8], doubled this to 2,048 tokens. For a time, 4,096 tokens was considered a large and capable context size. These limits were a direct consequence of the computational and memory costs of the attention mechanism.

The demand for larger contexts was immense. Developers wanted to perform tasks on entire documents, codebases, or transcripts without having to chunk them into small pieces. In May 2023, Anthropic announced a major breakthrough with a 100,000-token context window for its Claude model. This crossed a significant threshold, allowing an entire novel like The Great Gatsby to fit into a single prompt.

This set off a race among major AI labs. OpenAI first announced GPT-4 Turbo^[9] on November 6, 2023. The generally available model, gpt-4-turbo-2024-04-09, was released on April 9, 2024. In early 2024, Anthropic followed with its Claude 3 model family^[10], offering a 200,000-token window. Shortly after, Google announced Gemini 1.5 Pro^[11], which pushed the boundary to an astonishing 1 million tokens, enabled by architectural innovations like a Mixture-of-Experts (MoE) design.

The Hidden Costs and Failure Modes of Large Contexts

While impressive, these massive context windows come with critical tradeoffs that every technical leader must evaluate. The costs are not just financial but also impact performance and reliability.

Financial Costs

Leading model providers price their services based on the number of tokens processed, with separate rates for input (context) and output (generation). A large context window directly translates to a large input token count, making each API^[12] call more expensive.

Consider analyzing a 100,000-word corporate report, which is approximately 133,000 tokens. Using Anthropic's most capable model, Claude 3 Opus, at a rate of $15 per million input tokens (as of April 2024), this single analysis would cost about $2.00. Using OpenAI's GPT-4 Turbo at $10 per million input tokens would cost about $1.33. While these numbers seem small, for an application processing hundreds or thousands of such documents daily, the costs accumulate rapidly. Using a large context when a smaller one would suffice is a direct and unnecessary expense.

Performance Costs: Latency

The O(n²) complexity of attention also manifests as latency. The time it takes for the model to begin generating a response (the time to first token) increases with the size of the input context. Even with significant hardware and software optimizations, processing a 200,000-token prompt is substantially slower than processing a 4,000-token prompt. For interactive applications like chatbots or co-pilots, this added latency can degrade the user experience, making the system feel sluggish and unresponsive.

Accuracy Degradation: The "Lost in the Middle" Problem

Perhaps the most subtle and dangerous failure mode of large context windows is their non-uniform recall ability. One might assume that a model with a 200,000-token window can access information anywhere within that context with equal fidelity. Research shows this is not true.

A 2023 study from Stanford University, "Lost in the Middle: How Language Models Use Long Contexts,"^[13] demonstrated that models exhibit a U-shaped performance curve. They are highly accurate at recalling and using information presented at the very beginning and the very end of the context window. However, their performance drops significantly when trying to access information located in the middle of a long context. The model can effectively "forget" or ignore facts buried deep within a document.

This has critical implications. If you feed a model a long compliance document and ask it to verify a specific clause located in the middle, the model is more likely to fail or "hallucinate" an answer than if that clause were at the start or end. Simply putting more data into the context is not a guarantee of better performance; the position of that data matters.

A Practical Example: Summarizing a Long Document

Imagine you have a 15,000-word transcript of an earnings call, which translates to about 20,000 tokens. Your goal is to extract key financial results and summarize the CEO's outlook on the next quarter.

Approach 1: The Naive Long-Context Approach

With a model like Claude 3 or GPT-4 Turbo, the simplest approach is to place the entire transcript into the prompt and ask your question. The implementation is straightforward, using a client library like OpenAI's Python^[14] SDK.

# This is a conceptual example. `transcript_text` is a 20,000-token string.

from openai import OpenAI
client = OpenAI()

prompt = f"""Here is the full transcript of the earnings call:

{transcript_text}

---

Please extract the exact figures for Q4 Revenue and Net Income.
Then, provide a bulleted summary of the CEO's forward-looking statements for the next quarter.
"""

response = client.chat.completions.create(
 model="gpt-4-turbo",
 messages=[{"role": "user", "content": prompt}]
)

print(response.choices[0].message.content)

• Pros: Simple to code and implement. Good for tasks requiring a holistic understanding of the entire document.
• Cons: Expensive due to the large number of input tokens. Higher latency. Vulnerable to the "lost in the middle" problem if a key figure is buried on page 37 of the transcript.

Approach 2: Retrieval-Augmented Generation (RAG)

An alternative architecture is Retrieval-Augmented Generation (RAG)^[15]. Instead of putting the entire document into the context window, you use a more targeted approach.

1. Index: First, you split the 20,000-token transcript into smaller, semantically relevant chunks (e.g., 500 tokens each). You then use an embedding model to generate a vector for each chunk and store these vectors in a vector database^[16].
2. Retrieve: When the user asks a question ("What was the Q4 Revenue?"), you embed the question itself into a vector.
3. Search: You perform a similarity search against the vector database to find the chunks of the transcript that are most relevant to the question.
4. Augment and Generate: You then construct a much smaller, more focused prompt containing only these relevant chunks and ask the LLM to answer the question based on that limited context.

# This conceptual example assumes `find_relevant_chunks` queries a vector DB.

question = "What was the Q4 Revenue?"
relevant_chunks = find_relevant_chunks(question, vector_database)

prompt = f"""Using only the following context from an earnings call, answer the user's question.

Context:
{relevant_chunks}

---

Question: {question}
"""

response = client.chat.completions.create(
 # We can use a smaller-context, cheaper model if we want
 model="gpt-4-turbo",
 messages=[{"role": "user", "content": prompt}]
)

• Pros: Much lower per-query cost and latency, as the context window is small. Higher accuracy for factual recall because it bypasses the "lost in the middle" problem by providing only relevant information.
• Cons: More complex infrastructure (requires a vector database and an indexing pipeline). Can fail if the retrieval step doesn't find the correct chunks of text.

The Decision Lens: When to Use a Large Context Window

As an engineer or technical founder, the choice between a large context window and a RAG architecture depends entirely on the job to be done. There is no single best solution.

Use a large context window when:

• The task requires holistic synthesis. For questions like "What is the main theme of this novel?" or "Compare and contrast the arguments in the first and last chapters," the model needs access to the entire text to understand the global structure and narrative flow.
• The input is unstructured and dense. If you are trying to refactor a single, large file of code, putting the whole file in context is more effective than trying to chunk it.
• Prototyping speed is paramount. The long-context approach is far simpler to implement initially. It's an excellent way to quickly test an idea before investing in more complex infrastructure.
• Cost and latency are not primary constraints. For offline batch processing jobs, like generating nightly reports on a set of documents, the higher latency and cost may be acceptable.

Consider alternatives (like RAG or fine-tuning^[17]) when:

• The application is for question-answering over a large corpus. RAG is purpose-built for finding specific facts in a sea of information. It is more scalable, cheaper, and often more accurate for this use case.
• Low latency is a product requirement. Interactive chatbots and assistants must respond quickly. RAG's small, targeted prompts enable much faster response times.
• Per-query cost must be minimized. For any application at scale, the cost savings from RAG's smaller context windows are substantial.
• Factual accuracy is non-negotiable. By retrieving the exact source of information and placing it into a small context, RAG minimizes the risk of the "lost in the middle" problem and reduces hallucinations.

Correcting Common Misconceptions

As context windows have grown, so have misunderstandings about how they function.

Misconception 1: "A 1M token context is like 1M tokens of RAM."
This analogy is misleading. RAM offers fast, constant-time random access. An LLM's context window is a sequentially processed buffer with non-uniform access. Information in the middle is harder to retrieve, and the processing cost scales quadratically, not constantly.

Misconception 2: "Bigger is always better."
Bigger is slower, more expensive, and can be less accurate for targeted retrieval tasks. The optimal context size is the smallest one that reliably accomplishes the task. Using a 1M token window for a 500-token job is wasteful and inefficient.

Misconception 3: "The model 'reads' the context like a human."
A human reads sequentially, building a mental model over time. A Transformer processes the entire context in a massively parallel computation, calculating the relationships between all tokens at once. This is a fundamentally different mechanism that explains both its power and its quirks, like the importance of positional encodings.

The State of Context in 2024

This claim, written from a future perspective, is mostly correct but contains inaccuracies. Google did announce and release a 2 million token context window for Gemini 1.5 Pro, making it generally available to all developers in June 2024. It is also true that specific, older versions of Gemini 1.5 Pro models were scheduled for retirement. For example, the gemini-1.5-pro-001 version was scheduled for shutdown in May 2025^[18]. However, the phrasing "versions of Gemini 1.5 Pro have been retired" as a general statement about the model line by early 2025 is misleading, as the overall Gemini 1.5 Pro model and its newer versions remain available and are not retired as a whole.

These advances are not coming from brute-force computation alone. They rely on new techniques that attack the O(n²) bottleneck of the original Transformer. Methods like Sparse Attention^[19], which calculates scores for only a subset of tokens, and architectural patterns like Mixture of Experts (MoE)^[20], which only activates parts of the model for a given input, reduce the computational load. Other research, like Ring Attention^[21], explores ways to distribute the processing of a single, massive context across multiple accelerator chips.

While windows will likely continue to expand, the fundamental tradeoffs will remain. The engineering challenge is not to chase the largest number, but to understand the relationship between context size, cost, latency, and accuracy. The most effective AI systems will be those built with a deliberate and informed strategy, using the right tool and the right context size for the specific problem at hand.