- A 1M-token context window makes many common 4K-era LLM architectures (chunking, heavy summarization chains, default RAG) not just outdated but fundamentally misaligned with what models can now ingest.
- RAG becomes optional for many real-world corpora under roughly 750K words, eliminating retrieval and chunking errors for use cases like policy docs, contract analysis, and codebase understanding.
- Prompting shifts from crafting clever instructions to designing robust input schemas and context organization, with attention to token budgets, ordering effects, and cost control.
- Latency and cost become first-class architectural constraints at 1M tokens, forcing explicit tradeoffs between “send everything” and more selective preprocessing or retrieval.
- Prompt caching changes how you should structure prompts: stable, reusable context should come first, with variable user-specific content appended to maximize cache hits and reduce repeated compute.
OpenAI shipped GPT-5.4 on March 5 with a 1M token context window, and the reaction was predictable. Benchmarks got shared. Demo videos circulated. People stuffed entire codebases into prompts and posted the results. What almost nobody talked about was the uncomfortable implication: if you can send a million tokens in a single request, most of the architectural patterns you've been using are wrong.
Not outdated. Wrong.
The 4K Hangover
Most production LLM applications were designed when 4K tokens was the ceiling. Even teams that updated for 32K or 128K contexts kept the same fundamental patterns. Chunking strategies, retrieval pipelines, summarization chains, all of it exists because the model couldn't see enough at once.
RAG became the default architecture not because it was elegant, but because it was necessary. You couldn't fit the full document set into context, so you built retrieval layers to find the right chunks and hope they contained enough signal. The entire vector database ecosystem exists as a workaround for context limitations.
With 1M tokens, you can fit roughly 750,000 words into a single prompt. That's the entire Harry Potter series. Twice. Or a mid-sized company's complete policy documentation. Or six months of customer support transcripts.
The workaround just became optional.
What Actually Changes
This isn't about doing the same things with more text. A million-token context window changes the categories of problems you can solve in a single pass.
RAG Gets Demoted
RAG pipelines introduce retrieval error at every step. Your chunking strategy might split a critical paragraph across two chunks. Your embedding model might not surface the most relevant section. Your reranker might deprioritize exactly the context the model needed.
When you can fit the entire corpus into context, you eliminate retrieval error completely for document sets under ~750K words. That covers a surprising number of production use cases: legal contract analysis, compliance checking, codebase understanding, internal knowledge bases.
RAG doesn't disappear. You still need it for genuinely massive datasets. But for the workloads where teams spent months tuning chunk sizes and overlap parameters, the answer might now be: just send everything.
Prompt Engineering Becomes System Design
At 4K tokens, a prompt is a carefully crafted instruction. At 1M tokens, a prompt is a data pipeline. You're not writing prompts anymore, you're designing input schemas that might include thousands of documents, structured metadata, and complex instruction sets.
This means prompt engineering stops being a writing skill and starts being an engineering discipline. You need to think about token budgets, context organization, priority ordering (models still attend differently to content at the beginning versus the middle), and cost management.
A single 1M token request to GPT-5.4 isn't cheap. The economics of "just send everything" only work if you're thoughtful about when that approach actually beats a well-tuned retrieval pipeline.
Latency Math Changes
A million tokens takes time to process. Even with the inference speed improvements in GPT-5.4, you're looking at meaningfully longer time-to-first-token and total generation time compared to a focused 8K context request.
For interactive applications, this matters. A customer support bot that processes six months of conversation history on every message will feel slow. The architecture question becomes: when do you pay the latency cost of full context, and when do you pre-process into a shorter representation?
Caching helps. OpenAI's prompt caching means repeated prefixes don't get reprocessed. But you have to design your prompt structure to take advantage of that, putting stable context first and variable content last. That's an architectural decision, not a prompt tweak.
The Patterns That Emerge
Teams that have started building for 1M contexts are converging on a few approaches.
Tiered context loading. Not everything goes into every request. You maintain context tiers: always-included system context, session-level context that persists across a conversation, and request-specific context pulled in for individual queries. The architecture looks less like RAG and more like memory management in an operating system.
Pre-computation over retrieval. Instead of retrieving relevant chunks at query time, you pre-compute comprehensive summaries and structured extractions during ingestion. The model processes the full corpus once, produces condensed representations, and those representations serve subsequent requests. You trade ingestion-time compute for query-time speed.
Hybrid architectures. The pragmatic answer for most teams is using full context for high-value, low-frequency tasks (deep analysis, comprehensive review, complex reasoning) and keeping lightweight retrieval for high-frequency, latency-sensitive operations. One architecture doesn't fit all request types.
The Real Bottleneck Moved
The limiting factor in LLM applications used to be context size. Now it's everything else: cost management, latency budgets, prompt structure, caching strategy, and knowing when full context actually improves output quality versus when it just adds noise.
More context isn't automatically better. Models can get distracted by irrelevant information in long contexts. The "lost in the middle" problem, where models underweight information in the center of long prompts, hasn't been fully solved even at the architecture level.
The teams that will build the best applications on 1M context windows aren't the ones who stuff everything in. They're the ones who understand when to use the full window, when to use retrieval, and when to use something in between. That's not a model capability question. That's an architecture question.
And most teams haven't started asking it yet.
Audit your current LLM app for “context workarounds” you added only because of small windows—chunking rules, aggressive summarization, and complex retrieval layers—and identify where a full-context pass would be simpler and more reliable. Then redesign prompts like data pipelines: structure and order inputs for attention and caching, and be deliberate about when you pay the latency and cost of a million-token request versus using retrieval or preprocessing for interactive flows.