Giant context windows are a dead end, Nvidia and USC research argues

Two independent studies published on arXiv reach the same uncomfortable conclusion for the AI industry: the context window — a model's working memory — is, in many cases, far too expensive for what it delivers. The way out, the researchers say, is simpler than it sounds: summarize what matters and search for the rest when needed.

The Nvidia study: summarize instead of stockpiling

The first study comes from NVIDIA Research. In the technical report “You don’t need long context”, published on November 3, a 12-researcher team led by Peter Belcak and Pavlo Molchanov shows that generating tokens at the end of a long window can cost up to 7 times more than at the start, on frontier open-weight models. In tests with three models on H100 GPUs, a 256,000-token window cost 3.6 to 7 times more per generated token than a 16,000-token window.

More surprising: shortening the context did not hurt accuracy — in some cases, it improved it. According to the report, with context capped at 16,000 tokens and periodic summaries, Claude Opus 4.5 jumped from 65.0% to 84.0% on the AA-LCR benchmark. On SWE-bench Verified, the model scored 70.2% with one action per context, versus 70.0% with the full history. The team proposes a new class of models, “semantic state space models”, which carry a plain-language summary of the task state between turns instead of the whole history. The report, however, does not train such a model — it is a research direction, not a product. Coverage by The Decoder, published on November 21, sums up the thesis: periodic summaries and targeted lookups worked about as well as full context in many tests.

GrepMem: a plain-text log and a grep

The second study, from the University of Southern California, points the same way — with even stronger numbers. GrepMem, published on November 18 by Ziyan Yang, Taiwei Shi and Jieyu Zhao, swaps structured memory for something deliberately simple: a plain-text log holding the agent's full history plus a regular-expression search tool, grep_history. No embeddings, no retrieval index, no extra model.

Trained with Qwen3.5-9B using reinforcement learning, GrepMem reached 95% to 97% accuracy on ContextBench, versus at most 21% for lossy strategies such as summarization and masking. On BCP-Multi, it beat SUPO's summarization by 8.8 percentage points and MemGPT by 7.0 points — while using, on ContextBench, only 29% of MemGPT's tokens.

The contrast: the market still bets on giant context

There is a curious contrast with the rest of the market: while these studies question the giant context window, labs keep launching 1-million-token models — such as Tencent's Hy4. The window grew from 8,192 tokens in GPT-4, in March 2023, to around 1 million in GPT 5.6 Sol. Neither study says long context is useless: tasks that require cross-checking details scattered through a long document may still need it. What they show is that, for AI agents and for many long documents, summarization and search deliver nearly the same — at a fraction of the cost.

Both papers are arXiv preprints, meaning they have not yet been peer-reviewed, and the results are those reported by the authors themselves.

Featured image: NVIDIA headquarters in Santa Clara, California. Photo by Coolcaesar, licensed under CC BY-SA 4.0, via Wikimedia Commons.

Sources: arXiv:2611.05306 — “You don’t need long context” (NVIDIA Research); arXiv:2611.25123 — “Forget Freely, Grep Later” (USC); The Decoder.

Transparency note: this article was produced with the assistance of artificial intelligence, cross-checking public posts on X and sources on the internet, and reviewed before publication. Check the original sources to confirm the information.

Por Muse