The race to build artificial intelligence with ever-larger context windows has produced an awkward gap between capacity and comprehension. A model may accept an enormous block of software, yet still lose track of how one function calls another across files, packages, and repositories. New research from Germany argues that the training data—not only the number of available tokens—is part of the problem.
OctoLong, a paper posted August 5 by researchers at TU Darmstadt and the University of Würzburg, introduces a pipeline for constructing long training examples from real software dependencies. Instead of filling a context window with a single repository or loosely related documents, the system follows function and class references into external packages, then serializes the resulting dependency tree.
The paper’s strongest idea is that a useful long context should preserve relationships, not merely occupy space. For coding systems, that means training on the routes software actually follows when one component imports and invokes another.
Following code beyond one repository
The researchers began with actively maintained Python repositories and retained about 12,000 that declared dependencies and could be installed. They identified roughly 92,000 seed functions or classes that passed code-quality filters. Each seed was placed in a container with its repository and dependencies, allowing a language server to resolve definitions using the same package information that supports modern development tools.
OctoLong then traversed those references breadth-first. It stopped after ten dependency steps or 128,000 tokens and rejected shallow or unreliable traces. The process produced about 6.2 billion tokens of cross-repository code. Those examples became roughly 12 percent of a 50-billion-token mixture used to extend the context length of Qwen3-based models from 32,000 to 128,000 tokens. A separate instruction-tuning stage used about 10 billion tokens.
This is a different strategy from treating a large repository as one long book. A repository boundary is administrative; a dependency boundary is functional. Real applications routinely hand work to libraries maintained elsewhere, so stopping at the first boundary can omit the implementation that explains what the code will do.
What the reported tests show
The team trained models ranging from 600 million to 14 billion parameters and compared them with 18 open-weight long-context systems. In the eight-billion-parameter ablation, removing the cross-repository examples reduced the reported score on RepoQA from 64.63 to 61.58. The same removal lowered the general LongBench V2 average from 39.17 to 37.03 and the BFCL tool-use average from 61.65 to 57.74.
Those differences are more informative than a simple leaderboard win because the ablation holds most of the training recipe constant. They support a narrower claim: dependency-rich examples contributed measurable value inside this particular model family and evaluation setup. They do not establish that 128,000 tokens are always necessary, or that the same gains will appear in proprietary frontier systems.
The result changes the long-context question from “How much can the model hold?” to “What relationships was it trained to preserve?” A window can be technically large while the model’s effective understanding still decays when relevant evidence is scattered or surrounded by noise.
A broader repository-reasoning problem
That interpretation is reinforced by separate primary research. The RepoQA benchmark was designed to require models to retrieve functions from natural-language descriptions across 50 repositories and five programming languages, making simple keyword matching less useful. A newer study called RepoReasoner, posted in late July, tests stateful execution and call-chain prediction across files. Its authors reported that even the best model in their evaluation reached 69.1 percent on output prediction with the correct context supplied, while call-chain predictions tended to have high precision but low recall.
Read beside RepoReasoner, OctoLong looks less like a bid for a bigger memory number and more like an attempt to teach models the architecture inside that memory. Both studies point to dependency structure and distracting context as central obstacles in repository-level AI.
What remains unproven
OctoLong is a new preprint, and its findings have not yet been independently replicated. The collection pipeline is limited to Python, while the trained models top out at 14 billion parameters and 128,000 tokens. The authors also omit low-resource languages and acknowledge that their language-server approach cannot reliably follow dependencies that cross programming-language boundaries through foreign-function or application-binary interfaces.
The evaluation is broad, but it still combines many design choices: context extension, instruction tuning, checkpoint merging, and a custom data mixture. The ablations help isolate the cross-repository contribution, yet production coding agents must also edit safely, run tests, recover from failed plans, and work with private code that may differ sharply from public repositories.
That makes OctoLong a useful test of a training principle, not proof that long-context coding is solved. If the result holds across languages, larger models, and independent evaluations, the next advance in AI memory may come from making context more structurally faithful rather than simply making it longer.
Sources: OctoLong research paper; RepoQA benchmark paper; RepoReasoner research paper; LongBench V2 benchmark documentation.
Featured image: Computer code displayed on a monitor. Photo: Markus Spiske via Wikimedia Commons. CC0 1.0 public domain dedication. Center-cropped from the original to 16:9 and resized to 1,920 × 1,080 pixels; no generative or substantive alteration.
