Processing 200,000 tokens by means of a big language mannequin is pricey and gradual: the longer the context, the quicker the prices spiral. Researchers at Tsinghua College and Z.ai have constructed a method known as IndexCache that cuts as much as 75% of the redundant computation in sparse consideration fashions, delivering as much as 1.82x quicker time-to-first-token and 1.48x quicker era throughput at that context size.
The approach applies to fashions utilizing the DeepSeek Sparse Consideration structure, together with the newest DeepSeek and GLM households. It could actually assist enterprises present quicker consumer experiences for production-scale, long-context fashions, a functionality already confirmed in preliminary assessments on the 744-billion-parameter GLM-5 mannequin.
The DSA bottleneck
Giant language fashions depend on the self-attention mechanism, a course of the place the mannequin computes the connection between each token in its context and all of the previous ones to foretell the subsequent token.
Nonetheless, self-attention has a extreme limitation. Its computational complexity scales quadratically with sequence size. For purposes requiring prolonged context home windows (e.g., giant doc processing, multi-step agentic workflows, or lengthy chain-of-thought reasoning), this quadratic scaling results in sluggish inference speeds and vital compute and reminiscence prices.
Sparse consideration provides a principled resolution to this scaling drawback. As a substitute of calculating the connection between each token and all previous ones, sparse consideration optimizes the method by having every question choose and attend to solely essentially the most related subset of tokens.
DeepSeek Sparse Consideration (DSA) is a extremely environment friendly implementation of this idea, first launched in DeepSeek-V3.2. To find out which tokens matter most, DSA introduces a light-weight "lightning indexer module" at each layer of the mannequin. This indexer scores all previous tokens and selects a small batch for the principle core consideration mechanism to course of. By doing this, DSA slashes the heavy core consideration computation from quadratic to linear, dramatically rushing up the mannequin whereas preserving output high quality.
However the researchers recognized a lingering flaw: the DSA indexer itself nonetheless operates at a quadratic complexity at each single layer. Though the indexer is computationally cheaper than the principle consideration course of, as context lengths develop, the time the mannequin spends operating these indexers skyrockets. This severely slows down the mannequin, particularly through the preliminary "prefill" stage the place the immediate is first processed.
Caching consideration with IndexCache
To unravel the indexer bottleneck, the analysis workforce found a vital attribute of how DSA fashions course of information. The subset of essential tokens an indexer selects stays remarkably steady as information strikes by means of consecutive transformer layers. Empirical assessments on DSA fashions revealed that adjoining layers share between 70% and 100% of their chosen tokens.
To capitalize on this cross-layer redundancy, the researchers developed IndexCache. The approach partitions the mannequin’s layers into two classes. A small variety of full (F) layers retain their indexers, actively scoring the tokens and selecting an important ones to cache. The remainder of the layers turn into shared (S), performing no indexing and reusing the cached indices from the closest previous F layer.
Throughout inference, the mannequin merely checks the layer kind. If it reaches an F layer, it calculates and caches contemporary indices. Whether it is an S layer, it skips the maths and copies the cached information.
There’s a variety of optimization strategies that attempt to tackle the eye bottleneck by compressing the KV cache, the place the computed consideration values are saved. As a substitute of shrinking the reminiscence footprint like customary KV cache compression, IndexCache assaults the compute bottleneck.
“IndexCache shouldn’t be a standard KV cache compression or sharing approach,” Yushi Bai, co-author of the paper, instructed VentureBeat. “It eliminates this redundancy by reusing indices throughout layers, thereby lowering computation slightly than simply reminiscence footprint. It’s complementary to present approaches and might be mixed with them.”
The researchers developed two deployment approaches for IndexCache. (It’s price noting that IndexCache solely applies to fashions that use the DSA structure, similar to the newest DeepSeek fashions and the newest household of GLM fashions.)
For builders working with off-the-shelf DSA fashions the place retraining is unfeasible or too costly, they created a training-free methodology counting on a “grasping layer choice” algorithm. By operating a small calibration dataset by means of the mannequin, this algorithm mechanically determines the optimum placement of F and S layers with none weight updates. Empirical proof exhibits that the grasping algorithm can safely take away 75% of the indexers whereas matching the downstream efficiency of the unique mannequin.
For groups pre-training or closely fine-tuning their very own basis fashions, the researchers suggest a training-aware model that optimizes the community parameters to natively help cross-layer sharing. This method introduces a “multi-layer distillation loss” throughout coaching. It forces every retained indexer to discover ways to choose a consensus subset of tokens that might be extremely related for all the next layers it serves.
Actual-world speedups on manufacturing fashions
To check the affect of IndexCache, the researchers utilized it to the 30-billion-parameter GLM-4.7 Flash mannequin and in contrast it towards the usual baseline.
At a 200K context size, eradicating 75% of the indexers slashed the prefill latency from 19.5 seconds down to only 10.7 seconds, delivering a 1.82x speedup. The researchers notice these speedups are anticipated to be even better in longer contexts.
In the course of the decoding section, the place the mannequin generates its response, IndexCache boosted per-request throughput from 58 tokens per second to 86 tokens per second on the 200K context mark, yielding a 1.48x speedup. When the server's reminiscence is absolutely saturated with requests, whole decode throughput jumped by as much as 51%.
For enterprise groups, these effectivity good points translate instantly into price financial savings. “By way of ROI, IndexCache gives constant advantages throughout eventualities, however the good points are most noticeable in long-context workloads similar to RAG, doc evaluation, and agentic pipelines,” Bai mentioned. “In these instances, we observe not less than an approximate 20% discount in deployment price and comparable enhancements in user-perceived latency.” He added that for very short-context duties, the advantages hover round 5%.
Remarkably, these effectivity good points didn’t compromise reasoning capabilities. Utilizing the training-free method to remove 75% of indexers, the 30B mannequin matched the unique baseline's common rating on long-context benchmarks, scoring 49.9 towards the unique 50.2. On the extremely advanced AIME 2025 math reasoning benchmark, the optimized mannequin really outperformed the unique baseline, scoring 92.6 in comparison with 91.0.
The workforce additionally ran preliminary experiments on the production-scale 744-billion-parameter GLM-5 mannequin. They discovered that eliminating 75% of its indexers with the training-free methodology yielded not less than a 1.3x speedup on contexts over 100K tokens. On the similar time, the mannequin maintained an almost an identical high quality common on long-context duties.
Getting IndexCache into manufacturing
For improvement groups desirous to implement the training-free method as we speak, the method is simple however requires cautious setup. Whereas the grasping search algorithm mechanically finds the optimum layer configuration, the standard of that configuration is determined by the info it processes.
“We suggest utilizing domain-specific information as a calibration set in order that the found layer-sharing sample aligns with actual workloads,” Bai mentioned.
As soon as calibrated, the optimization is extremely accessible for manufacturing environments. Open-source patches are already accessible on GitHub for main serving engines. “Integration is comparatively easy — builders can apply the patch to present inference stacks, similar to vLLM or SGLang, and allow IndexCache with minimal configuration modifications,” Bai mentioned.
Whereas IndexCache gives a direct repair for as we speak’s compute bottlenecks, its underlying philosophy factors to a broader shift in how the AI business will method mannequin design.
“Future basis fashions will doubtless be architected with downstream inference constraints in thoughts from the start,” Bai concluded. “This implies designs that aren’t solely scalable when it comes to mannequin measurement, but additionally optimized for real-world throughput and latency, slightly than treating these as post-hoc issues.”
