In 2025, talking about factorization or distillation of models sounded like computing archaeology. In 2026 both come up in our product meetings at 498 Advance. When tasks move to local models, somebody has to decide what the provider used to decide: the temperature, the embedding, the reranker, and how much context window a task needs and how to get it. To me that is good news. We are taking the reins of our tools back.
The setup
A model was pretrained on sequences of 4,096 tokens (a token is a fragment of a word) and uses rotary embeddings (RoPE). We hand it a 32,000-token document. Perplexity looks normal for the first 4,000 tokens and then shoots up. We have budget for about 1,000 fine-tuning steps.
RoPE rotates each pair of query and key dimensions by an angle proportional to the token's position. In an attention head of 128 dimensions with base 10,000, the slowest pair only reached 27° during training. At 32,000 tokens it would reach 212°. Those are angles the model has never seen.
What works
There are four options: fine-tune on long documents and leave the positions alone, scale the positions, swap RoPE for ALiBi, or limit attention to a 4,000-token sliding window.
- Fine-tuning without touching the positions is not enough. On LLaMA 7B, more than 10,000 steps moved the effective window from 2,048 to just 2,560 tokens (Chen et al., Meta, 2023).
- Scaling the positions works. Each index is multiplied by 4,096/32,000, which is 0.128, so token 32,000 is read as if it sat at position 4,096. In the same paper, interpolation reached 8,192 in 200 steps.
- ALiBi is chosen before training. Swapping it into a model pretrained with RoPE throws away what it learned about positions. That is our reading, not a measured result.
- A sliding window stays on familiar ground but gives no direct access to what falls outside it. Also our reading.
After scaling, the model has to relearn to tell neighboring tokens apart, and that is what a short fine-tune of about 1,000 steps is for. A "fine-tuning step" is a small correction of the model's internal values, computed from a batch of examples: each step nudges all of its values a little. A thousand steps is little next to training a model from scratch, and Meta's paper calls that cost negligible.
Scaling is not free. Full attention costs 61 times more per layer, and the key-value cache of a typical 7B model goes from 2 GiB to 15.6 GiB (our calculation).
How to test that it uses the whole text
Fitting in the window proves nothing. Hide the answer far from the question and check that the model finds it: a needle-in-a-haystack grid by length and depth, with distractors, plus tasks harder than a needle. In RULER (Hsieh et al., 2024), 17 models advertised 32,000 tokens or more, and only 10 held up satisfactorily at 32,000.
In GEORadar's analytics module
In the analytics module of GEORadar we had to redefine the context window. In an illustrative example (40 tokens per question, 500 per answer), a ten-interaction conversation takes about 5,400 tokens. It does not fit in 4,096 but it fits in 8,192. The window follows from the unit of analysis.
The full article, with six diagrams, calculations and sources, is on 498advance.com.