Boa
Linear-Time Context Compression with State-Space Encoding
Computer Vision Center, Universitat Autònoma de Barcelona
Boa is a context compressor for a frozen language model. A Mamba-2 encoder reads the context in one linear-time pass and keeps a short sequence of continuous representations, which the model reads in place of the text.
This figure needs the page to be served over HTTP.
What context compression does
A language model normally reads every token of its context, and that work grows with the length of the context. Context compression replaces the text with a shorter sequence of continuous vectors, often called soft tokens, which the unchanged model reads instead. Boa keeps one vector for every r tokens, counting back from the final token, so a context of L tokens becomes ⌈L / r⌉ soft tokens.
This figure needs the page to be served over HTTP.
Time and memory
Compression pays off for a single query only if producing the compressed context costs less than reading the full one. Boa’s encoder runs in linear time, so its own cost stays small as the context grows, whereas Transformer-based compressors first make a long pass of their own. Choose a context length to compare Boa with the same LLM reading the full context.
This figure needs the page to be served over HTTP.
How Boa works
Boa has two trained parts, a Mamba-2 encoder and a projection, and one frozen part, the target LLM, which is never modified. Rather than entering at the input, the soft tokens are projected into keys and values in the LLM’s attention cache at every layer.
This figure needs the page to be served over HTTP.
- Mamba-2 reads the context once, in linear time.
- A temporal stride keeps every r-th hidden state.
- Per-layer K and V projections turn them into a key-value cache.
- The frozen LLM attends to that cache instead of the text.
- Training matches its next-token distribution to the full-context teacher (KL).
Results
On the in-domain MRQA question-answering datasets, Boa scores above Activation Beacon and SAC at every compression ratio tested, and loses little as the ratio grows. Out of domain the three methods are much closer, and which one leads depends on the ratio.
This figure needs the page to be served over HTTP.
Average exact match and F1
Averages over the datasets in each split, with a frozen Qwen3 decoder. The best compressed score in each column is in bold.
Limitations
- Verbatim span extraction is weaker.
- Relation Extraction asks the model to copy exact spans of text. This is the setting in which attention-based Transformers exceed state-space models, and here Activation Beacon leads Boa at every compression ratio.
- Long-context question answering trails Activation Beacon.
- At 32K tokens and 8× and 16× compression, Boa averages 1.3 to 1.4 F1 below Activation Beacon, and the gap is concentrated in question answering: on QASPER Boa trails by 5.4 to 6.4 F1. On summarisation the two are close: Boa leads on GovReport by 1.9 to 4.0 F1 and trails on BookSum by 0.8 to 1.4. We attribute the gap to a weaker long-context prior, and offer this as a hypothesis rather than a measured effect. The two encoder backbones start from very different pretraining. SAC and Activation Beacon encode the context with Qwen3 itself, which was pretrained on roughly 36 trillion tokens and includes a dedicated long-context stage with sequences of up to 32K tokens. Boa's encoder starts from a Mamba-2 checkpoint pretrained on roughly 300 billion tokens, about a hundred times fewer, in sequences of at most 2K tokens; our own compression pretraining adds 10 billion tokens in sequences of up to 10K. A 32K-token document is therefore well beyond anything the Mamba-2 encoder saw before fine-tuning, while it is within what Qwen3 was trained on, and Activation Beacon also works directly on full-attention windows across the extended context. A state-space encoder pretrained at a comparable scale and context length is the natural next step.
Citation
@article{serraortega2026boa,
title = {Linear-Time Context Compression with State-Space Encoding},
author = {Serra Ortega, Marc and Barsky, Andrey
and Boned Riera, Carlos and Ockier, Tom{\'a}s
and Karatzas, Dimosthenis},
journal = {arXiv preprint arXiv:2610.33006},
year = {2026},
url = {https://arxiv.org/abs/2610.33006}
}