Context Compression for Auto-regressive Transformers with Sentinel Tokens (original) (raw)

View PDF

Abstract:The quadratic complexity of the attention module makes it gradually become the bulk of compute in Transformer-based LLMs during generation. Moreover, the excessive key-value cache that arises when dealing with long inputs also brings severe issues on memory footprint and inference latency. In this work, we propose a plug-and-play approach that is able to incrementally compress the intermediate activation of a specified span of tokens into compact ones, thereby reducing both memory and computational cost when processing subsequent context. Experiments on both in-domain language modeling and zero-shot open-ended document generation demonstrate the advantage of our approach over sparse attention baselines in terms of fluency, n-gram matching, and semantic similarity. At last, we comprehensively profile the benefit of context compression on improving the system throughout. Code is available at this https URL.

Submission history

From: Siyu Ren [view email]
[v1] Thu, 12 Oct 2023 09🔞19 UTC (271 KB)
[v2] Sun, 15 Oct 2023 09:15:02 UTC (271 KB)