Authors
Sidi Lu, Jacky Dai, Xuezhe Ma, and Nanyun Peng
Overview
InsNeXt modernizes insertion-based language modeling for scalable pretraining. It combines sentence-level and document-level training, current language-model system components, and a context-encoding mechanism designed specifically for insertion-based decoding.
At inference time, sparse bidirectional re-encoding lets the model use both sides of an insertion context while preserving a computational profile comparable to conventional autoregressive decoding.
Results
The work scales InsNeXt from 154M to 573M parameters with a 4,096-token context window. Evaluations cover representation learning, commonsense reasoning, and controllable generation; the resulting models perform comparably to or better than similarly sized autoregressive baselines while retaining insertion-based controllability.
Paper
Status
Accepted at COLM 2026 Workshop.
Citation
Sidi Lu, Jacky Dai, Xuezhe Ma, and Nanyun Peng. “InsNeXt: Training Scalable Insertion-based Language Models from Scratch.” COLM 2026 Workshop, 2026.