BitDance: Scaling Autoregressive Generative Models with Binary Tokens

Abstract

For visual generation, discrete autoregressive models often struggle with poor tokenizer reconstruction, difficulties in sampling from large vocabularies, and slow token-by-token generation speeds. We present BitDance, which addresses these challenges via a large-vocabulary binary tokenizer, a binary diffusion head for sampling in large discrete space, and a next-patch diffusion paradigm that enables efficient multitoken prediction. BitDance is an open-source discrete autoregressive foundation model with 14B parameters, trained on large-scale multimodal tokens. While maintaining the standard language modeling paradigm for text tokens, BitDance employs a next-patch diffusion paradigm for visual tokens to predict multiple tokens in parallel—up to 64 per step. This unified multimodal framework is simple, scalable, and capable of efficiently generating high-resolution, photorealistic images.

Method

BitDance unifies a binary visual tokenizer, a diffusion-based sampling head, and a next-patch prediction strategy into a single autoregressive pipeline. Images are encoded into compact binary latents, flattened into patch-wise sequences, and decoded with joint multi-token sampling to balance quality and speed at high resolution.

The design targets three bottlenecks in autoregressive vision models: limited token expressiveness, sampling instability in large vocabularies, and slow decoding. BitDance addresses these with a 2^256 binary latent space, a diffusion head that models bit correlations directly, and patch-wise parallel prediction that preserves spatial dependencies while accelerating generation.

Binary Visual Tokenizer

BitDance scales token entropy via group-wise lookup-free quantization. By partitioning channels for entropy calculation, the tokenizer can grow to a 2^256 vocabulary while staying memory efficient and preserving reconstruction fidelity comparable to continuous VAEs. Tokens are learned with entropy regularization to prevent codebook collapse, and the grouping strategy keeps training tractable even as the vocabulary grows exponentially. This yields compact sequences with strong reconstruction quality, reducing error accumulation in long autoregressive chains. The reconstruction results can be found in the below table.

Binary Diffusion Head

Instead of predicting a categorical index, BitDance embeds binary tokens on a continuous hypercube and trains a diffusion head with a velocity-matching objective. This models all bits jointly without exponential classifier parameters and enables high-precision sampling in massive discrete spaces.

The head predicts clean binary latents from noisy inputs and applies a hard sign projection at the end of sampling. This preserves discrete structure while avoiding the independence assumptions and parameter blowups of conventional classification heads.

Next-Patch Diffusion

For faster decoding, BitDance predicts a patch of tokens in parallel using a block-wise causal mask that preserves dependencies within a patch. A lightweight DiT-style head aligns the training objective with joint multi-token sampling, improving both quality and throughput. Patch-wise raster ordering maintains global autoregressive structure while enabling intra-patch visibility. This closes the training-inference gap seen in parallel AR methods that sample tokens independently at inference time.

Results

Citation

@article{ai2026bitdance,
  title   = {BitDance: Scaling Autoregressive Generative Models with Binary Tokens},
  author  = {Ai, Yuang and Han, Jiaming and Zhuang, Shaobin and Hu, Xuefeng and Yang, Ziyan and Yang, Zhenheng and Huang, Huaibo and Yue, Xiangyu and Chen, Hao},
  journal = {arXiv preprint arXiv:xxxx.xxxx},
  year    = {2026}
}