QwenVL-SAMTok: Pixel-wise MLLM Integration

• The paper introduces QwenVL-SAMTok’s innovative approach of tokenizing pixel masks into discrete tokens, seamlessly integrating them into language modeling.
• It employs a residual vector quantization that converts continuous mask embeddings into token pairs, enabling autoregressive training without architectural modifications.
• The system achieves state-of-the-art performance on segmentation, interactive dialogue, and region captioning tasks using both language modeling and reinforcement learning objectives.

QwenVL-SAMTok is a multimodal LLM (MLLM) system that incorporates pixel-wise visual reasoning using the SAMTok discrete mask tokenizer. By converting arbitrary region segmentation masks into two discrete tokens and treating these as additional words in the language modeling vocabulary, QwenVL-SAMTok enables standard language modeling and reinforcement learning objectives to encompass pixel-level understanding and segmentation tasks without architectural modifications or specialized decoders. This paradigm achieves state-of-the-art or competitive performance across region-level understanding, interactive segmentation, referring expression segmentation, and multimodal dialogue benchmarks, demonstrating a scalable approach for integrating dense visual information into autoregressive transformers (Zhou et al., 22 Jan 2026).

1. Discrete Mask Tokenization Mechanism

SAMTok forms the core of QwenVL-SAMTok’s capacity for pixel-wise region understanding. The process proceeds as follows:

• Mask Encoder: The encoder $f_{\mathrm{enc}}$ operates on an image $\mathcal I$ and a binary mask $\mathcal M \in \{0,1\}^{H \times W}$, leveraging SAM’s image backbone $f_{\mathrm{img}}$ and prompt encoder $f_{\mathrm{prm}}$. The result is a continuous embedding $$\mathbf z = f_{\mathrm{msk}}(f_{\mathrm{img}}(\mathcal I), f_{\mathrm{prm}}(\mathcal M)) \in \mathbb R^d$$.
• Residual Vector Quantization: The embedding $\mathbf z$ is quantized in two stages. First, $$\mathbf e_1 = \arg\min_{\mathbf e \in \mathcal C} \|\mathbf z - \mathbf e\|_2^2$$ is selected from the shared codebook $\mathcal C$ of size $K$. The residual $\mathbf r_1 = \mathbf z - \mathbf e_1$ is then further quantized as $$\mathbf e_2 = \arg\min_{\mathbf e \in \mathcal C} \|\mathbf r_1 - \mathbf e\|_2^2$$, producing a discrete code pair $[\mathbf e_1, \mathbf e_2]$ (Equation (2)). Each quantized code corresponds uniquely to a new vocabulary token.
• Mask Reconstruction: Decoding is performed using a modified SAM model. Given the image and the two code embeddings, the reconstructed mask is obtained as $$\hat{\mathcal M} = f_{\mathrm{dec}}(\mathcal I, [\mathbf e_1, \mathbf e_2])$$.
• Training Objective: The loss function is a sum of pixel-wise cross-entropy, Dice loss for overlap, and a codebook commitment loss, ensuring the embedding remains close to the quantized representation (Equation (4)).

This architecture allows any binary segmentation mask to be losslessly mapped to a pair of “words” and subsequently reconstructed, facilitating seamless integration with token-based LLMs.

2. Mask-Token Driven MLLM Training

By encoding all region masks into token pairs, QwenVL-SAMTok reduces multimodal supervised fine-tuning to next-token prediction. Every mask in the training set is replaced by its two-token representation, allowing existing transformer architectures to learn both text and dense pixel-wise supervision via the same autoregressive language modeling objective: $$\mathcal L_{\mathrm{SFT}} = -\sum_{i} \log p_\theta(s_i \mid s_{<i}, \mathcal I)$$

For pixel-wise mask generation or selection tasks, reinforcement learning is adopted with a purely textual reward based on the overlap between model-predicted and ground-truth token pairs. The reward is defined as the fraction of correctly produced mask words: $$R_{\mathrm{mask}}(\hat t, t^*) = \frac{\#\{\hat t_i \in t^*\}}{\max(|\hat t|, |t^*|)}$$ The RL objective is thus: $$\mathcal L_{\mathrm{RL}} = -\,\mathbb E_{\hat t \sim \pi_\theta} \left[ R_{\mathrm{mask}}(\hat t, t^*) \log \pi_\theta(\hat t \mid \mathcal I) \right]$$

This approach requires no pixel-level loss, segmentation head, or deviation from the base MLLM’s architecture.

3. Data Scaling and Training Protocol

• Tokenizer Pre-training: The SAMTok module is pre-trained on 209 million masks derived from diverse instance, entity, and part-level segmentation datasets (COCO, ADE20K, Cityscapes, as well as proprietary and UI-derived masks).
• Supervised Fine-tuning: Approximately 5 million image-text-mask triplets are used, covering mask generation (grounding, referring segmentation, scene parsing), region captioning, VQA, and interactive segmentation tasks. All masks are encoded as token pairs.
• Reinforcement Learning: Further refinement uses 26,000 chain-of-thought simulated cold-start samples, supplemented with 8,000 GRES and 41,000 GCG samples, employing GRPO with the answer-matching reward.
• Optimization Details: Training is performed on NVIDIA A100 GPUs using AdamW. SFT uses a learning rate of $2 \times 10^{-5}$, batch size 256; RL fine-tuning uses a learning rate of $1 \times 10^{-6}$.

4. Integration with QwenVL and MLLM Infrastructure

QwenVL-SAMTok requires no architectural modification to its underlying QwenVL transformer except for the introduction of 512 mask tokens—256 per codebook step—as well as special delimiters. These tokens are randomly initialized embeddings and trained alongside the LLM’s output head. The MLLM’s image encoder remains frozen, with only the projection layer and LLM parameters updated during fine-tuning.

In this framework:

• All region processing reduces to token prediction rather than regression/segmentation decoding.
• No segmentation head, pixel-wise loss, or mask-specific auxiliary objectives are used.
• The system enables direct supervised and RL-based optimization of dense region prediction through standard language modeling and policy objectives.

5. Benchmark Performance and Result Highlights

QwenVL-SAMTok demonstrates state-of-the-art or highly competitive results across numerous pixel-wise vision-language tasks:

Task	Metric/Result Improvements
GCG (Interleaved Text–Mask)	+1.3 METEOR, +5.5 CIDEr, +5.3 AP50, +5.2 mIoU, +4.7 Recall on val set vs prior best; similar test gains
GRES (Text-to-Mask)	gIoU +1.5%, cIoU equiv., N-acc +4.3% over expert decoders; RL delivers +6.8 gIoU, +4.9 cIoU, +18.9 N-acc
RefCOCO/+/g (Referring Expr. Seg)	cIoU ≈82–85%, matching or surpassing task-specific architectures
GroundingSuite Zero-Shot	gIoU 67.8 vs prior 62.6
Multi-Round Interactive Segmentation	+7.7% average cIoU per round, +10.7% at part-level
Panoptic Scene Graph Generation	R@20 = 19.8, mR@20 = 15.4 vs 20.6/14.8 of expert systems
Region Captioning/Cross-modal Video Bench	Qwen3VL-SAMTok-4B: 65.6 Avg (DLC-Bench), 145.0 (MDVP OCR ZS), 2.88 (VideoRefer-D) compared to best prior models

Such results indicate that precise region representations and dense visual reasoning can be accomplished by autoregressive models without explicit vision decoders or task-specific network heads.

6. Significance and Scalability of the Paradigm

By discretizing arbitrary pixel masks as fixed-length token pairs, QwenVL-SAMTok demonstrates a generic, modular approach to equipping any LLM with pixel-level vision capabilities. This tokenization bridges vision-language modalities, enabling joint training and inference under a fully language-centric regime. The methodology lends itself to scaling (as both tokenizer and MLM can be pre-trained/fine-tuned on growing datasets with standard hardware and techniques) and porting to other foundational MLLMs.

A plausible implication is a general reduction in technical debt for pixel-wise multimodal modeling, opening possibilities for further integration of continuous perceptual modalities into discrete language-transformer architectures while leveraging RL-based optimization for structured prediction (Zhou et al., 22 Jan 2026).