Q-Zoom is a query-aware adaptive perception framework that lets a Multimodal LLM look at high resolution only when it needs to, and only where it needs to — without changing the underlying backbone. Instead of indiscriminately flooding the quadratic self-attention with redundant high-resolution tokens, Q-Zoom reasons in a coarse-to-fine manner using two lightweight modules grafted on top of an off-the-shelf MLLM:
Together, these three pieces let Q-Zoom establish a dominant Pareto frontier: on Qwen2.5-VL-7B it is 2.52× faster on Doc/OCR benchmarks and 4.39× faster on high-resolution benchmarks at matched accuracy, and when configured for maximum fidelity it surpasses the baseline's peak by +1.1% and +8.1% respectively. The recipe transfers seamlessly to Qwen3-VL, LLaVA-1.5, and emerging RL-based thinking-with-image models.
Q-Zoom 是一种查询感知的自适应视觉感知框架，使多模态大模型 只在真正需要时、也只在真正相关的区域进行高分辨率观察，而且无需改动底层骨干网络。 它不再把冗余的高分辨率视觉 token 一股脑送入具有二次复杂度的自注意力模块， 而是在现成的 MLLM 之上增添两个轻量模块，以由粗到细的方式完成推理：
这三个组件共同让 Q-Zoom 建立起显著占优的帕累托前沿：在 Qwen2.5-VL-7B 上，在精度相当的前提下，它在 Doc/OCR 基准上可提速 2.52 倍， 在高分辨率基准上可提速 4.39 倍；当配置为最高保真模式时，又分别比基线峰值 高出 +1.1% 和 +8.1%。这一方案还能无缝迁移到 Qwen3-VL、LLaVA-1.5，以及新兴的基于强化学习的图像思考类模型。
At inference, Q-Zoom always emits a direct response from the low-resolution pass; the high-res gating head decides per sample whether to also emit a RoI-based response by re-decoding the cropped region predicted by SD-RPN.
推理时，Q-Zoom 会先基于低分辨率输入生成直接响应；随后由高分辨率门控头按样本判断， 是否需要对 SD-RPN 预测出的裁剪区域重新解码，从而生成额外的基于 RoI 的响应。
Each stage trains a different subset of parameters while everything else stays frozen:
每个阶段只训练不同的一部分参数，其余参数全部保持冻结：
Train only the twig_T twig layers (the gating branch grafted on top of base
layer twig_K) using pseudo ROI maps generated by the base VLM. The base LLM,
lm_head, and vision encoder all stay frozen.
仅训练 twig_T 个 twig 层（即接在第 twig_K 层之上的 twig 分支）；
监督信号来自基础 VLM 自行生成的伪 RoI 注意力图。基础 LLM、lm_head 和
视觉编码器均保持冻结。
The frozen TWIG branch is carried over from Stage 1; only the LLM decoder +
lm_head are trained on a hard-sample mixture mined by an
LLM-as-a-Judge that compares the direct response against the RoI response on
a fresh question pool.
沿用并冻结阶段 1 得到的 TWIG 分支；只在由 LLM-as-a-Judge 从新问题池中挖掘出的
困难样本混合集上，训练 LLM 解码器和 lm_head。这些困难样本来自
对直接响应与基于 RoI 的响应的比较结果。
Train only the high-res gating network (high_res_layers +
high_res_head) that decides per input whether to fire the
high-resolution RoI refinement pass. Stage 3 uses a consistency-aware
sample-generation pipeline that filters unstable / contradictory examples
before BCE supervision.
仅训练高分辨率门控网络（high_res_layers + high_res_head），
由它决定每个输入是否需要触发高分辨率 RoI 细化流程。阶段 3 采用一致性感知
的样本生成流程，会在 BCE 监督前剔除不稳定或相互矛盾的样本。
Q-Zoom is competitive with — and often surpasses — recent detail-zoom and tool-use baselines across five backbones (LLaVA-1.5-7B/13B, Qwen2.5-VL-3B/7B, Qwen3-VL-4B).
Q-Zoom 在五种骨干网络（LLaVA-1.5-7B/13B、Qwen2.5-VL-3B/7B、 Qwen3-VL-4B）上，与近期的细节缩放和工具调用基线相比均具竞争力，而且多数情况下更优。
Performance on Document & OCR benchmarks. Dataset subscripts denote the evaluation split. Performance subscripts show the absolute improvement (↑) over the baseline. Throughput is relative to the baseline, measured on a single NVIDIA A6000 GPU, and our results are evaluated under a per-single-image constraint of 576 maximum visual tokens. Document & OCR 基准上的结果。 数据集下标表示评测划分；性能下标表示相对基线的绝对提升（↑）。 吞吐量以基线为参照，在单张 NVIDIA A6000 GPU 上测得； 所有结果均在单张图像最多 576 个视觉 token的约束下评测。
Performance on Vision-Centric and High-Resolution benchmarks. Dataset subscripts denote the specific evaluation split; performance subscripts indicate the absolute improvement (↑) of Q-Zoom over the baseline. Tp denotes relative inference throughput, and averages are computed exclusively across the four Overall metrics (V* Bench, MME-RW-lite, HR-Bench-4K, HR-Bench-8K). Unless otherwise noted, all Q-Zoom results are evaluated under a per-single-image constraint of 4,096 maximum visual tokens. The † symbol marks numbers cited directly from the corresponding original publications. 视觉中心与高分辨率基准上的结果。 数据集下标表示具体评测划分；性能下标表示 Q-Zoom 相对基线的绝对提升（↑）。 Tp 表示相对推理吞吐量；平均值只在四项 Overall 指标 （V* Bench、MME-RW-lite、HR-Bench-4K、HR-Bench-8K）上计算。除非另有说明， 所有 Q-Zoom 结果均在单张图像最多 4,096 个视觉 token的约束下评测。 † 表示直接引自对应原始论文的数值。
Sweeping the visual-token budget from 256 to 4096 traces a Pareto curve. Q-Zoom (orange) shifts the curve up and to the right of the baseline VLM (gray) on both Doc/OCR-heavy and high-resolution vision benchmarks: at matched accuracy it is multiple times faster, and at matched throughput it lifts average accuracy by several points. The trend is consistent across both supported backbones.
将视觉 token 预算从 256 扫描到 4096，可以得到一条帕累托曲线。无论是在 Doc/OCR 密集型基准，还是在高分辨率视觉基准上，Q-Zoom（橙色）都把这条曲线整体推到了 基线 VLM（灰色）的右上方：在相同精度下可实现数倍提速，在相同吞吐量下又能把平均精度 提升数个百分点。这一趋势在两种支持的骨干网络上都保持一致。
Doc/OCR — 2.52× speedup at matched accuracy, +1.1% over baseline peak. Doc/OCR：在相同精度下提速 2.52×，比基线峰值高 1.1%。
High-resolution — 4.39× speedup at matched accuracy, +8.1% over baseline peak. 高分辨率：在相同精度下提速 4.39×，比基线峰值高 8.1%。
Doc/OCR — 1.70× speedup at matched accuracy, 53.1% visual-token reduction. Doc/OCR：在相同精度下提速 1.70×，视觉 token 数减少 53.1%。
High-resolution — 2.73× speedup at matched accuracy, 72.9% visual-token reduction. 高分辨率：在相同精度下提速 2.73×，视觉 token 数减少 72.9%。
SD-RPN produces a per-token attention map (middle column) that isolates the region the question is actually about. The model then re-decodes only that region to recover the correct answer. SD-RPN 会生成逐 token 的注意力图（中间列），定位出问题真正关注的区域； 模型随后只对该区域重新解码，即可恢复正确答案。
Two real samples processed by the same Q-Zoom-Qwen2.5-VL-7B checkpoint at the canonical resolution. On the left (InfoVQA), the gating head fires (score 0.984) → SD-RPN finds the small "Wages per day" bar chart → re-decoding the crop reads the Ecuador and Panama bars and sums them. On the right (TextVQA), the gating head correctly skips (score 0.000) — the answer is globally obvious — and the model answers directly without any extra compute.
Note: the SD-RPN heatmap and gate scores shown here are produced under a per-single-image constraint of 576 maximum visual tokens.
两个真实样本，由同一个 Q-Zoom-Qwen2.5-VL-7B 模型在相同分辨率下处理。 左侧（InfoVQA）：门控头触发（分数 0.984）→ SD-RPN 找到那个小小的“每日工资”柱状图 → 仅对裁剪区域重解码即可读出厄瓜多尔与巴拿马的数值并相加。 右侧（TextVQA）：门控头正确选择跳过（分数 0.000）— 因为答案在全局视图中显而易见 — 模型无需额外计算就 直接给出回答。
注：此处的 SD-RPN 热图和门控分数是在 单张图像最多 576 个视觉 token 的约束下得到的。
SD-RPN heatmap → bbox → re-decode the chart → "$18.25" SD-RPN 热图 → 边界框 → 重新解码该图表 → "$18.25"
For the two right-hand examples the gating sigmoid is essentially zero — SD-RPN never runs at all and the model decodes directly. 对于右侧两个样本，门控 sigmoid 几乎为零 — SD-RPN 完全不运行，模型直接解码。
The final Stage-3 weights are on Hugging Face for all three supported backbones.
三种已支持骨干网络的最终 Stage-3 权重均已发布到 Hugging Face。
| Backbone骨干模型 | HF modelHugging Face 模型 | TWIG configTWIG 配置 |
|---|---|---|
| Qwen2.5-VL-3B | YuhengSSS/Q-Zoom-Qwen2.5VL-3B |
K=24, T=3 |
| Qwen2.5-VL-7B | YuhengSSS/Q-Zoom-Qwen2.5VL-7B |
K=18, T=3 |
| Qwen3-VL-4B | YuhengSSS/Q-Zoom-Qwen3VL-4B |
K=24, T=3 |
huggingface-cli download YuhengSSS/Q-Zoom-Qwen2.5VL-7B \
--local-dir ./checkpoints/Q-Zoom-Qwen2.5VL-7B \
--local-dir-use-symlinks False
CHECKPOINT_PATH=./checkpoints/Q-Zoom-Qwen2.5VL-7B \
bash examples/eval_only/eval_qwen2_5vl_stage3.sh@article{qzoom,
title = {Q-Zoom: Query-Aware Adaptive Perception for Efficient
Multimodal Large Language Models},
author = {Shi, Yuheng and Pei, Xiaohuan and Wen, Linfeng and
Dong, Minjing and Xu, Chang},
journal= {arXiv preprint arXiv:2604.06912},
year = {2026}
}You may also be interested in our earlier work that introduced the self-distilled RoI predictor used by Q-Zoom's SD-RPN branch: 您也许会对我们更早的一项工作感兴趣；该工作首次提出了 Q-Zoom 的 SD-RPN 分支所采用的 自蒸馏 RoI 预测器：
@article{shi2025catching,
title = {Catching the Details: Self-Distilled RoI Predictors for
Fine-Grained MLLM Perception},
author = {Shi, Yuheng and Pei, Xiaohuan and Dong, Minjing and Xu, Chang},
journal= {arXiv preprint arXiv:2509.16944},
year = {2025}
}