InsertFuse: A Unified Framework for Multi-Category Reference-Guided Image Insertion

Abstract

We present InsertFuse, a unified framework for multi-category reference-guided image insertion. Its key idea is to decouple category-specific expertise learning from cross-category capability consolidation. InsertFuse first trains specialized experts for different insertion categories and then introduces Insertion On-Policy Distillation (IOPD) to consolidate their capabilities into a single student. By querying the matched expert at states visited by the student, IOPD preserves category-specific insertion behavior while mitigating the cross-category interference caused by direct joint training.

To improve spatial control, we propose Token-Aligned Geometry Conditioning (TAGC), which maps mask-derived geometric cues to the visual token grid, and Region-Balanced Flow Matching, which separately normalizes prediction errors inside and outside the insertion region. We further introduce Reference CFG to isolate and strengthen the guidance induced by the visual reference under fixed scene and geometry conditions, with IOPD transferring this enhanced supervision into the unified student.

Extensive experiments on the public AnyInsertion benchmark and our multi-category test set demonstrate state-of-the-art performance on most metrics, showing strong reference fidelity and generation quality across diverse insertion categories.

0.916 CLIP-I (AnyInsertion) best reference fidelity

24.26 PSNR (AnyInsertion) best reconstruction

55.4% User Overall Pref. multi-way study

5→1 Experts → Student via IOPD

Key Ideas

Different insertion categories need different strategies. InsertFuse first learns category specialists, then consolidates them into a single student without the interference of direct joint training—while adding explicit spatial and reference supervision.

Core Framework

Insertion On-Policy Distillation (IOPD)

Train one expert per category (accessory, animal, garment, object, human), freeze them, and distill into a unified student on the student's own denoising trajectory. The matched expert $E_{k(y)}$ provides Reference-CFG-enhanced velocity targets at student-visited states $\widetilde{\mathbf{x}}_n^{S}$:

$$ \mathcal{L}_{\mathrm{IOPD}} = \mathbb{E}_{n} \Big\| \mathbf{v}_{S_{\theta}}\!\left(\widetilde{\mathbf{x}}_{n}^{S}, t_{n}\right) - \operatorname{sg}\!\big[ \mathbf{v}_{E_{k(y)}}^{\mathrm{ref}}\!\left(\widetilde{\mathbf{x}}_{n}^{S}, t_{n}\right) \big] \Big\|_{2}^{2} $$

After distillation, only the single-branch student is kept—no expert routing or extra null-reference pass at inference.

Spatial Control

TAGC & Region-Balanced Flow Matching

Coarse mask concatenation and global flow-matching leave insertion geometry and small regions under-supervised. We inject token-aligned geometry and rebalance regional losses:

TAGC geometry encoding

$$ \mathbf{G}(\mathbf{M}) = \big[ \mathbf{M},\; \tanh(D_{\mathbf{M}}/\tau),\; e^{-|D_{\mathbf{M}}|/\tau} \big] $$

Signed-distance geometry is encoded and residual-injected into both target and context branches: $\widetilde{\mathbf{z}}=\mathbf{z}+\Delta\mathbf{z}^{\mathrm{geo}}$.

Region-Balanced FM

$$ \mathcal{L}_{\mathrm{RBFM}} = \mathbb{E} \!\left[ \frac{ \lambda_{\mathrm{ins}}\,\ell_{\mathrm{ins}} + \lambda_{\mathrm{bg}}\,\ell_{\mathrm{bg}} }{ \lambda_{\mathrm{ins}}+\lambda_{\mathrm{bg}} } \right] $$

$\ell_{\mathrm{ins}}$ / $\ell_{\mathrm{bg}}$ are area-normalized so supervision does not scale with mask size ($\lambda_{\mathrm{ins}}{=}3$, $\lambda_{\mathrm{bg}}{=}1$).

Reference Fidelity

Reference Classifier-Free Guidance

Standard CFG drops text only and does not isolate the visual reference. Reference CFG keeps the source scene and TAGC fixed, and amplifies the velocity gap induced solely by the reference condition:

$$ \mathbf{v}_{k}^{\mathrm{ref}} = \mathbf{v}_{E_k}(\cdot\,;\,\varnothing) + s_r \Big[ \mathbf{v}_{E_k}(\cdot\,;\,\mathbf{z}^{\mathrm{ref}}) - \mathbf{v}_{E_k}(\cdot\,;\,\varnothing) \Big] $$

Experts train with reference dropout $p_{\mathrm{drop}}{=}0.2$; distillation uses $s_r{=}4.0$. The student learns the enhanced policy and needs only a single positive-reference prediction at inference.

Method

Built on Qwen-Image-Edit with LoRA experts for accessory, animal, garment, general-object, and human insertion, then unified via IOPD.

**Overview of InsertFuse.** **Left:** masked context and reference images are encoded; TAGC converts the target mask into grid-aligned geometry features integrated with patchified visual features. **Right:** category-specific experts are trained first; during student training, on-policy rollouts and IOPD are interleaved so the matched frozen expert supplies velocity targets at visited states. The unified student handles all categories without retaining experts at inference.

Stage 1 · Experts

Category-Specific Expert Learning

Train five LoRA experts on accessory, animal, garment, general-object, and human insertion. Shared insertion components—TAGC, Region-Balanced Flow Matching, and Reference CFG—equip each expert with spatial control and reference fidelity before consolidation.

Stage 2 · IOPD

Insertion On-Policy Distillation

A unified student rolls out its own denoising trajectory; the matched frozen expert provides Reference-CFG-enhanced velocity targets at the same states. Direct joint training is avoided, so cross-category interference is mitigated while specialized behavior is retained.

Quantitative Results

InsertFuse achieves the best results on most metrics on both the public AnyInsertion benchmark and our multi-category test set.

Method	AnyInsertion						Multi-Category Test Set
Method	DINO-I↑	CLIP-I↑	PSNR↑	SSIM↑	LPIPS↓	FID↓	DINO-I↑	CLIP-I↑	PSNR↑	SSIM↑	LPIPS↓	FID↓
(a) Reference-Guided Insertion Methods
AnyDoor	0.6939	0.8544	17.77	0.6902	0.2553	4.75	0.6331	0.8769	18.57	0.7606	0.1999	1.98
Insert Anything	0.7884	0.9070	23.83	0.8687	0.0855	1.55	0.7009	0.8996	24.35	0.8997	0.0825	1.35
A²-Edit	0.7586	0.8965	22.25	0.8427	0.1127	2.31	0.6836	0.8926	23.56	0.8930	0.0924	1.48
(b) General-Purpose Image Editing Models
FLUX.1 Kontext	0.7219	0.8771	20.58	0.8366	0.1339	3.87	0.5170	0.8525	20.45	0.8820	0.1126	1.84
Qwen-Image-Edit-2511	0.7705	0.8987	21.73	0.8090	0.1158	1.82	0.6164	0.8670	19.16	0.7492	0.1814	1.43
Qwen-Image-2.0	0.7671	0.8954	21.66	0.8074	0.1175	1.98	0.6110	0.8659	18.95	0.7428	0.1863	2.51
(c) Our Method
InsertFuse	0.7838	0.9156	24.26	0.8850	0.0948	1.49	0.7148	0.9020	24.40	0.8967	0.0795	1.39

Best in bold, second-best underlined.

User Study (preference share %)

Multi-way preference on the multi-category test set.

Reference Fidelity51.8%
Insertion Quality58.2%
Overall Quality55.4%

InsertFuse preferred over Insert Anything, A²-Edit, and general editors.

Ablation (multi-category set)

Components and consolidation strategy.

Baseline0.689 / 23.47
+ TAGC / + RBFM / + Ref CFG0.697 / 0.696 / 0.708
Direct Joint Training0.648 / 19.36
InsertFuse (IOPD)0.715 / 24.40

Metrics: DINO-I / PSNR (where applicable).

Qualitative Comparison

Across accessory, animal, garment, object, and human insertion, InsertFuse better preserves reference-specific appearance while adapting to target geometry and occlusion.

Ablation Study

TAGC and Region-Balanced Flow Matching improve spatial alignment; Reference CFG and IOPD strengthen reference fidelity over direct joint training.

Qualitative ablations of TAGC and Region-Balanced Flow Matching. — **(a) TAGC** improves geometric alignment and produces a cleaner transition between the cap and the head. **(b) RBFM** reduces bracelet artifacts and recovers clearer chain and star details.

Qualitative ablations of Reference CFG and IOPD. — **(a) Reference CFG:** increasing reference guidance strength improves preservation of the beanie's texture and logo. **(b) IOPD:** direct joint training distorts the pendant, whereas the category expert and IOPD produce a more reference-consistent necklace.

BibTeX

@misc{li2026insertfuse,
  title         = {InsertFuse: A Unified Framework for Multi-Category
                   Reference-Guided Image Insertion},
  author        = {Guangzhao Li and Qingyan Wei and Huayu Zheng and Yige Zheng
                   and Chaoyang Zhang and Jie Yang and Yunan Ding and Yan Tai
                   and Siqi Luo and Xiaohong Liu},
  year          = {2026},
  note          = {arXiv preprint (coming soon)},
  url           = {https://insertfuse.github.io}
}