Contrastive Analysis aims to separate factors that are common between two data distributions from those that are salient to only one of them. Existing contrastive methods are based on generative models (e.g., VAEs or GANs) that often suffer from limited reconstruction and image quality, which hampers effective latent factor separation and limits their applicability to high-fidelity image generation and editing. We propose a novel conditioning framework for diffusion models that enables contrastive decomposition without compromising generation quality. We first train a prompt-free, image-conditioned diffusion model, and then learn to decompose the conditioning into a common and a salient factor, using weak supervision. We prove that the additive contrastive factorization, commonly assumed in prior work, is identifiable under mild conditions. This factorization enables targeted editing operations by swapping or interpolating only the salient factor while preserving the common factor.
Diff-CA applies Contrastive Analysis to the conditioning tokens Z of a latent diffusion model, in two stages. In Stage 1, we train a prompt-free, image-conditioned diffusion model from scratch: a cross-query module compresses DINOv3 features into a short sequence of tokens, completed by a color token, so that the generated attributes depend on Z rather than on the starting noise. In Stage 2, a small Transformer encoder splits Z into a salient code ẐS and a common code ẐC whose sum reconstructs Z. A pinning loss pushes ẐS to zero on background samples, adversarial training discourages ẐC from predicting the dataset label, and a cycle-consistency loss on swapped codes encourages the two factors to be independent. We prove that this additive factorization is identifiable under mild conditions.
Exchanging the salient codes ẐS of two images while keeping their common codes ẐC transfers the salient attribute without altering the rest of the image. We evaluate on three domains, with only binary labels: faces (FFHQ: glasses, smile), animals (AFHQ: cat vs. dog) and brain MRI (BraTS 2023: tumor).
Existing CA methods (MM-cVAE, SepVAE, DoubleInfoGAN) rely on low-capacity VAE or GAN generators: they swap the salient factor, but produce blurry images and lose identity. On FFHQ glasses, trained with binary labels only, Diff-CA reaches 94.5% fine-grained glasses accuracy after swapping, against 47.0% for the best baseline, while preserving gender, smile and head pose.
| Reconstruction | Swapping (fine-grained glasses classes) | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | SSIM ↑ | LPIPS ↓ | ID-Sim ↑ | Acc (%) ↑ | ID-Sim ↑ | Gender (%) ↑ | Smile (%) ↑ | Pose MAE (°) ↓ |
| MM-cVAE | .370 | .643 | .307 | 11.8 | .302 | 62.3 | 57.7 | 4.62 |
| SepVAE | .358 | .642 | .307 | 19.6 | .301 | 57.7 | 56.4 | 8.29 |
| DoubleInfoGAN | .326 | .430 | .327 | 47.0 | .314 | 68.5 | 79.2 | 7.35 |
| Diff-CA (ours) | .610 | .115 | .522 | 94.5 | .474 | 92.5 | 92.9 | 1.87 |
FFHQ, glasses attribute; all models are trained with binary labels only (glasses vs. no glasses). Acc: accuracy of a fine-grained glasses classifier on the swapped images. ID-Sim: cosine similarity between face-recognition (ArcFace) embeddings of the original and generated images. Gender, smile and pose measure the preservation of common factors. Swapping results are macro-averaged over no glasses, reading glasses and sunglasses.
Fixing the common code of one image and interpolating its salient code toward that of another image produces semantically meaningful intermediate images: the glasses style changes gradually while identity is preserved, and cat features appear on a dog while pose and background are kept.
The salient space is also organized beyond the labels it was trained with. Although training only used glasses / no-glasses labels, the salient codes of faces without glasses, with reading glasses and with sunglasses form distinct groups, most clearly in the UMAP projection: a form of concept discovery.
On BraTS 2023, the salient factor is the presence of a brain tumor, and the only supervision is a slice-level label (tumor / no tumor) on 2D slices. Such binary labels are routinely assigned in hospitals, while the corresponding imaging patterns can be hard to describe with a text prompt. Swapping ẐS adds a tumor to a healthy slice or removes it from a pathological one, and interpolating ẐS between two slices changes the tumor progressively.
@inproceedings{soumm2026diffca,
title={Diff-CA: Separating Common and Salient Factors with Diffusion Models},
author={Soumm, Micha{\"e}l and Fournier-Montgieux, Alexandre and He, Yunlong and Gori, Pietro and Newson, Alasdair},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2026}
}