arXiv：cs.AI· Daniel Mensing, Jan Kapar, Jochen G. Hirsch, Matthias G\"unther, Horst Hahn, Marvin N. Wright·· 20 小时前
多模态潜在扩散模型通过交叉注意力在联合潜在空间中联合合成 MRI 与表格数据
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
AI 导读
研究人员提出一种多模态潜在扩散模型，通过交叉注意力在共享潜在空间中联合合成体积 MRI 与表格临床数据，并用变分自编码器先融合两种模态再扩散合成，MRI 与表格数据各用独立解码器重建。
正文
Abstract:We propose a multimodal latent diffusion model that jointly synthesizes volumetric magnetic resonance imaging (MRI) and tabular clinical data within a shared latent space via cross-attention. This approach enables coherent joint representation learning of MRI and tabular modalities for generative modeling. Our model utilizes a variational autoencoder to fuse the two modalities before diffusion-based synthesis, allowing modality-appropriate reconstruction with separate decoders for MRI and tabular data. We evaluated the framework on data from the German National Cohort (NAKO Gesundheitsstudie), comprising over 10,000 participants with MRI scans and clinical tabular features such as age, sex, body measurements, and ethnicity. The generated MRI volumes exhibited anatomical plausibility and body composition consistent with the synthesized tabular attributes. Quantitative evaluation using Fréchet distance and precision-recall metrics confirmed high-fidelity image generation. In the tabular modality, our model outperformed CTGAN across standard evaluation metrics and achieved results comparable to TVAE, demonstrating competitive performance relative to established unimodal baselines. This work is, to our knowledge, the first to demonstrate the feasibility of jointly modeling MRI and mixed-type tabular data in a single latent diffusion framework, offering a proof-of-concept for generating coherent synthetic multimodal patient data and aligning with the broader goal of developing digital twins in healthcare.
| Subjects: | Image and Video Processing (eess.IV); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.06699 [eess.IV] |
| (or arXiv:2605.06699v2 [eess.IV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06699 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proc. SPIE 13925, Medical Imaging 2026: Image Processing, 139252D (April 03, 2026) |
| Related DOI: | https://doi.org/10.1117/12.3086603
DOI(s) linking to related resources |
Submission history
From: Jan Kapar [view email]
[v1]
Tue, 5 May 2026 08:30:20 UTC (1,116 KB)
[v2]
Mon, 12 Oct 2026 11:28:05 UTC (1,116 KB)
来源：arXiv：cs.AI · arxiv.org