Clarification needed for reproducing the released ImpactSynth CBCT regional metrics

[Ajuning]

Hi Valentin,

Thank you again for releasing ImpactSynth and the pretrained models.
We are trying to reproduce the regional CBCT-to-CT results reported in CBCT_MODEL_INFO.md using the currently released VBoussot/ImpactSynth checkpoints, the public IMPACT registrations, and the published validation split.

We have now performed a fairly extensive reproduction audit, but we still cannot fully reproduce the reported metrics. We would greatly appreciate your help identifying the missing protocol detail.

Setup used

• Model repository: VBoussot/ImpactSynth
• Model snapshot/revision: d00d991
• Checkpoints: CBCT/CV_0.pt to CBCT/CV_4.pt
• Public Prediction.yml
• Public SynthRAD2023 and SynthRAD2025 CT/CBCT data
• Public IMPACT registration transforms
• The 150-case regional validation split listed in CBCT_MODEL_INFO.md
• Native five-fold ensemble with the released TTA configuration
• We tested KonfAI 1.5.3, 1.5.9, and the current 1.8.x-compatible pipeline

We did not mix the SynthRAD2025 challenge models with the newer ImpactSynth models.

Main reproduction results

For the 21 pelvis cases:

Evaluation setting	MAE	PSNR	SSIM
Reported ensemble	34.84 HU	35.12 dB	0.924
Our result using the public MASK.mha	approximately 48.30 HU	32.61 dB	approximately 0.906
Our result using MASK.mha AND CT > -300 HU with KonfAI 1.5.9	34.851 HU	36.005 dB	0.915
Same ROI using the official SynthRAD2023 metric implementation	34.857 HU	35.861 dB	0.954

The pelvis MAE obtained with CT > -300 HU differs from the reported value by only 0.017 HU. This strongly suggests that the Mask_0 used to generate the model-card table may differ from the currently available public MASK.mha.

However, PSNR and SSIM still do not simultaneously match, so we cannot consider this a complete protocol reproduction.

For the 26 brain cases:

Evaluation setting	MAE
Reported ensemble	39.42 HU
IMPACT-registered CBCT evaluated against the public CT/MASK	approximately 77.69 HU
Original CBCT without IMPACT transformation	approximately 79.08 HU
IMPACT CBCT with MASK.mha AND CT > -300 HU	approximately 68.30 HU

Therefore, the brain discrepancy cannot be explained simply by using original versus IMPACT-registered CBCT.

#Hypotheses already tested

We also tested the following possibilities:

1. Current versus initial released checkpoints: only approximately 0–1 HU difference on fixed cases.
2. Current versus earlier public IMPACT transforms: approximately 1 HU difference on tested cases.
3. Opposite registration direction, i.e. transforming the CT instead of the CBCT: clearly worse.
4. Native five-fold inference versus our application wrapper: no large difference.
5. Single CV0 versus the five-fold ensemble: the expected relative improvement is present.
6. Historical versus current body-segmentation model used during inference: insufficient to explain the discrepancy.
7. Using the historical CT body mask for evaluation: did not reproduce the reported values.
8. Using the MR_CBCT checkpoint directory instead of the CBCT directory: worse results.
9. Using the original CBCT instead of the IMPACT-registered CBCT: worse overall, especially for the brain cohort.

Information that would resolve the issue

Could you please clarify or provide the following?

1. What exactly was used as Mask_0 when generating the metrics in CBCT_MODEL_INFO.md?
   • Was it the original SynthRAD mask.mha?
   • A CT intensity threshold mask?
   • An ImpactSeg/TotalSegmentator body mask?
   • An intersection of the CT body mask and CBCT field of view?
2. Could you provide the exact 150 evaluation masks or the script used to generate them?
3. Were the reported regional metrics generated from the current five released checkpoints, or from an earlier checkpoint/data snapshot?
4. Could you provide the exact frozen validation case list, data/transform checksums, KonfAI version, PyTorch version, and evaluation command?
5. If possible, could you provide the generated sCT or per-case metrics for one pelvis case, for example pelvis_2PA002, and one brain case, for example brain_2BA001?

A single reference prediction would allow us to determine whether the remaining difference comes from inference/data preprocessing or only from evaluation.

Our goal is to reproduce the released pretrained baseline faithfully before performing controlled abdominal/pelvic soft-tissue improvements.

Thank you very much for your help.

[Ajuning]

Your any possible answers would truly do me a favor. I sincerly appreciate and expect your favorable hand