Draft Status
Draft - team will hold off on page creation
Category
Infrastructure
Key Investigators
- Robin Peretzke (DKFZ, Heidelberg)
- Maximilian Fischer (DKFZ, Heidelberg
Project Description
- New contrast-enhancing lesions following treatment of intracranial tumors may reflect either true tumor recurrence or radiation-induced contrast enhancements (RICE). Distinguishing between these entities remains unreliable but is critical for subsequent treatment decisions.
- We have developed and trained a multimodal deep learning model (RICE-Net) that achieves good performance in differentiating RICE from tumor recurrence using longitudinal MRI data (e.g., post-intervention, post-pseudoprogression) in combination with radiation treatment plans.
- RICE-Net is currently integrated into Kapaana, providing a structured environment for processing imaging and radiotherapy data and enabling inference on these inputs within a clinical routine.
- However, relevant clinical information, including pathology and radiology reports as well as medication plans, is not yet integrated, although it could substantially improve model performance and clinical utility. Corresponding multimodal extensions are currently under development and planned for integration into a new Kapaana workflow.im
- However, such models cannot yet be integrated into Kapaana, as the platform currently does not support text processing or the integration of large language and vision–language models.
Objective
- Extend Kapaana to support inference with vision-language models.
- Enable ingestion, parsing, and tokenization of PDF-based clinical documents for use as model inputs.
- Create the technical basis for multimodal models that combine imaging, textual, and structured clinical data.
Approach and Plan
- Develop the necessary infrastructure within Kapaana for deployment and inference of LLM/VLM models.
- Implement robust pipelines for extracting and tokenizing text from clinical documents and aligning them with imaging data.
- Integrate textual and tabular clinical information with longitudinal MRI and radiation data in a unified multimodal model.
- Evaluate the extended model on retrospective cohorts with respect to performance, robustness, and interpretability.
- Prepare the system for subsequent prospective evaluation and use in interdisciplinary tumor board settings.
Progress and Next Steps
-> Within Kaapana, PDF reports can now be uploaded and embeddings are generated
- The pdf is converted into DICOM-based representations.
- The operator reads the DICOM from the input folder.
- The text is extracted, cleaned, and tokenized into model-readable units.
- The tokens are transformed into embeddings using the multilingual E5 transformer model.
- The embeddings are stored together with the DICOM output.
Illustrations
No response
Background and References
No response
Draft Status
Draft - team will hold off on page creation
Category
Infrastructure
Key Investigators
Project Description
Objective
Approach and Plan
Progress and Next Steps
-> Within Kaapana, PDF reports can now be uploaded and embeddings are generated
Illustrations
No response
Background and References
No response