How to determine the final corrected data matrix

[juyanmei]

If I want to perform batch correction on cross-study research (e.g., 10 studies) with the ultimate goal of obtaining a corrected data matrix, theoretically, the choice of different studies as the test set will influence the corrected data obtained through dmc.transform(X_with_batch). So, how should I determine the final corrected data matrix?

[gaustin15]

Thanks for the question. The best approach would depend on your end goal and how you plan to use the corrected data. That said, here are three general strategies you could consider for obtaining a final corrected matrix when working across multiple studies:

1. Train on all studies at once:
   You can run dmc.fit(X_with_batch, y) using all your data as the training set. This will give you a single corrected matrix, but be aware that this introduces train/test leakage. That could be acceptable if you're using the matrix for certain downstream analyses, but wouldn’t be viable for something such as testing generalization of signals.
2. Train on all studies at once, using labels not related to your primary analysis:
   You can run dmc.fit(...) on all data using a label or set of labels unrelated to your main research question, and then transform everything. This way you avoid fitting transformations to the primary signal of interest while still leveraging all the data at once.
3. Leave-one-study-out correction: It would be possible to treat each study as a held-out test set once, run DEBIAS ~10 times (once per held-out study), and then concatenate the transformed test sets. This would avoid leakage, but could introduce some weird patterns. We haven't validated this approach, so for the time being I would recommend 1 or 2 over this approach, although depending on your intended analyses, this approach might make sense.

Hope this helps—happy to clarify further depending on your specific use case.

[juyanmei]

Thank you for your reply. I would like to use the corrected data matrix for subsequent differential analysis between disease and healthy groups, as well as for building a classifier model. Since it's a meta-analysis, aside from the case/control grouping, it's difficult to find common labels for the second method you mentioned. I will give the first method a try and see how it goes.

[gaustin15]

Thank you for sharing the additional details. One point to emphasize is that using all sample labels (including test labels) as a preprocessing step—prior to cross-group generalization prediction in a scenario such as evaluating classification models—risks inadvertently inflating performance estimates. When correction is performed using labels from the entire dataset, including those from the test set, it can introduce information leakage because the procedure has been informed by data that should remain hidden during model training. This compromises the independence of the test set and can result in overly optimistic performance (which would apply to DEBIAS-M and similar methods).

To avoid this, for that particular type of analysis I would recommend applying DEBIAS-M in a way that respects the training-test split: specifically, DEBIAS-M should be run separately for each training set, using only the labels within that training set. This approach helps maintain the cross-group generalization assessment and provides a more reliable estimate of model performance.