Inquiry About evo_score and Reported Spearman Correlations in ncRNA Scanning Mutagenesis

[wangleiofficial]

I hope you're doing well. I'm currently analyzing the evo_score values reported in your work on ncRNA scanning mutagenesis data and would like to ask for a clarification.

As I understand it, evo_score refers to the Spearman correlation between evolutionary conservation and measured functional effects. However, I noticed some discrepancies between your reported evo_score values and those in previous publications. For example:

Guy (2014): Evo 1 reports Spearman ρ = 0.24, but your evo_score is 0.0023

Domingo (2018): Evo 1 reports ρ = 0.45, your evo_score is 0.4207 (very close)

Andreasson (2020): Evo 1 reports ρ = 0.14, your evo_score is 0.0519

Given that the evo_score is intended to be the Spearman rho, could you clarify what might account for these differences?
Is it possible that different subsets of variants or filtering criteria were used when computing evo_score?

I would greatly appreciate any insight you could provide.

Thank you very much for your time and help!

[kevinhyj]

Hi @wangleiofficial,😊

First of all, please accept my sincere apologies for the delayed response. I appreciate your patience and your detailed observation regarding the evo_score discrepancies.

Regarding the three datasets you mentioned (Guy 2014, Domingo 2018, and Andreasson 2020), here are a few clarifications:

Data Collection: These datasets were manually collected and processed by me at the time of the study. I am not entirely sure if other databases had standardized versions of these specific datasets available during my collection process, which might account for differences in the variant subsets used.
Scoring Script: I used the official scoring script provided by Evo1 to ensure the computation logic remained consistent with their framework.
Transparency: To help clear up any confusion, I will open-source these three specific processed datasets here on GitHub as soon as possible. This should allow you to directly compare the variants and filtering criteria.

On a related note:
I’ve recently completed a new work involving larger-scale LLM zero-shot fitness prediction.[https://github.com/GENTEL-lab/EVA/tree/main/notebooks/prediction] In this new project, I utilized standard benchmark data from existing established datasets, which I believe provides more credible and comparable results.

This new work is fully open-source, and if there is interest, I am more than happy to open-source the entire pipeline code for all model benchmarks as well.

If you have any further confusion or would like to discuss the methodology deeper, please don't hesitate to reach out. I'll be keeping a close eye on this thread!!!!🙏🙏😊😊