Very poor model performance for parameter combinations with alpha + beta >= 1

[hansenp]

I investigated the performance of the BOQA model depending on the parameters alpha and beta.

Our BOQA implementation in Java generates a JSON file containing the results for one or more queries (phenopackets). For each query, the file contains a diagnosed disease (taken from the phenopacket) and the four BOQA counts (FP, TP, FN, TN) for each of the approximately 8000 annotated diseases in HPOA.

I used our BOQA implementation to generate JSON files for all phenopackets in phenopacket-store. Based on the counts in these files, I calculated the BOQA scores for different parameter combinations of alpha and beta and determined the number of phenopackets in which the diagnosed disease was ranked 1st. The results are shown in the following heatmap.

For the parameter combination alpha = 0.05 and beta = 0.90, the model performance is best, but for parameter combinations with alpha + beta >= 1, the model performance is very poor.

@pnrobinson In a personal conversation, you claimed that this observation could best be explained by a bug in the analysis script that I used to create the heatmap above.

I tested our Java implementation for some parameter combinations with alpha + beta >= 1, without using this analysis script. Result: Also with our Java implementation, the model performance for such combinations is very poor. This means that if there is a bug in the analysis script, then this bug must also be present in our Java implementation.

As an example, I have attached a JSON file for the parameter combination alpha = 0.5 and beta = 0.6. The following code uses the BOQA output file to determine how often our Java implementation ranked the diagnosed disease first for a specific parameter combination of alpha and beta.

boqa_results_pc_a05b06.json

import json
BOQA_JSON_INPUT_FILE = "boqa_results_pc_a05b06.json"
with open(BOQA_JSON_INPUT_FILE) as f: 
    d = json.load(f)
    print("Alpha: " + str(d["metadata"]["algorithmParams"]["alpha"]))
    print("Beta: " + str(d["metadata"]["algorithmParams"]["beta"]))
    ppkt_total = 0
    ppkt_diagnosed_disease_at_rank_1 = 0
    for ppkt in d["results"]:
        ppkt_total += 1
        diagnosed_disease = ppkt["patientData"]["diagnosis"][0]["id"]
        disease_at_rank_1 = ppkt["boqaResults"][0]["counts"]["diseaseId"]
        if diagnosed_disease == disease_at_rank_1:
            ppkt_diagnosed_disease_at_rank_1 += 1

print("Total number of phenopackets: " + str(ppkt_total))
print("Phenopackets with diagnosed diseases at rank 1: " + str(ppkt_diagnosed_disease_at_rank_1))

Result:

Alpha: 0.5
Beta: 0.6
Total number of phenopackets: 8207
Phenopackets with diagnosed diseases at rank 1: 13