Forest vs tree

[deepakpandita57]

Summary of the changes:

• Support for multiprocessing to run trials in parallel and compression to save storage space
• Support for categorical responses and associated metrics
• Methods used to run experiments reported in the Forest vs Tree paper and the EACL paper
• Some minor changes to replace sklearn mean_squared_error() with root_mean_squared_error() as the former method is deprecated

[tripperroc]

Hi Deepak, I’m going to take Samay and Mamadou out for dinner tomorrow, to celebrate submission of our paper. It’s also Mamadou’s birthday and Samay is leaving at the end of the week. Would you like to join us (on me)? We haven’t celebrated our conference attendances (or even acceptances, have we?) C From: Deepak Pandita ***@***.***> Date: Wednesday, February 25, 2026 at 4:46 PM To: google-research/vet ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [google-research/vet] Forest vs tree (PR #2) @deepakpandita57 commented on this pull request.

________________________________ In cat_machine_contest_metrics.py<#2 (comment)>:

+ machine1: np.ndarray,

+ machine2: np.ndarray, +) -> tuple[float, float]: + """Compute accuracy relative to human labels. + + Args: + human: A list of human scores. + machine1: A list of machine scores. + machine2: Another list of machine scores. + + Returns: + A pair of accuracy scores, for machines 1 and 2, relative to + human scores. + """ + + num_categories = np.max(np.concatenate(( Yes, I can use stack and max here. — Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAQNUTLXPLMDH5RAIL5X5VT4NYJZFAVCNFSM6AAAAACUZ4H2ZOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTQNJXGE3TIOJXGE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[pk-at-g]

Give a pointer to the formula that would be used to compute an "actual" (analytic?) p-value.

[deepakpandita57]

This does not refer to the analytic p-values. Here it means that the categorical parameters sampled are directly used for p-value calculation.

[pk-at-g]

Why these default values? Note: they're different from the _ALPHA flag default values.

[deepakpandita57]

No particular reason, except this is the fraction of categories in the DICES dataset. I have changed the flag values to match these, but I'm open to your suggestions.