Submission interface and dataset license for third-party attacks?

[sattyamjjain]

Hi, reading the RoboJailBench paper and this repo together.

The paper describes an evolving repository with standardised metrics and a unified process for assessing and integrating new attacks and defenses. The repo currently has the dataset, the croissant metadata, and the leaderboard page, and I could not find the submission side of that. Two specific questions:

1. What is the intended interface for a third party submitting an attack? A PR adding a row in a documented schema, an issue with results attached, or something else you have in mind but have not written up yet?

2. There is no LICENSE file. What governs the dataset and the results?

Context for why I am asking rather than just waiting: I maintain an open-source red-team harness for VLA policies, and I have results in a compatible shape, meaning paired benign controls and Wilson intervals per attack family. Your motivating gap, that prior work emphasises attack success while neglecting the trade-off against following benign commands, is the same problem I built the benign-FPR calibration for. Different layer, though: you evaluate VLM planners; I run closed-loop low-level policies, so the numbers are complementary rather than comparable.

Happy to just follow whatever process you land on. Mostly wanted to flag that someone is waiting on it.

[sattyamjjain]

Following up on this one. Still happy to do the work rather than just ask for it.

If the submission interface is not built yet, I can open a PR with a minimal schema for third-party attack entries, modelled on what you already store per row, and you can throw it away if the shape is wrong. Same for the LICENSE question: a one-line answer is enough for me to know whether I can reference the taxonomy in a crosswalk doc.

Context for why I care: I maintain an open-source VLA red-team tool that reports an attack success rate with a benign control arm, and your 18-category taxonomy is the closest thing to a shared vocabulary anyone has published. I would rather crosswalk to it than invent a parallel one.