CareEval

Evaluation benchmark for CareEval: assessing LLM decision-making in physical robot caregiving across Activities of Daily Living (ADLs).

Models receive a caregiving scenario and provide a free-form response. A separate grader LLM checks which predefined actions are present. Scores accumulate based on matched actions (positive for safe, negative for harmful).

Data Access

To prevent data contamination in LLM training sets, we provide access to the full benchmark data via a brief access form: https://forms.gle/Lkuwt81tbUx7PBCo9

We aim to provide access shortly after receiving your response.

Setup

pip install -r requirements.txt
cp .env.example .env   # add your API keys

Usage

Configure models and settings in eval_config.yaml, then run:

python -m careeval.evaluate

Options:

• --config PATH — config file (default: eval_config.yaml)
• --provider NAME — only run one provider (openai, anthropic, google, together)

Citation

@InProceedings{HRI26p57,
  author    = {Ziang Liu and Katherine Dimitropoulou and Christy Cheung and Tapomayukh Bhattacharjee},
  title     = {CareEval: Evaluating Large Language Models for Decision-Making in Physical Robot Caregiving},
  booktitle = {Proc.\ HRI},
  publisher = {ACM},
  pages     = {57-62},
  doi       = {10.1145/3776734.3794354},
  year      = {2026},
}

License

BSD 3-Clause License