Skip to content

feat(dataset): add generative CivilComments config - #2625

Open
motodriver wants to merge 1 commit into
open-compass:mainfrom
motodriver:feat/civilcomments-gen-api
Open

feat(dataset): add generative CivilComments config#2625
motodriver wants to merge 1 commit into
open-compass:mainfrom
motodriver:feat/civilcomments-gen-api

Conversation

@motodriver

Copy link
Copy Markdown

Motivation

The existing CivilComments configuration uses CLPInferencer and AUCROCEvaluator, which require class logits. Text-only API models cannot run that path.

Changes

  • keep the existing CLP/AUC configuration unchanged
  • add a separate civilcomments_gen configuration using GenInferencer and AccEvaluator
  • map non-toxic comments to A/No and toxic comments to B/Yes
  • parse only A/B answers with cushion=False
  • expose both stable configurations in dataset-index.yml
  • add offline tests for the 0.5 threshold boundary, label direction, config structure, and CLP compatibility

Verification

  • focused offline tests: 4 passed
  • YAPF: passed
  • Python compilation: passed
  • dataset suffix/hash verification: passed
  • git diff --check: passed

Compatibility

The original CLP configuration and AUC metric remain available. The new generated-answer path reports accuracy and does not attempt to emulate probability-based AUC.

Closes #1416.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] I have a question: why doesn't the civilcomments dataset support API evaluation?

1 participant