Describe the feature
Current LLM evaluation benchmarks in OpenCompass focus heavily on English-centric tasks or surface-level semantic understanding, while fine-grained syntactic structure evaluation for morphologically rich languages (such as Turkish) remains limited.
In particular, existing benchmarks do not adequately evaluate a model’s ability to:
represent embedded clause structures
track nominalization (-DIK structures)
resolve pro-drop subjects
maintain cross-clause dependency relations
preserve syntactic function mapping in interrogative constructions
To address this gap, I propose a structured DIK-style syntactic annotation dataset for Turkish, designed as a pilot benchmark that can be integrated into OpenCompass evaluation pipelines.
The dataset consists of fully structured JSON annotations of Turkish sentences, where each sentence encodes:
sentence type (interrogative / declarative / etc.)
main clause decomposition
embedded DIK-complement clause structure
verb morphology (tense, mood, person, lemma)
syntactic roles (subject, object, adverbials)
pro-drop resolution
case marking and nominalization tracking
Example structure:
{
"text": "...",
"sentence_type": "...",
"main_clause": {
"subject": "...",
"verb": {...},
"object": {
"type": "DIK_complement_clause",
"embedded_clause": {...}
}
}
}
A pilot set of 20 fully standardized sentences has been completed, maintaining a consistent annotation schema throughout, with manually validated syntactic decomposition and a scalable design that does not require schema modification.
If this pilot is positively evaluated, the dataset will be expanded to:
100–300 high-quality sentences (Phase 1)
optional extension to multi-language morphologically rich benchmarks (Phase 2)
integration into structured reasoning / syntax evaluation tasks
This feature enables OpenCompass to:
evaluate syntactic reasoning beyond surface semantics
benchmark LLM performance on morphologically rich languages
support structured linguistic evaluation tasks
extend evaluation diversity beyond English-centered datasets
Will you implement it?
Describe the feature
Current LLM evaluation benchmarks in OpenCompass focus heavily on English-centric tasks or surface-level semantic understanding, while fine-grained syntactic structure evaluation for morphologically rich languages (such as Turkish) remains limited.
In particular, existing benchmarks do not adequately evaluate a model’s ability to:
represent embedded clause structures
track nominalization (-DIK structures)
resolve pro-drop subjects
maintain cross-clause dependency relations
preserve syntactic function mapping in interrogative constructions
To address this gap, I propose a structured DIK-style syntactic annotation dataset for Turkish, designed as a pilot benchmark that can be integrated into OpenCompass evaluation pipelines.
The dataset consists of fully structured JSON annotations of Turkish sentences, where each sentence encodes:
sentence type (interrogative / declarative / etc.)
main clause decomposition
embedded DIK-complement clause structure
verb morphology (tense, mood, person, lemma)
syntactic roles (subject, object, adverbials)
pro-drop resolution
case marking and nominalization tracking
Example structure:
{
"text": "...",
"sentence_type": "...",
"main_clause": {
"subject": "...",
"verb": {...},
"object": {
"type": "DIK_complement_clause",
"embedded_clause": {...}
}
}
}
A pilot set of 20 fully standardized sentences has been completed, maintaining a consistent annotation schema throughout, with manually validated syntactic decomposition and a scalable design that does not require schema modification.
If this pilot is positively evaluated, the dataset will be expanded to:
100–300 high-quality sentences (Phase 1)
optional extension to multi-language morphologically rich benchmarks (Phase 2)
integration into structured reasoning / syntax evaluation tasks
This feature enables OpenCompass to:
evaluate syntactic reasoning beyond surface semantics
benchmark LLM performance on morphologically rich languages
support structured linguistic evaluation tasks
extend evaluation diversity beyond English-centered datasets
Will you implement it?