Skip to content

Fix English connector boundary scoring - #40

Merged
aoiminamoto merged 2 commits into
mainfrom
fix/english-connector-boundaries
Aug 24, 2026
Merged

Fix English connector boundary scoring#40
aoiminamoto merged 2 commits into
mainfrom
fix/english-connector-boundaries

Conversation

@aoiminamoto

Copy link
Copy Markdown

Summary

  • count English discourse connectors as complete normalized token sequences instead of raw substrings
  • preserve repeated connector counts and support future multi-word connectors
  • keep Japanese and Chinese connector behavior language-specific
  • increment the scoring model version and prevent the trend UI from mixing model versions
  • document the corrected metric definition and version behavior

Validation

  • Candy is a gift. now reports connectors=0 and score=14.0
  • capitalization, punctuation, repeated connectors, and multi-word token sequences have regression coverage
  • version-filtered history has regression coverage
  • 29 Aoi-maintained application tests pass

The repository-wide legacy test suite could not be collected in the local environment because optional dependencies (fastapi, speech_recognition, and hypothesis) are not installed. The complete aoi_kinabot_app suite, excluding only the FastAPI test that requires the missing fastapi package, passes.

Closes #39

@aoiminamoto
aoiminamoto merged commit 047c947 into main Aug 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Sentence Complexity counts connector substrings inside English words

1 participant