Large difference in score from official site

[Zarxrax]

I was attempting to use Jreadability to score stories from hukumusume.com, and as I started comparing against the official implementation on https://jreadability.net/sys/en I noticed huge differences in scores between it and this python version.

For example, on one text, I get a score of 2.46 on yours, while the official site gives 4.2. That's a massive difference!

Here is the exact text that was passed in:

福娘童話集 > きょうのイソップ童話 > ６月のイソップ童話 > ハチとシャコとお百姓

6月4日のｲｿｯﾌﾟ童話

　ハチとシャコが、喉が乾いてたまらないので、

　お百姓の所へ水をもらいに行きました。

「水を少しくれれば、その代わりにシャコはブドウ畑を耕しますし、

　ハチは泥棒を刺して追い払ってあげましょう」
と、言うと、お百姓は、

「わしにはウシが２頭いて、

　奴らは何も約束しないのに何でもやってくれる。

　だから、あんたたちに何かくれてやるよりも、

　ウシたちに何かをやるほうがいいよ」

　このお話しは、口先だけの約束で先に報酬をもらおうとする人を、けっして信用してはいけないという事を教えています。

おしまい

隠しキャラの正解

おまけのメッセージカード
(画面をクリックすると印刷用の大サイズ(2880px×2160px)になります。

Do you have any thoughts on why there is such a difference in scoring?

[joshdavham]

This is a known issue for some text passages and it’s likely due to the underlying dictionary being used. See the README: https://github.com/joshdavham/jreadability#note-on-model-consistency

Please feel free to open a PR also if you’re up for trying to solve this :)

[Zarxrax]

Haha, yeah I saw that note that the scores could differ "slightly", but this seemed like much more than slightly, so I suspected it could be something else.

I took a look at the code, and it was surprisingly much more straightforward than I expected. However, one thing immediately jumped out at me. I see your algorithm is using {percentage of particles} * -0.044
However the official paper uses {proportion of number of auxiliary verbs * -0.044 }

Am I missing something, or is it using the entirely wrong part of speech?

Another thing I just noticed is that punctuation seems to have an extremely large impact on your implementation.
Simply adding a period 。to the lines that didn't end with one significantly increased the score, while it just slightly increased the score on the official implementation.