model incorrectly not flagging a profane

[ambarberena]

Is missing the word 'bitch' in the following sentence:

"Happy Birthday to Sofía on turning 15, from a ballerina sent on behalf of you bitch Auntie Tee."

PROFANITY results:[[0]]
PROFANITY probabilities:[[0.45764046]]

Python version: Python 3.12.8

Name: alt-profanity-check
Version: 1.9.0

Name: scikit-learn
Version: 1.9.0

[menkotoglou]

Hey @ambarberena,

Thanks a lot for using alt-profanity-check and reporting this.

The good news is that bitch is definitely in the model. If you pass it on its own, it comes back as 1.0000. What's tripping things up here is the length of your sentence.

The way the model works, it looks at all the words in a string and weighs up the evidence across the whole thing. That means a single bad word has to share the spotlight with everything around it. In your sentence there are seventeen perfectly innocent words such as happy, birthday, ballerina, auntie, and they water down the one word that matters until it can't quite push the score over the line. You'd see the same thing with any flagged phrase if you padded it out with enough harmless text.

So it's less that the model missed the word, and more that it got outvoted. That's a known trade-off with this kind of model, it's fast and doesn't rely on a word blacklist, but the flip side is that it has no way of saying "one word here is bad enough on its own, never mind the rest". There's a note about this in the Caveats section of the README.

If you're checking longer text, a couple of things that help: use predict_prob() and pick your own cut-off rather than relying on the default.

We're also exploring the possibility of adding the following features in the future:

• Picking the library threshold manually so that you can loosen or tighten the library's boundaries.
• Potentially add a list that will flag some words out of the box and will always set the sentence as profane, if these words exist in the sentence.