Improve group assistant feature

[cccaballero]

The current group assistant feature relies on a very basic heuristic: detecting the presence of a question mark (?) to identify questions. This leads to inaccurate question detection and misses many genuine inquiries.

Problem:

• The current question detection is too simplistic, resulting in false positives (e.g., rhetorical questions or questions within quotes) and false negatives (questions without a question mark, or complex sentence structures).
• The assistant frequently fails to differentiate between general group questions and questions directed at specific individuals.
• The assistant lacks contextual awareness of the ongoing conversation.

Proposed Solution:

Implement a two-layered, multilingual, context-aware question detection mechanism:

1. Initial Multilingual Detection:

   • Continue using the basic ? check as a preliminary filter.
   • Implement language detection to identify the language of the message.
   • Extend the initial detection by using a multilingual NLP model to analyze the message's structure and semantics. This should help identify questions even without a question mark.
   • Implement rules for common question phrases (e.g., "who", "what", "where", "when", "why", "how", "can anyone", "does anyone know", etc.) in multiple languages.
   • Use a multilingual tokenizer to handle different word structures.

2. Context-Aware Model-Based Confirmation:

   • After the initial detection, send the identified potential question, along with the last n messages (e.g., 5-10) from the group chat, to the language model.
   • Prompt the model to determine:
     • Whether the message is indeed an open question, considering the provided context.
     • Whether the question is directed at the group as a whole, rather than a specific individual, considering the provided context.
   • If the model confirms that the message is an open group question, trigger the group assistant to generate and provide a response.
   • If the model responds negatively, do not respond.

Benefits:

• Increased accuracy in question detection across multiple languages.
• Improved relevance of the group assistant's responses.
• Reduced instances of the assistant responding to non-question messages.
• More intelligent responses, thanks to contextual awareness.
• Support for a wider user base.

[codeshard]

Kinda have a working solution for this ;)

[codeshard]

What I've tested so far:

• Continue using the basic ? check as a preliminary filter.
  • tl;dr
• Implement language detection to identify the language of the message.
  • pretty straightforward using spacy_langdetect
• Extend the initial detection by using a multilingual NLP model to analyze the message's structure and semantics. This should help identify questions even without a question mark.
  • using sentencizer in the spacy pipeline helps a lot.
• Implement rules for common question phrases (e.g., "who", "what", "where", "when", "why", "how", "can anyone", "does anyone know", etc.) in multiple languages.
  • kinda have a working version for English
• Use a multilingual tokenizer to handle different word structures.
  • the one provided by spacy just works, xx_ent_wiki_sm.

Maybe and just maybe I'll have a PoC within a few days, gonna try make it for eng/esp so far.