What is the best way to look for similar texts when I have input text, its keywords and named entities and also input text embedding? I have the same for all texts in the dataset. I need to find the closest relative texts. Texts that are about the same person or location (i.e. same NE) or about the same events (same keywords maybe) must have the highest similarity score.
This is done now in util.kwne_similarity.py and generate_context function in context_gen.py, but does not work really well. Sometimes one of top relatives texts are texts that have similar theme to the input text but have totally different subject.
For example, my input text is about Covid. Between top relative texts, there are texts that also have health and medicine theme but these texts are not about Covid. And these non-covid texts still have higher similarity score than some Covid texts from dataset.
What is the best way to look for similar texts when I have input text, its keywords and named entities and also input text embedding? I have the same for all texts in the dataset. I need to find the closest relative texts. Texts that are about the same person or location (i.e. same NE) or about the same events (same keywords maybe) must have the highest similarity score.
This is done now in
util.kwne_similarity.pyandgenerate_contextfunction incontext_gen.py, but does not work really well. Sometimes one of top relatives texts are texts that have similar theme to the input text but have totally different subject.For example, my input text is about Covid. Between top relative texts, there are texts that also have health and medicine theme but these texts are not about Covid. And these non-covid texts still have higher similarity score than some Covid texts from dataset.