Discussions on Automated Metadata Enrichment - LLMMetadataExtractor

[davidsbatista]

This is the discussion board for a component supporting automatic metadata enrichment.

Currently, there's one component that allows to automatic extract metadata from a Document based on an LLM prompt: LLMMetadataExtractor

Let us know what you think, and if you have any questions about this feature.

Note: Experimental features live in this repository for a fixed period of time. We don't guarantee that we will continue maintaining experimental features. But if they are successful, stable and if you like it, we will move the feature to the core Haystack package.

PS: The first version of this component was implemented by @davidsbatista 🚀

[d-kleine]

For me, this looks really good!

There is a small nitpick in the example code:

...
docs = [
        Document(content="deepset was founded in 2018 in Berlin, and is known for its Haystack framework"),
        Document(content="Hugging Face is a company founded in Paris, France and is known for its Transformers library")
    ]
...

HF has been founded in New York, USA - and I believe it still has its headquarters there (but afaik, most of their developers work from Paris, France).

[d-kleine]

Thanks! After running the notebook, I'd like to share a few observations:

• With regard to LLMMetadataExtractor component:

  • Technically it works fine, easy to understand and to implement imho 👍🏻

• With regard to code example:

  • In the example, some entities will be extracted that have not been specified in the prompt before, e.g. city.

  Siemens is a German multinational conglomerate company headquartered in Munich and Berlin, founded in 1847 by Werner von Siemens
  {'entities': [{'entity': 'Siemens', 'entity_type': 'company'}, {'entity': 'Germany', 'entity_type': 'country'}, {'entity': 'Munich', 'entity_type': 'city'}, {'entity': 'Berlin', 'entity_type': 'city'}, {'entity': 'Werner von Siemens', 'entity_type': 'person'}]}
  

  • The prompt also contains some contradictions, e.g. that the output should be capitalized:

  - entity_name: Name of the entity (capitalized)
  

  But the provided example output for the entity name isn't capitalized:

  output:
      {"entities": [{"entity": "Visa", "entity_type": "company"}, ... ]}
  

  • HF example has wrong foundation location (foundation: Paris, France -> NY, USA); Also applies to the extractor tests

  -> Source: https://www.crunchbase.com/organization/hugging-face

  • Typo:
    Pegeout is a French automotive manufacturer ... -> Peugeot

[d-kleine]

I just have noticed another detail in the example code:

Pegeout is a French automotive manufacturer that was founded in 1810 by Jean-Pierre Peugeot
{'entities': [{'entity': 'Peugeot', 'entity_type': 'company'}, {'entity': 'Jean-Pierre Peugeot', 'entity_type': 'person'}]}

The input appears to be misspelled as Pegeout, but the LLM correctly predicts Peugeot for the metadata. This kind of autocorrection could be critical in certain cases, such as when the input data and metadata need to match exactly.

Perhaps the prompt could be refined to ensure the original input data is always used, or alternatively, the system could return a warning when the LLM’s prediction for the metadata differs from the input. What are your thoughts on this?

In any case, I think it would be beneficial to reflect this in the documentation explaining how the component handles input variations and LLM autocorrections, as well as any potential implications for data consistency.

[sjrl]

That's a good catch! I agree that the prompt could be refined for such a case which is why any prompt can be passed in by a user. This allows great flexibility since it's possible that some people would like spelling errors to be corrected whereas in other cases not.

We could consider trying to add a warning to check if the metadata extraction doesn't match anything in the input, but it's possible that's intended. For example, I may want to do topic classification where I classify if this text is about news. Even though news doesn't appear anywhere in the text the classification could still be correct.

[d-kleine]

Yeah, I have raised this issue as I saw this case from the example code, but there are certainly cases were the matching should not happen. Maybe it would also be better to spin-off such function in a separate input-metadata validation component instead of having it directly implemented as an arg in the LLMMetadataExtractor itself.

[d-kleine]

@julian-risch @davidsbatista I have added a few more observations regarding the provided code example — I hope this helpsl! 🙂

[sjrl]

Hey @davidsbatista could we go ahead and get this moved to main Haystack? We are needing it for multiple clients on dC.

[davidsbatista]

green light from my side; what do you say, @julian-risch ?

[julian-risch]

Had a chat with David and we agreed to add it to the milestone for the next release: deepset-ai/haystack#8785
We might change the component to use ChatGenerators instead of Generators though.

[sjrl]

Yeah totally fine on the ChatGenerators!

Just watch out for the from_dict method. I realized while working on this Haystack doesn't allow us to easily reuse the from_dict methods of existing components like the Generators which is why I had to create one that basically combined all supported Generators. Could be worth revisiting this to see if there is a way we could reuse the from_dict methods like we do for to_dict.

[davidsbatista]

The LLMMetadaExtractor was integrated in Haystack 2.11.0