_id collision when indexing with Genkit

[dancarvajc]

Hi there — I’m seeing this error when indexing with Genkit:

Document already exists with the given _id.

Looking at commit 3ad7b4f, I noticed you originally changed the behavior to avoid collisions (which makes sense), but then reverted it. As far as I can tell, the document _id is currently derived from the document content.

 _id: Md5.hashStr(JSON.stringify(embeddingDocs[j])),

Would it be possible to leave _id empty and let AstraDB generate a unique ID by default? In my use case, duplicate documents are likely to appear, and that’s fine for now — I’d just prefer not to fail on insert due to _id collisions.

Could you advise on how to enable auto-generated IDs with AstraDB in this flow, or enable the upset optionally to avoid these collisions?

Thanks!

[philnash]

I think I changed this back to closer match the behaviour of the existing database plugins that the Genkit team had built.

With Astra DB, when you perform an insertMany, the documents that fail do not stop the other documents from succeeding. So this insertMany, rather than an updateMany with an upsert, means that you avoid duplicate content, but you don't fail to add new or edited documents if they are present too.

I can't think of a reason to want duplicate content within your vector database. Can you help me understand why that would be a useful feature?

Thanks!

[dancarvajc]

Thanks for the response, it really makes sense what you say about avoiding duplicates… In my project I should probably handle this somehow… here’s the scenario:

User A uploads a document and it gets ingested into AstraDB, then another user B uploads the same document (which is possible), so the same IDs are generated and the ingestion fails. As metadata I store a document ID that I assign from the frontend. So if user B searches this document, it would fail because it wasn’t ingested… How would you recommend solving this? I was thinking about adding the document ID as metadata for user B… but I don’t know how to do it in order to detect and update all the chunks of the document.

Change this metadata:

from:

{ "bookId": "11b7dfad-4365-41dc-9ee8-a49bcb01c260" }

to:

{ "bookId": ["11b7dfad-4365-41dc-9ee8-a49bcb01c260", "8d8081fa-8926-4adf-b038-c09d6b23d641"] }

Then filter in Genkit’s retriever. Is that possible?

Here’s a snippet of my code:

const docs = await ai.retrieve({
  retriever: astraDBRetriever,
  query,
  options: {
    filter: {
      'metadata.bookId': { $eq: bookId },
    },
    k: 5,
  },
});

//...
const rawChunks = await splitter.splitText(text);
const documents = rawChunks.map(chunk => Document.fromText(chunk, { bookId }));
for (let i = 0; i < documents.length; i += 150) {
  await ai.index({ indexer: astraDBIndexer, documents: documents.slice(i, i+150) });
}
//...

[philnash]

You can use the $in filter when your metadata is an array like that.