feat: add embed task for text embeddings

[Nina-mvH]

Adds an initial implementation of an embed task (currently only supports text inputs and .npy outputs)

Tested with sentence-transformers/all-MiniLM-L6-v2

Deferred

• Support image inputs (feat(embed): add support for img and pdf inputs #180)
• Support audio inputs (feat(embed): add support for audio inputs #181)
• Support video inputs
• Support saving embedding to json output
• Integration tests

Open design question

This task does not fall neatly into the text grouping -- it will support all input types and outputs embeddings; it seems like the task should not live in a sub-folder?

[cswaney]

Approved for merge after the following updates:

• Remove per-line
• Replace prompt and prompt-name with kwargs
• Add use_encode_document = False params to top into encode_document. Also, explains what encode_document does more precisely.

[Nina-mvH]

What encode_document does

If the provided model has a Router module and was trained to embed "documents" differently from "queries", using encode_document routes inputs to a processing modules which is specifically trained to embed documents (inputs are automatically tagged as "document").

So, when the documentation explains:

This method is a specialized version of encode() that differs in exactly two ways:

1. If no prompt_name or prompt is provided, it uses the first available prompt from the following candidates: "document", "passage", "corpus" (checked in that order).
2. It sets the task to “document”. If the model has a Router module, it will use the “document” task type to route the input through the appropriate submodules.

This means:

1. The input text is tagged as "document" (or "passage"/"corpus" depending on what the model recognizes). This will tell the router that the input is the passage/corpus side of a retrieval pair.
2. The router (if the model has one) will recognize that the input is a "document" and will send the input to a specialized submodule which is set up specifically for the document role. This would be different from the "query" submodule or generic encoding. The differences between these submodules are model dependent.

When to use it

Using encode_document primarily makes sense when you want to encode documents with the intention of querying them (with encode_query). If you are simply encoding a long input (for purposes other than retrieval), just use encode