Merge pull request #15354 from nextcloud/more-task-types

docs(taskprocessing): Document improve, and multimodal task types

Author: lukasdotcom

developer_manual/digging_deeper/task_processing.rst

@@ -49,6 +49,18 @@ The following built-in task types are available:
     * Output shape:
        * ``output``: ``Text``
        * ``tool_calls``: ``Text`` A string containing a JSON array with ``{"name": string, "type": "tool_call", "id": string, "args": object}``
+ * ``'core:text2text:multimodal-chatwithtools'``: This task allows chatting with the language model with tools calling support and file attachments. It is implemented by ``\OCP\TaskProcessing\TaskTypes\MultimodalChatWithTools``
+    * Input shape:
+       * ``system_prompt``: ``Text``
+       * ``input``: ``Text``
+       * ``input_attachments``: ``ListOfFiles`` Files to send along with the chat message
+       * ``tool_message``: ``Text`` A string containing a JSON array of ``{"name": string, "content": string, "tool_call_id": string}``
+       * ``history``: ``ListOfTexts`` Each list item is a JSON string with ``{"role": "human", "content": string}`` or ``{"role": "assistant", "content": string, (optional: "tool_calls": array<{"name": string, "type": "tool_call", "id": string, "args": object}>)}`` or ``{"role": "tool", "content": string, "name": string, "tool_call_id": string}``
+       * ``tools``: ``Text`` The tools parameter should be a JSON array formatted according to the OpenAI API specifications: https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools
+    * Output shape:
+       * ``output``: ``Text``
+       * ``output_attachments``: ``ListOfFiles`` Files generated by the model
+       * ``tool_calls``: ``Text`` A string containing a JSON array with ``{"name": string, "type": "tool_call", "id": string, "args": object}``
  * ``'core:contextagent:interaction'``: This task allows chatting with an agent. It is implemented by ``\OCP\TaskProcessing\TaskTypes\ContextAgentInteraction``
     * Input shape:
        * ``input``: ``Text``
@@ -58,6 +70,19 @@ The following built-in task types are available:
        * ``output``: ``Text``
        * ``conversation_token``: ``Text``
        * ``actions``: ``Text``
+ * ``'core:contextagent:multimodal-interaction'``: This task allows chatting with an agent using file attachments. It is implemented by ``\OCP\TaskProcessing\TaskTypes\MultimodalContextAgentInteraction``
+    * Input shape:
+       * ``input``: ``Text``
+       * ``input_attachments``: ``ListOfFiles`` Files to send along with the chat message
+       * ``confirmation``: ``Number`` Boolean integer indicating whether to confirm previously requested actions: 0 to reject or 1 to confirm.
+       * ``conversation_token``: ``Text`` Token representing the conversation
+       * ``memories``: ``ListOfTexts`` Memories to send to the agent
+    * Output shape:
+       * ``output``: ``Text``
+       * ``output_attachments``: ``ListOfFiles`` Files generated by the agent
+       * ``conversation_token``: ``Text``
+       * ``actions``: ``Text``
+       * ``sources``: ``ListOfTexts`` Sources of the generated response
  * ``'core:text2text:formalization'``: This task will reformulate the passed input text to be more formal in tone. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToTextFormalization``
      * Input shape:
         * ``input``: ``Text``
@@ -68,6 +93,12 @@ The following built-in task types are available:
         * ``input``: ``Text``
      * Output shape:
         * ``output``: ``Text``
+ * ``'core:text2text:improve'``: This task will improve the passed input text based on instructions. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToTextImprove``
+     * Input shape:
+        * ``input``: ``Text``
+        * ``instructions``: ``Text``
+     * Output shape:
+        * ``output``: ``Text``
  * ``'core:text2text:reformulation'``: This task will reformulate the passed input text arbitrarily. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToTextReformulation``
      * Input shape:
         * ``input``: ``Text``