Declarative UI: audio file -> create a transcript (md)

[tobiasKaminsky]

Describe the feature you'd like to request

Related to nextcloud/files-clients#86 (comment)

With Declarative UI any server app can add interactions dynamically to Files clients.
This reduces code on client side and gives lots of opportunities to display simple stuff in a native way or interact in an easy way.

To have a starting set, we discussed and defined some actions.
One action is to offer an menu entry in context menu to convert an audio file file to md.

To do so:

• add info to capability like https://github.com/tobiasKaminsky/declarativetest/blob/main/lib/Capabilities.php#L29
• create an endpoint that can receive a fileID and create a MD file out of it

Ref: #312

Describe the solution you'd like

See above

Describe alternatives you've considered

/

[julien-nc]

There is already an endpoint. It's the one used by the web frontend when the assistant file actions are triggered:
POST /ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/{taskTypeId}
with {fileId} being the input file and {taskTypeId} being the operation you want to trigger.

You can use it with those task type IDs:

• core:audio2text on audio files
• core:text2speech on text, markdown and office text files
• core:text2text:summary on text, markdown and office text files

You need to check if those task types are available on the instance by using:
GET /ocs/v2.php/taskprocessing/tasktypes
and check that the ocs.data.types object contains the task type ID as a key.

This endpoint takes care of everything:

• scheduling the task
• storing the result in a new file next to the input file (same dir)
• notifying the user

Makes sense?

[tobiasKaminsky]

Sounds great 👍

[julien-nc]

@tobiasKaminsky Some remarks/wishes on the declarative UI capability structure:

• We need to be able to add such filter: text/*, application/pdf, application/vnd.oasis.opendocument.text, ... for the summarize file action because it supports all those mimetypes. We can duplicate the endpoint entry for each filter but it would be much more convenient and consistent with other tools to be able to declare a list of mimetype filters
• filter could be renamed to mimetype_filters
• My suggestion on how filters should be represented:

"mimetype_filters": ["text/*", "application/pdf"]

or

"mimetype_filters": "text/*, application/pdf"

• I'm not sure hooks should be a list because it makes it possible to insert multiple entries with "type": "context-menu" for example. hooks could be an object indexed by the type. Under each type key, there could just be the list of endpoints:

"declarativeui": {
    "hooks": {
        "context-menu": [
            { "name": "entry1", "url": "url1" },
            { "name": "entry2", "url": "url2" },
            { "name": "entry3", "url": "url3" },
        ],
        "create-new": [
            { "name": "entry4", "url": "url4" },
            { "name": "entry5", "url": "url5" },
        ],
    }
}

• We need a placeholder for the fileId in the URLs. {fileId} seems fine to me. URL example:
  `https://nextcloud.local/ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/core:audio2text
• I think we need a "method" param for the entries. The endpoints we reach might already exist and be another method than GET. Let's not enter the rabbit hole to allow passing a request body. Having an empty request body is fine IMO.

[tobiasKaminsky]

"mimetype_filters": "text/*, application/pdf"

Using a commpa separated "list" is good.
But please only use "text/", since we simply test if mimetype starts with the filtername.

• I'm not sure hooks should be a list because it makes it possible to insert multiple entries with "type": "context-menu" for example. hooks could be an object indexed by the type. Under each type key, there could just be the list of endpoints:

I expected that server would combine those.
So if app1 and app2 both thave capabilities -> declarative-ui -> hooks -> context-menu with 1 or 2 entries, I had thought it would be then one "context-menu" with 3 entries.

• We need a placeholder for the fileId in the URLs. {fileId} seems fine to me.

Just add it like this and then I can test ✔

• I think we need a "method" param for the entries.

method sounds fin ✔