Skip to content

Add a url property to GetDirectoryContentsOptions for easier implementation of link header pagination #412

Description

@star2000

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Link

The story begins with my use of the jianguoyun platform's webdav. This platform limits a single getDirectoryContents query to 750 items, and pagination is handled via an absolute URL containing params, provided in a link response header with rel="next"

This is the patch I made

diff --git a/node_modules/webdav/dist/node/operations/directoryContents.js b/node_modules/webdav/dist/node/operations/directoryContents.js
index c0d7709..ba0c7d0 100644
--- a/node_modules/webdav/dist/node/operations/directoryContents.js
+++ b/node_modules/webdav/dist/node/operations/directoryContents.js
@@ -6,7 +6,7 @@ import { request, prepareRequestOptions } from "../request.js";
 import { handleResponseCode, processGlobFilter, processResponsePayload } from "../response.js";
 export async function getDirectoryContents(context, remotePath, options = {}) {
     const requestOptions = prepareRequestOptions({
-        url: joinURL(context.remoteURL, encodePath(remotePath), "/"),
+        url: options.url || joinURL(context.remoteURL, encodePath(remotePath), "/"),
         method: "PROPFIND",
         headers: {
             Accept: "text/plain,application/xml",
diff --git a/node_modules/webdav/dist/node/types.d.ts b/node_modules/webdav/dist/node/types.d.ts
index f240a6e..e99ff16 100644
--- a/node_modules/webdav/dist/node/types.d.ts
+++ b/node_modules/webdav/dist/node/types.d.ts
@@ -117,6 +117,7 @@ interface GetDirectoryContentsOptions extends WebDAVMethodOptions {
     deep?: boolean;
     glob?: string;
     includeSelf?: boolean;
+    url?: string;
 }
 export interface GetDirectoryContentsOptionsWithDetails extends GetDirectoryContentsOptions {
     details: true;

Thus, querying all pages can be implemented like this:

import { parse as parseLinkHeader } from 'http-link-header';
import type { FileStat } from 'webdav';
import { createClient as createWebdavClient } from 'webdav';

const cloud = createWebdavClient(
  'https://dav.jianguoyun.com/dav/xxxx',
  {
    username: 'xxxx',
    password: 'xxxx',
  },
);
async cloudFiles(path: string) => {
  let files: FileStat[] = [];
  let url: string | undefined;
  do {
    const response = await cloud.getDirectoryContents(path, {
      deep: false,
      details: true,
      includeSelf: false,
      url,
    });
    files = files.concat(response.data);
    if (response.headers.link) {
      const links = parseLinkHeader(response.headers.link);
      const nextLinks = links.rel('next');
      if (nextLinks) {
        url = decodeURI(nextLinks[0].uri);
        continue;
      }
    }
    break;
  } while (true);
  return files;
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions