SharePoint connector ignores `@odata.nextLink` — folders with more than one page of children are silently truncated

[wichmann-git]

Summary

SharePointConnector._walk_folder issues a single GET .../children request per folder and reads only the value array. Microsoft Graph paginates this endpoint: when a folder has more children than fit in one page, the response carries an @odata.nextLink pointing at the next page. That field is never read, so everything beyond the first page is invisible to the connector.

There is no error. raise_for_status() passes, the sync reports success, and the missing files are indistinguishable — from oikb's point of view — from files that do not exist in SharePoint.

Impact

Worse than "some files are not uploaded": because the manifest drives the diff, files that were synced before the folder crossed the page boundary are now considered deleted at the source and get removed from the Knowledge Base on the next run. A sync that reports N unchanged, M deleted looks healthy while silently shrinking the KB.

Two aggravating factors:

1. /children returns folders and files in one list, so the page budget is shared. A folder with 150 files and 60 subfolders is already past 200 entries without ever holding 200 files.
2. A truncated subfolder entry costs the entire subtree. _walk_folder only recurses into folders it actually saw in value, so one dropped folder entry silently removes every file beneath it, at any depth.

Affected code

src/oikb/connectors/sharepoint.py — identical on main and in the v0.4.0 release image:

def _walk_folder(self, folder_path: str, prefix: str, entries: list[ManifestEntry]) -> None:
    url = f"/drives/{self._drive_id}/root/children" if folder_path == "/" else f"/drives/{self._drive_id}/root:/{folder_path}:/children"
    resp = self._http.get(url)
    resp.raise_for_status()

    for item in resp.json().get("value", []):
        if "folder" in item:
            ...
            self._walk_folder(child_path, sub, entries)
        elif "file" in item:
            ...

Reached from build_manifest(), so every SharePoint sync is affected.

Reproduction

The default page size for driveItem children is 200, so this only bites on large folders. $top reproduces it at any size — the connector's behaviour is identical, only the threshold moves:

# Same request shape the connector makes, against a folder with 187 children.
base = f"https://graph.microsoft.com/v1.0/drives/{drive_id}/root:/{folder}:/children"

for url in (base, base + "?$top=50"):
    page = requests.get(url, headers=auth).json()
    print(len(page.get("value", [])), "@odata.nextLink" in page)

# 187 False   <- fits in one page, connector is correct
#  50 True    <- connector returns 50 of 187 and reports success

Following @odata.nextLink on the same folder yields all 187 entries across 4 pages.

Observed against a real tenant: a document library with 536 files in three monthly folders (154 / 181 / 187 entries). Nothing is lost today; the largest folder is 13 entries from the threshold and grows monthly.

Suggested fix

Loop over pages instead of reading one response:

def _walk_folder(self, folder_path: str, prefix: str, entries: list[ManifestEntry]) -> None:
    url = (
        f"/drives/{self._drive_id}/root/children"
        if folder_path == "/"
        else f"/drives/{self._drive_id}/root:/{folder_path}:/children"
    )

    while url:
        resp = self._http.get(url)
        resp.raise_for_status()
        data = resp.json()

        for item in data.get("value", []):
            if "folder" in item:
                sub = f"{prefix}/{item['name']}" if prefix else item["name"]
                child_path = f"{folder_path}/{item['name']}" if folder_path != "/" else item["name"]
                self._walk_folder(child_path, sub, entries)
            elif "file" in item:
                etag = (item.get("eTag") or item.get("cTag", "")).strip('"')
                entries.append(ManifestEntry(
                    filename=item["name"],
                    path=prefix,
                    checksum=etag[:16] if etag else "",
                    size=item.get("size", 0),
                ))

        url = data.get("@odata.nextLink")

One implementation note: @odata.nextLink is an absolute URL. httpx uses absolute URLs as given even when the client has a base_url, so it can be passed straight back into self._http.get().

Prior art in this repo: the Teams connector talks to Graph and already follows @odata.nextLink, and #69 fixed the same class of bug for Jira. SharePoint looks like an oversight rather than a design decision.

Environment

• oikb v0.4.0 (ghcr.io/open-webui/oikb:0.4.0), daemon mode
• Auth: client secret (SHAREPOINT_CLIENT_ID / SHAREPOINT_TENANT_ID / SHAREPOINT_CLIENT_SECRET), Graph application permission Sites.Read.All
• Source: sharepoint:<site-id>/Documents
• Verified unfixed on main at the time of writing

Only sharepoint.py was examined — other connectors may share the pattern.