Nextcloud updater failed to "Move new files in place" when the working directory is in an NFS filesystem that is under anti-malware live threat detection

[brlin-tw]

Steps to reproduce

1. Install an anti-malware/Endpoint Detection and Response sofware(e.g. Kaspersky Endpoint Security for Linux) that is capable of scanning NFS filesystem(this unfortunately rules out ClamAV) and enable their live threat detection feature (e.g. File Threat Protection in KESL).

   Note that the presence of such software is common in enterprise/government deployment environments due to compliance reasons.

2. Mount an NFS filesystem as Nextcloud's data directory and install Nextcloud.

3. In the Overview page in Nextcloud settings, initiate a Nextcloud upgrade using the web updater.

I've implemented a Vagrant-based bug reproducer that can easily reproduce this bug in a virtual environment, refer to https://github.com/brlin-tw/nextcloud-updater-nfs-silly-rename-bug#usage for detailed instructions.

Expected behaviour

The "Move new files in place" step shouldn't fail.

Actual behaviour

The "Move new files in place" step errors with "Could not rmdir <random-dir>"

(The directory updater failed to remove isn't always the same.)

After some investigation with the help of Claude Code, we believe the following series of event causes this problem:

1. Since the updater staging directory (NFS) and the destination folder(EXT4) isn't in the same file system, the rename() call in moveWithExclusions() does a copy-and-unlink operation rather than the real rename(move) operation.

   To rename across filesystems, PHP "fakes it" by calling copy(), unlink(), chown(), and chmod() (not necessarily in that order). -- ben at indietorrent dot org

2. When another process(e.g. the anti-malware/EDR software's realtime threat detection feature) is scanning the file that is being unlinked in the moveWithExclusions() working directory, the NFS client will silly rename the unlinked file as .nfsXXXXXXXX instead of actually removing it. Leaving the directory in a non-empty state until the file is no longer opened by the anti-malware software and is cleaned up by the NFS client.

3. moveWithExclusions() then attempts to remove the working directory as it believes to be empty but not really, the rmdir() call fails with Could not rmdir /path/to/working/dir and interrupts the update process.

We have checked out the code of revision c285f9a of this repository and believed this bug still existed in the main development branch.

Known workarounds

The following are the known workarounds that can avoid this problem:

• Clicking the "Retry update" button several times should eventually work as the anti-malware software should finish scanning the unlinked file and the NFS client will then remove the silly renamed file shortly after, but it is really a bad experience(In my case I skipped the updater entirely and fallback to manual upgrade as I don't know that button exist during the event).
  
• Temporarily disable the anti-malware/endpoint protection software's realtime threat protection feature during the upgrade.
• Choose a different directory as the updater staging dir via the updatedirectory Nextcloud configuration settings.

Potential solutions

The following are the potential fixes of this problem:

• Bail out when the updater staging directory is in NFS, since it isn't really a sane choice anyway.
• Display a warning text regarding the potential disruption from anti-malware like software when the updater staging directory is in NFS.
• Improve the error handling logic of moveWithExclusions() so that it will retry at least for a few times when the rmdir() call fails.

The following 100% vibe-coded patches are provided only for your convenience, feel free to ignore them if they aren't salvageable:

• fix/773-bail-out-on-nfs-staging-dir
• fix/773-warn-on-nfs-staging-dir
• fix/773-retry-rmdir-on-failure

Server configuration

Web server: Apache

Database: Maria

PHP version: 8.2

Nextcloud version: 33.0.5

List of activated apps

Enabled:
  - activity: 6.0.0
  - app_api: 33.0.0
  - bruteforcesettings: 6.0.0
  - circles: 33.0.0
  - cloud_federation_api: 1.17.0
  - comments: 1.23.0
  - contactsinteraction: 1.14.1
  - dashboard: 7.13.0
  - dav: 1.36.0
  - federatedfilesharing: 1.23.0
  - federation: 1.23.0
  - files: 2.5.0
  - files_downloadlimit: 5.1.0
  - files_pdfviewer: 6.0.0
  - files_reminders: 1.6.0
  - files_sharing: 1.25.2
  - files_trashbin: 1.23.0
  - files_versions: 1.26.0
  - firstrunwizard: 6.0.0
  - logreader: 6.0.0
  - lookup_server_connector: 1.21.0
  - nextcloud_announcements: 5.0.0
  - notifications: 6.0.0
  - oauth2: 1.21.0
  - password_policy: 5.0.0
  - photos: 6.0.0
  - privacy: 5.0.0
  - profile: 1.2.0
  - provisioning_api: 1.23.0
  - recommendations: 6.0.0
  - related_resources: 4.0.0
  - serverinfo: 5.0.0
  - settings: 1.16.0
  - sharebymail: 1.23.0
  - support: 5.0.0
  - survey_client: 5.0.0
  - systemtags: 1.23.0
  - text: 7.0.1
  - theming: 2.8.0
  - twofactor_backupcodes: 1.22.0
  - twofactor_totp: 15.0.0
  - updatenotification: 1.23.0
  - user_status: 1.13.0
  - viewer: 6.0.0
  - weather_status: 1.13.0
  - webhook_listeners: 1.5.0
  - workflowengine: 2.15.0
Disabled:
  - admin_audit: 1.23.0
  - encryption: 2.21.0
  - files_external: 1.25.1
  - suspicious_login: 11.0.0
  - twofactor_nextcloud_notification: 7.0.0
  - user_ldap: 1.24.0

Nextcloud configuration

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "192.168.56.20"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "33.0.5.1",
        "overwrite.cli.url": "http:\/\/localhost",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "updater.release.channel": "beta"
    }
}

Browser

Browser name: Firefox

Browser version: 151.0.4

Operating system: Ubuntu 26.04

Browser log

{"proceed":false,"response":"Could not rmdir \/data\/nextcloud-data\/updater-ocv9jfdc5z41\/downloads\/nextcloud\/\/3rdparty\/symfony\/uid"}