DO Spaces Client - No candidate objects found

[ndribeiro]

Summary

The DigitalOcean Spaces client (tool_objectfs\local\store\digitalocean\client) never initializes $maxupload, causing get_maximum_upload_size() to return null. This silently breaks the candidate-selection query used by push_objects_to_storage (and likely other manipulators that filter on filesize < :maximum_file_size), so no objects are ever selected for transfer to remote storage, regardless of file size, age, or any other setting. There is no error or warning — the task simply reports "No candidate objects found" forever.

Environment

• Plugin branch: MOODLE_404_STABLE
• Moodle version: 5.1.4 (Build: 20260420)
• PHP: 8.3 (erseco/alpine-moodle Docker image)
• Database: MariaDB 10.11.13
• Storage backend: DigitalOcean Spaces
• $CFG->alternative_file_system_class = '\tool_objectfs\digitalocean_file_system';

Root cause

In classes/local/store/digitalocean/client.php:

public function __construct($config) {
    if ($this->get_availability() && !empty($config)) {
        $this->bucket = $config->do_space;
        $this->set_client($config);
    } else {
        parent::__construct($config);
    }
}

public function get_availability() {
    return true;
}

get_availability() is hardcoded to always return true. As a result, for any normally-configured client, the if branch always executes and parent::__construct($config) is never called — including on a fully valid config with credentials present.

Compare this to the parent S3 client (classes/local/store/s3/client.php), where $maxupload = OBJECTFS_BYTES_IN_TERABYTE * 5; is set inside its own constructor. Since the DigitalOcean subclass bypasses that constructor entirely, $maxupload (declared in object_client_base.php) is never assigned and stays null.

Every other backend client explicitly sets $maxupload in its own constructor:

• s3/client.php: $this->maxupload = OBJECTFS_BYTES_IN_TERABYTE * 5;
• swift/client.php: $this->maxupload = OBJECTFS_BYTES_IN_TERABYTE * 5;
• azure_blob_storage/client.php: $this->maxupload = api::MAX_BLOCK_SIZE;
• azure/client.php: $this->maxupload = \MicrosoftAzure\Storage\Common\Internal\Resources::MAX_BLOCK_BLOB_SIZE;
• digitalocean/client.php: never set

How this breaks candidate selection

In classes/local/object_manipulator/candidates/pusher_candidates.php:

public function get_candidates_sql() {
    return 'SELECT contenthash,
                   filesize
              FROM {tool_objectfs_objects}
             WHERE filesize > :threshold
               AND filesize < :maximum_file_size
               AND timeduplicated <= :maxcreatedtimestamp
               AND location = :object_location';
}

public function get_candidates_sql_params() {
    $filesystem = new $this->config->filesystem();
    return [
        'maxcreatedtimestamp' => time() - $this->config->minimumage,
        'threshold' => $this->config->sizethreshold,
        'maximum_file_size' => $filesystem->get_maximum_upload_filesize(),
        'object_location' => OBJECT_LOCATION_LOCAL,
    ];
}

get_maximum_upload_filesize() delegates to the client's get_maximum_upload_size(), which returns null. In MySQL/MariaDB, filesize < NULL evaluates to unknown (never true) for every row, so the query returns zero candidates unconditionally — independent of sizethreshold, minimumage, region, credentials, or enabletasks. This makes the bug very difficult to diagnose from configuration alone, since every setting can be correct and the task will still silently do nothing.

Steps to reproduce

1. Configure tool_objectfs with the DigitalOcean Spaces backend (valid do_key, do_secret, do_region, do_space).
2. Set $CFG->alternative_file_system_class = '\tool_objectfs\digitalocean_file_system'; in config.php.
3. Confirm the connection status check on the settings page reports success.
4. Upload any file into Moodle (any size).
5. Run php admin/cli/scheduled_task.php --execute='\tool_objectfs\task\push_objects_to_storage'.

Expected: eligible files are transferred to the Space.
Actual: "No candidate objects found," regardless of sizethreshold/minimumage values (confirmed down to 0 for both), regardless of file size (reproduced on files from 861 bytes to 32MB).

Suggested fix

Explicitly initialize $maxupload in the DigitalOcean client's constructor, matching the pattern used by every other backend:

public function __construct($config) {
    if ($this->get_availability() && !empty($config)) {
        $this->bucket = $config->do_space;
        $this->set_client($config);
        $this->maxupload = OBJECTFS_BYTES_IN_TERABYTE * 5;
    } else {
        parent::__construct($config);
    }
}

Applying this fix (verified locally) immediately resolves candidate selection — a subsequent push_objects_to_storage run correctly identified and transferred previously-stuck local files to Spaces.

Scope / impact

This affects any Moodle site using tool_objectfs with the DigitalOcean Spaces backend. Since get_maximum_upload_size() is used generically by the manipulator/candidates layer, other manipulators that filter on this value (not just push_objects_to_storage) are likely affected the same way. Given the file's author byline differs from the plugin's primary maintainers, this may be a contributed backend that hasn't received the same test coverage as S3/Azure/Swift.

Secondary issue: the region list is hardcoded and out of date (missing sfo3, syd1, blr1)

In the same file, define_client_section():

$regionoptions = [
    'sfo2'      => 'sfo2 (San Fransisco)',
    'nyc3'      => 'nyc3 (New York City)',
    'ams3'      => 'ams3 (Amsterdam)',
    'sgp1'      => 'spg1 (Singapore)',
    'fra1'      => 'fra1 (Frankfurt)',
];

This list predates several regions DigitalOcean Spaces has supported directly for some time (sfo3, syd1, blr1 at minimum). Since this is a plain admin_setting_configselect, there is no way to select an unlisted region through the Moodle admin UI at all — the dropdown simply doesn't offer it, regardless of what's actually valid on DigitalOcean's side.

Our Spaces bucket lives in sfo3 (matching our droplets' region, both for latency and to use DigitalOcean's free intra-region private bandwidth). To work around this, we had to set do_region directly via admin/cli/cfg.php --set=sfo3, bypassing the settings page entirely, and then lock it with $CFG->forced_plugin_settings — otherwise the admin UI, which cannot display or re-select a value outside its own option list, silently reverts it back to sfo2 the next time the settings page is saved for any unrelated change.

This is functional once forced (the underlying value is a plain string, and the S3-compatible endpoint is built from it directly at connection time), but it leaves the setting unmanageable through the normal admin UI for anyone on a region newer than this hardcoded list, with no indication to an admin unfamiliar with the CLI config tool that this is even possible to fix.

Suggested fix

Update $regionoptions to include all current DigitalOcean Spaces regions (nyc3, ams3, sgp1, sfo3, fra1, blr1, syd1), and consider deprecating the legacy sfo2 in favor of sfo3 per DigitalOcean's own guidance to prefer newer regions. Also, unrelated typo in the existing list: sgp1's label reads 'spg1 (Singapore)'.

Related reports

This appears to be a longstanding, previously undiagnosed issue. Two older reports describe the identical symptom — files not transferring to a DigitalOcean Space despite correct-looking configuration, with the presigned-URL connection test passing (consistent with that test using a separate code path that never touches the candidate-selection query):

• [#416 "HELP Digitalocean space"](HELP Digitalocean space #416) (2021) — "Pre-Signed URL Test runs well, it uploads some files into the space but any other files are [not] uploaded."
• [#421 "Moodle 3.10 - ObjectFS"](Moodle 3.10 - ObjectFS  #421) (2021) — files not synchronizing to a DigitalOcean Space.

Neither thread appears to have reached a root cause. Given $maxupload has apparently never been set in this client since it was introduced, it's likely these are the same bug, undiagnosed at the time.

There's also a related history of settings drift specifically in the DigitalOcean client: [#318](#318) reported that the DO client was missing settings present in the S3 implementation, addressed in [PR #323](#323) — though that PR is still unmerged and targets the long-deprecated DEPRECATED_master branch, so it's unclear whether its fixes (or this specific gap) ever made it into currently maintained branches like MOODLE_404_STABLE.

[giselebrugger]

I believe I found the root cause of the issue where push_objects_to_storage always reports "No candidate objects found.", even though there are many valid candidates in the database.

Environment
Moodle: 5.0.8+ (Build: 20260722)
PHP: 8.4.23
MySQL: 8.4.10
Storage backend: Linode Object Storage using \tool_objectfs\digitalocean_file_system
Problem

The scheduled task:

php admin/cli/scheduled_task.php --execute="\tool_objectfs\task\push_objects_to_storage"

always reported:

No candidate objects found.

However, running the equivalent SQL manually returned more than 190,000 candidate objects.

Investigation

I added some debugging to manipulator_candidates_base::get():

$sql = $this->get_candidates_sql();
$params = $this->get_candidates_sql_params();

mtrace($sql);
mtrace(print_r($params, true));

The generated parameters were:

Array
(
[maxcreatedtimestamp] => 1784856163
[threshold] => 10240
[maximum_file_size] =>
[object_location] => 0
)

Notice that maximum_file_size is NULL.

The resulting SQL becomes effectively:

WHERE filesize > 10240
AND filesize < NULL
AND timeduplicated <= ...
AND location = 0

Since filesize < NULL is never true in MySQL, the query always returns zero rows.

Temporary workaround

Replacing:

'maximum_file_size' => $filesystem->get_maximum_upload_filesize(),

with:

'maximum_file_size' => PHP_INT_MAX,

immediately fixed the problem.

The same task then reported:

=== TOTAL ===
10000

and the upload process started selecting candidate objects correctly.

Root cause

The problem appears to be that:

$filesystem->get_maximum_upload_filesize()

returns NULL.

This value is passed directly into the SQL query without validation.

Suggested fix

The plugin should guard against a NULL return value before building the query, for example:

$maximumfilesize = $filesystem->get_maximum_upload_filesize();

if ($maximumfilesize === null) {
$maximumfilesize = PHP_INT_MAX;
}

or ensure that get_maximum_upload_filesize() always returns a valid integer.

Passing NULL into the SQL query causes all candidates to be filtered out, making push_objects_to_storage appear to find no eligible objects even when many exist.