[feature:gsoc26] Add Mass Command asynchronous execution pipeline for batch device commands

[dee077]

Is your feature request related to a problem? Please describe.

Currently, in the controller module, command execution is limited to a single device at a time through the Command model and the existing asynchronous launch_command task. While this works well for individual device management, it becomes inefficient for operational scenarios where administrators need to run the same command across dozens or hundreds of devices, such as checking uptime, verifying interface status, or collecting diagnostics across an organization, device group, or location.

At the moment, there is no backend abstraction for grouping command executions under a single operation, tracking the overall execution state of a batch, or associating individual command executions with a parent operation. This makes large-scale command execution difficult to manage, monitor, and extend through admin views or APIs.

Describe the solution I would implement

I would like to introduce a backend foundation for mass command execution by adding a new MassCommand model and execution pipeline that builds on top of the existing single-command workflow.

1. Introduce an abstract AbstractMassCommand model (and concrete MassCommand) to represent a batch command operation and support device targeting using filters such as:

   • all devices
   • organization
   • device group
   • location
   • manual device selection

2. Store command metadata such as:

   • command type
   • command input
   • execution status (idle, processing, success, failed)

3. On creation of a MassCommand, schedule a new asynchronous Celery task launch_mass_command using transaction.on_commit. The batch task will:

   • create individual Command objects for each device
   • reuse the existing launch_command execution pipeline for actual device execution
   • Aggregate execution progress at the MassCommand level based on the status of child commands.

4. Add a new WebSocket endpoint to listen to real-time status updates for mass command execution, allowing the frontend to receive and display command progress and final execution states such as success or failure.

5. Add automated tests covering:

   • target resolution for every supported scope
   • permission enforcement, especially the superuser-only all devices scope
   • MassCommand creation and child Command creation
   • status aggregation

Details to preserve during implementation

Status aggregation should be defined more clearly before or during implementation. The implementation can investigate and document how to map existing Command statuses (in-progress, success, failed) to MassCommand states, including total success, total failure, partial success, in-progress, idle, and no-target cases. The behavior can be aligned with BatchUpgradeOperation in openwisp-firmware-upgrader if that remains the best fit.

Target resolution and validation must respect multi-tenancy. When an organization is selected, related locations, device groups, and manual device selections should be constrained to compatible objects. The all devices target should remain restricted to superusers. A MassCommand may need to support a nullable organization for superuser-created cross-organization operations, while non-superusers should be constrained to an organization they are allowed to manage.

Execution scale should be considered explicitly. The proposal pseudocode mentions batch_size=100, but the final implementation can investigate bulk creation, chunking, throttling, and avoiding unbounded task fan-out for very large target sets.

The WebSocket requirement should remain visible, but it can be implemented in this issue, moved to the admin UI issue, or split into a dedicated issue after the admin UI data requirements are clearer.

Additional tests should cover multi-tenant target combinations, superuser-only all devices behavior, organization requirements for non-superusers, status aggregation edge cases, and large target resolution behavior.

[pandafy]

Aggregate execution progress at the MassCommand level based on the status of child commands.

We need to define this more clearly as it took us some time for firmware-upgrader project last year. We can simply mirror what has been done for the BatchUpgradeOperation in openwisp-firmware-upgrader. We need to think about cases with partial success, total success, total failure, etc. Right now, we don't have an Idle status for the Command object. We only have in-progress, success and failed.

[pandafy]

4. Add a new WebSocket endpoint to listen to real-time status updates for mass command execution, allowing the frontend to receive and display command progress and final execution states such as success or failure.

I think it will be better to move this deliverable to #1345, because we won't be able to test this without a working UI. All efforts may become futile if the frontend required data in some other format than previously decided.

We can also move this to a dedicated issue which can be done after we have Mass Command admin views ready.

[pandafy]

• all devices
• organization
• device group
• location
• manual device selection

We need to make sure the field values respects multi-tenancy, i.e. when organization A is selected, then it should not allow to select device group from organization B. Same goes for location and manual device selection. We need explicit test cases for possible combinations.

[pandafy]

The Mass Command can be shared across different organization (when it is created by superuser).

So, the organization field will be nullable, but it will be required for non-super users.

[nemesifier]

Updated the issue body to preserve the main open points from the proposal review.

Summary of changes:

• Added a section for MassCommand status aggregation, including partial success, total success, total failure, in-progress, idle, and no-target cases.
• Added notes to investigate how existing Command statuses map to MassCommand statuses and whether to align with BatchUpgradeOperation.
• Promoted multi-tenancy and permission details from comments, including organization-scoped filtering, superuser-only all devices, and nullable organization for superuser-created cross-organization operations.
• Added execution-scale considerations such as batching, chunking, throttling, and avoiding unbounded task fan-out.
• Kept the WebSocket deliverable open as an implementation placement decision rather than deciding whether it belongs here, in the admin UI issue, or in a dedicated issue.
• Expanded the testing notes around multi-tenancy, status aggregation, and large target resolution.

To review the body edits, open this issue, navigate to the top bar of the issue, click last edited by nemesifier, then click most recent in the edit history dialog. Unfortunately there is not a quick URL to share for this.