[feature] Persistent Mass Upgrades#436
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds persistent firmware upgrade support. Batch and standalone operations gain persistence and retry metadata, while Estimated code review effort: 4 (Complex) | ~75 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error)
✅ Passed checks (7 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ace6c8c to
5119b30
Compare
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (1 file)
Previous Review Summaries (10 snapshots, latest commit 74ff536)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 74ff536)Status: No Issues Found | Recommendation: Merge Incremental re-review at HEAD Files Reviewed (incremental, 3 files)
Previous review (commit b49103e)Status: No Issues Found | Recommendation: Merge Incremental re-review at HEAD
No new critical bugs or security vulnerabilities were introduced. The change resolves a previously latent stuck-state bug and is backed by a regression test. Files Reviewed (incremental, 2 files)
Previous review (commit 49c90c2)Status: No Issues Found | Recommendation: Merge Incremental re-review at HEAD
No new critical bugs or security issues were introduced. The previously flagged Files Reviewed (incremental, 4 files)
Previous review (commit 87d1206)Status: 1 Issue Found | Recommendation: Merge with follow-up Incremental re-review at HEAD The only carried-forward finding is on the unchanged Overview
Issue Details (click to expand)WARNING
Files Reviewed (incremental)
Fix these issues in Kilo Cloud Previous review (commit a56e5aa)Status: 1 Issue Found | Recommendation: Merge with follow-up Incremental re-review at HEAD The notification refactor is correct: both call sites now pass the required One of the two previously flagged index locks is now resolved: Overview
Issue Details (click to expand)WARNING
Resolved Since Last Review
Files Reviewed (incremental)
Fix these issues in Kilo Cloud Previous review (commit 76ac8f0)Status: 2 Issues Found | Recommendation: Merge with follow-up Incremental re-review at HEAD The two previously flagged deploy-time migration index locks remain unresolved on unchanged migration files and are carried forward below (both already have active inline comments). The core persistent mass-upgrade pipeline is unchanged and remains sound: atomic claim on retry, Overview
Issue Details (click to expand)WARNING
Files Reviewed (incremental)
Fix these issues in Kilo Cloud Previous review (commit 99b5988)Status: 2 Issues Found | Recommendation: Merge with follow-up Re-reviewed the persistent mass-upgrade pipeline at the current HEAD. The core logic remains solid: No new critical bugs or security issues found in the changed code. The only outstanding items are two deploy-time migration index locks, both already flagged inline. Overview
Issue Details (click to expand)WARNING
Files Reviewed
Fix these issues in Kilo Cloud Previous review (commit 8c60dc3)Status: 2 Issues Found | Recommendation: Merge with follow-up The persistent mass-upgrade pipeline is solid: atomic status claims ( The only outstanding items are two deploy-time migration index locks, both already noted inline. Consider a maintenance window or a Overview
Issue Details (click to expand)WARNING
Files Reviewed
Fix these issues in Kilo Cloud Previous review (commit b6f5543)Status: 2 Issues Found | Recommendation: Merge with follow-up The persistent mass-upgrade pipeline is solid: atomic status claims, conditional re-checks before dispatch, an immutable Overview
Issue Details (click to expand)WARNING
Files Reviewed
Fix these issues in Kilo Cloud Previous review (commit a6f27f0)Status: 3 Issues Found | Recommendation: Merge with follow-up The persistent mass-upgrade pipeline is solid: atomic status claims, conditional Overview
Issue Details (click to expand)WARNING
Other Observations (not in diff)The The Beat scanner ( Files Reviewed (18 files)
[Snapshot truncated.] Additional previous summary content was truncated to keep this comment within platform limits. Reviewed by step-3.7-flash · Input: 183.3K · Output: 10K · Cached: 1.4M |
There was a problem hiding this comment.
Pull request overview
This PR lays the model-layer groundwork for “persistent” mass upgrades by extending the upgrade/batch operation schema, adding a new pending operation status, propagating batch persistence to child operations, and enforcing immutability of the persistence flag after launch.
Changes:
- Added
persistentflags to batch and per-device upgrade operations, plusretry_countandnext_retry_atfields for retry scheduling. - Introduced
pendingtoUpgradeOperation.STATUS_CHOICESand propagatedbatch.persistentinto child operations during creation. - Added unit tests for defaults, propagation, DB index presence, and immutability guards; added corresponding migrations (including sample app migration).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
openwisp_firmware_upgrader/base/models.py |
Adds persistence fields, pending status, batch→child propagation, and immutability validation in clean(). |
openwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_persistent_and_more.py |
Introduces DB fields and extends status choices at the schema level. |
openwisp_firmware_upgrader/tests/test_models.py |
Adds tests for new field defaults, propagation behavior, DB index flag, and immutability validation. |
tests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_persistent_and_more.py |
Mirrors the migration in the sample app used by the test project. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ("failed", _("failed")), # failed at late stage or can't reconnect | ||
| ("cancelled", _("cancelled")), # cancelled by the user | ||
| ("aborted", _("aborted")), # aborted due to prerequisites not met | ||
| ("pending", _("pending")), # offline device; waiting for periodic retry | ||
| ) |
| ("failed", _("failed")), # failed at late stage or can't reconnect | ||
| ("cancelled", _("cancelled")), # cancelled by the user | ||
| ("aborted", _("aborted")), # aborted due to prerequisites not met | ||
| ("pending", _("pending")), # offline device; waiting for periodic retry |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@openwisp_firmware_upgrader/base/models.py`:
- Around line 617-631: When checking immutability for the persistent field in
the model (inside the BatchUpgradeOperation instance logic), use the persisted
DB status instead of the in-memory self.status so callers can't flip status to
"idle" during the same edit; fetch the stored status via
load_model("BatchUpgradeOperation").objects.values_list("status",
flat=True).get(pk=self.pk) (similar to the existing persistent fetch) and change
the gate from self.status != "idle" to stored_status != "idle" before comparing
self.persistent to the persisted persistent and raising the ValidationError.
In `@openwisp_firmware_upgrader/tests/test_models.py`:
- Around line 606-608: Replace hard-coded app-label string patching with
model-bound patching: import or reference the UpgradeOperation model class and
use mock.patch.object(UpgradeOperation, "upgrade", return_value=None) instead of
mock.patch(f"{self.app_label}.models.UpgradeOperation.upgrade", ...); do the
same for the other occurrence that patches UpgradeOperation.upgrade (the second
instance mentioned) so tests remain resilient to model swapping.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 4eaa450b-cfb3-458c-8956-a3dc2e9dbe2a
📒 Files selected for processing (4)
openwisp_firmware_upgrader/base/models.pyopenwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_persistent_and_more.pyopenwisp_firmware_upgrader/tests/test_models.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_persistent_and_more.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: copilot-pull-request-reviewer
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{py,html}
📄 CodeRabbit inference engine (Custom checks)
For Django pull requests, ensure all user-facing strings are marked as translatable using the Django i18n framework
Files:
openwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_persistent_and_more.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_persistent_and_more.pyopenwisp_firmware_upgrader/tests/test_models.pyopenwisp_firmware_upgrader/base/models.py
🧠 Learnings (3)
📚 Learning: 2026-02-21T20:21:02.014Z
Learnt from: nemesifier
Repo: openwisp/openwisp-firmware-upgrader PR: 377
File: openwisp_firmware_upgrader/migrations/0015_add_group_to_batchupgradeoperation.py:16-26
Timestamp: 2026-02-21T20:21:02.014Z
Learning: In Django migrations within this repo, avoid adding explicit related_name to ForeignKey fields when the default reverse accessor (<model>_set) is acceptable. Rely on Django’s default naming convention to keep migrations concise and maintainable. Only specify related_name if you have a concrete, needed override for reverse relations.
Applied to files:
openwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_persistent_and_more.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_persistent_and_more.py
📚 Learning: 2026-02-23T21:36:22.028Z
Learnt from: nemesifier
Repo: openwisp/openwisp-firmware-upgrader PR: 377
File: tests/openwisp2/sample_firmware_upgrader/migrations/0001_initial.py:61-67
Timestamp: 2026-02-23T21:36:22.028Z
Learning: In the sample_firmware_upgrader app within tests, avoid creating new migration files for small changes since the sample is for demonstration and testing. Review migrations with this context: the sample's initial migration may differ from the main app and need not be strictly aligned. If you change models for the demo, keep migrations minimal and document why divergence exists; don’t require parity with the main app migrations in PRs focused on the sample.
Applied to files:
tests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_persistent_and_more.py
📚 Learning: 2026-02-27T19:08:56.218Z
Learnt from: nemesifier
Repo: openwisp/openwisp-firmware-upgrader PR: 383
File: openwisp_firmware_upgrader/tests/test_admin.py:50-50
Timestamp: 2026-02-27T19:08:56.218Z
Learning: In tests for openwisp-firmware-upgrader, derive app_label values from model meta instead of hard-coding. Specifically, let BaseTestAdmin compute app_label from Build._meta.app_label and config_app_label from Device._meta.app_label to ensure tests remain correct if models are swapped. This pattern should apply to all test files under openwisp_firmware_upgrader/tests.
Applied to files:
openwisp_firmware_upgrader/tests/test_models.py
🔇 Additional comments (4)
openwisp_firmware_upgrader/tests/test_models.py (1)
586-602: LGTM!Also applies to: 609-630, 634-673
openwisp_firmware_upgrader/base/models.py (1)
461-461: LGTM!Also applies to: 571-579, 846-846, 870-894, 896-913
openwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_persistent_and_more.py (1)
13-67: LGTM!tests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_persistent_and_more.py (1)
13-67: LGTM!
6595d4f to
4dba3f9
Compare
nemesifier
left a comment
nemesifier
left a comment
There was a problem hiding this comment.
It's looking good, please batch in more work or it will be hard for me to get an idea of whether we're really going in the right direction.
| ("failed", _("failed")), # failed at late stage or can't reconnect | ||
| ("cancelled", _("cancelled")), # cancelled by the user | ||
| ("aborted", _("aborted")), # aborted due to prerequisites not met | ||
| ("pending", _("pending")), # offline device; waiting for periodic retry |
There was a problem hiding this comment.
make sure this is cancellable and add a test for this: pending upgrades can be cancled.
There was a problem hiding this comment.
I would include the minimum behavior that makes the new state coherent:
BatchUpgradeOperation.progress_report:pendingmust not be counted as completed.BatchUpgradeOperation.calculate_and_update_status(): a batch with pending child operations should remain active, not becomesuccess.UpgradeOperation.cancel(): pending operations should be cancellable.UpgradeOperation.upgrade(): an existingpendingoperation should block a second upgrade for the same device, the same wayin-progressdoes.
I would also update the BatchUpgradeOperation.clean() immutability check to use the stored database status, not self.status, otherwise a caller can set status="idle" and change is_persistent in the same edit.
Please rename persistent to is_persistent consistently on both BatchUpgradeOperation and UpgradeOperation.
For tests, I would add a batch with a mix of success and pending child operations. The expected behavior should be: progress does not count pending as completed, batch status remains active, pending operation can be cancelled, and another upgrade for the same device is blocked while the pending operation exists.
45fad3d to
6d84175
Compare
Multiple Test Failures DetectedHello @Eeshu-Yadav, There are multiple test failures in your commit:
Fix: Please review the failing tests and their corresponding code to identify the root cause of the failures. Common issues include incorrect test setup, assertion errors, or unexpected behavior in the code being tested.
Fix: Escape backslashes in string literals and regular expressions by using double backslashes ( |
b32e764 to
f910f6a
Compare
|
The CI is failing due to transient infrastructure issues (not related to your code). I have restarted the failed jobs automatically (1/3). |
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
f910f6a to
741148f
Compare
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| name="next_retry_at", | ||
| field=models.DateTimeField( | ||
| blank=True, | ||
| db_index=True, |
There was a problem hiding this comment.
WARNING: This migration adds a blocking index on upgradeoperation.next_retry_at
AddField(... db_index=True) will create the index with regular DDL during the deploy. On populated installations that can block writes to upgradeoperation, so this index should be moved into a separate non-atomic concurrent migration, for example with AddIndexConcurrently.
741148f to
509a56a
Compare
|
The CI is failing due to transient infrastructure issues (not related to your code). I have restarted the failed jobs automatically (1/3). |
509a56a to
e25d8d8
Compare
Eeshu-Yadav
moved this to In progress
in [GSoC26] Persistent & Scheduled Firmware Upgrades
nemesifier
left a comment
nemesifier
left a comment
There was a problem hiding this comment.
It's going in the right direction, next steps:
- admin
- docs: start adding a mention of this feature in the intro page, let's add a dedicated page for persistent upgrades
Keep pushing to this branch, I need to see the whole thing before I am confident in merging, if I merge now, reviewing the diff for other fragments will hide details that make my life as a reviewer harder. Go ahead, keep iterating, pushing, advancing.
Update the PR title and description when due, add screenshots when there's UI changes.
| batch_status, | ||
| stats["completed"], | ||
| stats["total_operations"], | ||
| stats["pending"], |
There was a problem hiding this comment.
check if the docs need to be updated due to this
There was a problem hiding this comment.
yes docs updated related to this
| completedInfo.text(`${data.completed} complete, ${pending} pending`); | ||
| } else { | ||
| completedInfo.text(`${data.completed} out of ${data.total}`); | ||
| } |
There was a problem hiding this comment.
add assertions in the existing selenium tests for this logic please
| ) | ||
| PERSISTENT_RETRY_DISPATCH_JITTER = getattr( | ||
| settings, "OPENWISP_FIRMWARE_UPGRADER_PERSISTENT_RETRY_DISPATCH_JITTER", 300 | ||
| ) |
There was a problem hiding this comment.
These need to be documented. Are you sure we can't add one setting with a dictionary for all of them to avoid bloating the docs?
| CELERY_BROKER_URL = "memory://" | ||
| CELERY_RESULT_BACKEND = "cache+memory://" | ||
|
|
||
| CELERY_BEAT_SCHEDULE = { |
There was a problem hiding this comment.
do we have issues in ansible-openwisp2 and docker-openwisp to add these?
There was a problem hiding this comment.
i have opened the issues in the ansible and docker repos :
openwisp/docker-openwisp#623
openwisp/ansible-openwisp2#621
nemesifier
left a comment
nemesifier
left a comment
There was a problem hiding this comment.
It's going very well, let's continue iterating as we find details to refine.
The log lines are a bit inconsistent:
We need each log line related to persistent retries to stand on its own, e g:
The upgrade operation will be retried soon.
Failed to connect with AX820-Fed-Staging using SSH Key (SSH). Error: timed out
All immediate retries exhausted: Failed to establish connection with the device, tried all DeviceConnections.
Scheduled persistent retry #1 at 2026-07-09 16:43:32.102524+00:00.
Persistent retry #1 starting.
Failed to connect with AX820-Fed-Staging using SSH Key (SSH). Error: timed out
Detected a recoverable failure: Failed to establish connection with the device, tried all DeviceConnections.
The upgrade operation will be retried soon.
Failed to connect with AX820-Fed-Staging using SSH Key (SSH). Error: timed out
Detected a recoverable failure: Failed to establish connection with the device, tried all DeviceConnections.
The upgrade operation will be retried soon.
Failed to connect with AX820-Fed-Staging using SSH Key (SSH). Error: timed out
Detected a recoverable failure: Failed to establish connection with the device, tried all DeviceConnections.
The upgrade operation will be retried soon.
Failed to connect with AX820-Fed-Staging using SSH Key (SSH). Error: timed out
Detected a recoverable failure: Failed to establish connection with the device, tried all DeviceConnections.
The upgrade operation will be retried soon.
Failed to connect with AX820-Fed-Staging using SSH Key (SSH). Error: timed out
All immediate retries exhausted: Failed to establish connection with the device, tried all DeviceConnections.
Scheduled persistent retry #2 at 2026-07-09 17:17:38.648201+00:00.
The pending-reminder and persistent-failure notifications used two custom notification types (pending_upgrade_reminder, persistent_upgrade_failed). #426 asks for the built-in generic_message type instead, so both now send generic_message. The clean title (otherwise "Generic notification with generic verb and level info by ...") is kept by passing message alongside description, which get_message() prefers over the type's default template. The reminder also passes a url to the batch change page filtered with ?status=pending, so admins land directly on the devices still waiting. Related to #379
… migrations #379 next_retry_at (0018) keeps db_index=True: check_pending_upgrades scans status="pending", next_retry_at <= now every 10 minutes on the per-device UpgradeOperation table, and that btree is the query's only usable, highly selective index (next_retry_at is null for every non-pending op). The header documents the write-blocking build: the non-concurrent CREATE INDEX takes a SHARE lock (reads keep working), so on a large table run the migration in a maintenance window or pre-build with CREATE INDEX CONCURRENTLY. last_reminder_at (0019) drops the index instead: the reminder scan runs about weekly against the small BatchUpgradeOperation table and its OR/IS-NULL due filter cannot use a single-column btree anyway, so an index there would only add build cost. Both migrations are mirrored in the sample app. Related to #379
The retry-flow detail lived in both upgrade-status and the dedicated persistent-mass-upgrades page; upgrade-status now keeps a short pointer and links out. The intro page and the module landing page mention persistent upgrades and link to the dedicated page, and the settings page refers to the reminder as a generic_message notification. Related to #379
…tegration The persistent mass-upgrade reminder relies on openwisp-notifications honoring the per-notification "url" for the web notification link, so the "Open" button lands on the batch filtered to its pending devices. That support lands in openwisp/openwisp-notifications#490; pin the dependency to that commit so the integration is reproducible until it is released.
7d97360 to
a56e5aa
Compare
|
@coderabbitai full review |
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@openwisp_firmware_upgrader/base/models.py`:
- Line 1143: Update the display formatting in the surrounding string
representation to pass self.next_retry_at through timezone.localtime(...) before
calling strftime, matching the existing local-time formatting conventions while
preserving the current output format.
In
`@openwisp_firmware_upgrader/static/firmware-upgrader/css/upgrade-selected-confirmation.css`:
- Around line 22-42: Merge the two fieldset.aligned label.is-persistent-checkbox
declarations into one rule, preserving all existing properties and resolving the
final padding values as padding-top: 0 and padding-bottom: 0.
In `@openwisp_firmware_upgrader/tests/test_models.py`:
- Around line 835-842: The non-persistent unreachable-upgrade path in the
`error.connection is None` branch of `upgrade` must transition to a terminal
failure state instead of remaining in-progress. Update the status handling in
`upgrade` accordingly, ensure the associated batch is no longer left active, and
revise `test_no_connection_non_persistent_op_stays_in_progress` to assert the
new terminal behavior.
In `@openwisp_firmware_upgrader/tests/test_tasks.py`:
- Around line 64-73: Update the upgrade_firmware flow to re-check or atomically
claim the operation immediately before invoking UpgradeOperation.upgrade,
preventing a cancellation between the initial status check and flashing; add a
test covering cancellation during that window and assert upgrade is not called.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: afc9b399-69a2-406d-8cc4-d383cad6b185
📒 Files selected for processing (37)
CHANGES.rstdocs/index.rstdocs/user/intro.rstdocs/user/persistent-mass-upgrades.rstdocs/user/rest-api.rstdocs/user/settings.rstdocs/user/upgrade-status.rstdocs/user/websocket-api.rstopenwisp_firmware_upgrader/admin.pyopenwisp_firmware_upgrader/api/serializers.pyopenwisp_firmware_upgrader/api/views.pyopenwisp_firmware_upgrader/apps.pyopenwisp_firmware_upgrader/base/models.pyopenwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/migrations/0019_batchupgradeoperation_last_reminder_at.pyopenwisp_firmware_upgrader/settings.pyopenwisp_firmware_upgrader/static/firmware-upgrader/css/upgrade-progress.cssopenwisp_firmware_upgrader/static/firmware-upgrader/css/upgrade-selected-confirmation.cssopenwisp_firmware_upgrader/static/firmware-upgrader/js/batch-upgrade-progress.jsopenwisp_firmware_upgrader/static/firmware-upgrader/js/upgrade-progress.jsopenwisp_firmware_upgrader/static/firmware-upgrader/js/upgrade-utils.jsopenwisp_firmware_upgrader/tasks.pyopenwisp_firmware_upgrader/templates/admin/upgrade_selected_confirmation.htmlopenwisp_firmware_upgrader/tests/base.pyopenwisp_firmware_upgrader/tests/test_admin.pyopenwisp_firmware_upgrader/tests/test_api.pyopenwisp_firmware_upgrader/tests/test_handlers.pyopenwisp_firmware_upgrader/tests/test_models.pyopenwisp_firmware_upgrader/tests/test_notifications.pyopenwisp_firmware_upgrader/tests/test_selenium.pyopenwisp_firmware_upgrader/tests/test_tasks.pyopenwisp_firmware_upgrader/tests/test_websockets.pyopenwisp_firmware_upgrader/websockets.pyrequirements-test.txttests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.pytests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.pytests/openwisp2/settings.py
💤 Files with no reviewable changes (1)
- CHANGES.rst
📜 Review details
🧰 Additional context used
📓 Path-based instructions (3)
docs/**/*.rst
📄 CodeRabbit inference engine (AGENTS.md)
Update docs when behavior, settings, public APIs, setup steps, or supported versions change
Files:
docs/user/websocket-api.rstdocs/user/intro.rstdocs/index.rstdocs/user/settings.rstdocs/user/rest-api.rstdocs/user/upgrade-status.rstdocs/user/persistent-mass-upgrades.rst
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Place imports at the top of the file. Only defer imports when necessary (e.g., Django model imports inside functions or methods where the app registry is not yet ready)
Avoid unnecessary blank lines inside function and method bodies
Mark user-facing strings for translation with Django i18n helpers in Django code
Write comments and docstrings only when they explain why code is shaped a certain way. Put comments before the relevant code block instead of scattering them inside it
Files:
tests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.pyopenwisp_firmware_upgrader/migrations/0019_batchupgradeoperation_last_reminder_at.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/settings.pyopenwisp_firmware_upgrader/api/views.pyopenwisp_firmware_upgrader/tests/test_notifications.pyopenwisp_firmware_upgrader/apps.pyopenwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/websockets.pytests/openwisp2/settings.pyopenwisp_firmware_upgrader/tests/base.pyopenwisp_firmware_upgrader/tests/test_admin.pyopenwisp_firmware_upgrader/tasks.pyopenwisp_firmware_upgrader/tests/test_websockets.pyopenwisp_firmware_upgrader/tests/test_handlers.pyopenwisp_firmware_upgrader/api/serializers.pyopenwisp_firmware_upgrader/tests/test_tasks.pyopenwisp_firmware_upgrader/tests/test_models.pyopenwisp_firmware_upgrader/tests/test_selenium.pyopenwisp_firmware_upgrader/tests/test_api.pyopenwisp_firmware_upgrader/admin.pyopenwisp_firmware_upgrader/base/models.py
**/*.{py,html}
📄 CodeRabbit inference engine (Custom checks)
For Django pull requests, ensure all user-facing strings are marked as translatable using the Django i18n framework.
Files:
tests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.pyopenwisp_firmware_upgrader/migrations/0019_batchupgradeoperation_last_reminder_at.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/settings.pyopenwisp_firmware_upgrader/templates/admin/upgrade_selected_confirmation.htmlopenwisp_firmware_upgrader/api/views.pyopenwisp_firmware_upgrader/tests/test_notifications.pyopenwisp_firmware_upgrader/apps.pyopenwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/websockets.pytests/openwisp2/settings.pyopenwisp_firmware_upgrader/tests/base.pyopenwisp_firmware_upgrader/tests/test_admin.pyopenwisp_firmware_upgrader/tasks.pyopenwisp_firmware_upgrader/tests/test_websockets.pyopenwisp_firmware_upgrader/tests/test_handlers.pyopenwisp_firmware_upgrader/api/serializers.pyopenwisp_firmware_upgrader/tests/test_tasks.pyopenwisp_firmware_upgrader/tests/test_models.pyopenwisp_firmware_upgrader/tests/test_selenium.pyopenwisp_firmware_upgrader/tests/test_api.pyopenwisp_firmware_upgrader/admin.pyopenwisp_firmware_upgrader/base/models.py
🧠 Learnings (8)
📚 Learning: 2026-02-21T20:21:02.014Z
Learnt from: nemesifier
Repo: openwisp/openwisp-firmware-upgrader PR: 377
File: openwisp_firmware_upgrader/migrations/0015_add_group_to_batchupgradeoperation.py:16-26
Timestamp: 2026-02-21T20:21:02.014Z
Learning: In Django migrations within this repo, avoid adding explicit related_name to ForeignKey fields when the default reverse accessor (<model>_set) is acceptable. Rely on Django’s default naming convention to keep migrations concise and maintainable. Only specify related_name if you have a concrete, needed override for reverse relations.
Applied to files:
tests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.pyopenwisp_firmware_upgrader/migrations/0019_batchupgradeoperation_last_reminder_at.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_is_persistent_and_more.py
📚 Learning: 2026-02-23T21:36:22.028Z
Learnt from: nemesifier
Repo: openwisp/openwisp-firmware-upgrader PR: 377
File: tests/openwisp2/sample_firmware_upgrader/migrations/0001_initial.py:61-67
Timestamp: 2026-02-23T21:36:22.028Z
Learning: In the sample_firmware_upgrader app within tests, avoid creating new migration files for small changes since the sample is for demonstration and testing. Review migrations with this context: the sample's initial migration may differ from the main app and need not be strictly aligned. If you change models for the demo, keep migrations minimal and document why divergence exists; don’t require parity with the main app migrations in PRs focused on the sample.
Applied to files:
tests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.py
📚 Learning: 2026-06-02T08:16:00.439Z
Learnt from: asmodehn
Repo: openwisp/openwisp-firmware-upgrader PR: 362
File: tests/openwisp2/urls.py:10-38
Timestamp: 2026-06-02T08:16:00.439Z
Learning: In this repository, treat `SAMPLE_APP` as a conventionally truthy environment variable: code should use the established pattern `os.environ.get("SAMPLE_APP", False)` and rely on the resulting value’s truthiness (truthy string → enabled). During code reviews, avoid recommending changes to explicit boolean parsing (e.g., checking for `
Applied to files:
tests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.pyopenwisp_firmware_upgrader/migrations/0019_batchupgradeoperation_last_reminder_at.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/settings.pyopenwisp_firmware_upgrader/api/views.pyopenwisp_firmware_upgrader/tests/test_notifications.pyopenwisp_firmware_upgrader/apps.pyopenwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/websockets.pytests/openwisp2/settings.pyopenwisp_firmware_upgrader/tests/base.pyopenwisp_firmware_upgrader/tests/test_admin.pyopenwisp_firmware_upgrader/tasks.pyopenwisp_firmware_upgrader/tests/test_websockets.pyopenwisp_firmware_upgrader/tests/test_handlers.pyopenwisp_firmware_upgrader/api/serializers.pyopenwisp_firmware_upgrader/tests/test_tasks.pyopenwisp_firmware_upgrader/tests/test_models.pyopenwisp_firmware_upgrader/tests/test_selenium.pyopenwisp_firmware_upgrader/tests/test_api.pyopenwisp_firmware_upgrader/admin.pyopenwisp_firmware_upgrader/base/models.py
📚 Learning: 2026-06-15T20:19:28.326Z
Learnt from: atif09
Repo: openwisp/openwisp-firmware-upgrader PR: 437
File: openwisp_firmware_upgrader/tests/test_admin.py:1072-1147
Timestamp: 2026-06-15T20:19:28.326Z
Learning: In this codebase (OpenWISP projects, including openwisp-firmware-upgrader), prefer test methods that are fully self-contained: each test should do its own setup and make its own explicit assertions within the test body. Avoid introducing shared assertion helpers or parameterized helper methods that abstract away assertions across tests, even if it leads to some duplication.
Applied to files:
tests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.pytests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.pyopenwisp_firmware_upgrader/tests/test_notifications.pytests/openwisp2/settings.pyopenwisp_firmware_upgrader/tests/base.pyopenwisp_firmware_upgrader/tests/test_admin.pyopenwisp_firmware_upgrader/tests/test_websockets.pyopenwisp_firmware_upgrader/tests/test_handlers.pyopenwisp_firmware_upgrader/tests/test_tasks.pyopenwisp_firmware_upgrader/tests/test_models.pyopenwisp_firmware_upgrader/tests/test_selenium.pyopenwisp_firmware_upgrader/tests/test_api.py
📚 Learning: 2026-02-23T20:09:13.275Z
Learnt from: pandafy
Repo: openwisp/openwisp-firmware-upgrader PR: 377
File: requirements-test.txt:1-1
Timestamp: 2026-02-23T20:09:13.275Z
Learning: When reviewing OpenWISP projects (e.g., openwisp-firmware-upgrader), do not flag floating branch references like refs/heads/1.3 in requirements files as a violation. This project workflow intentionally uses development-version refs for openwisp-utils dependencies and should not require pinning to a specific commit SHA or tag. If you encounter such refs, treat them as acceptable in requirements-*.txt files and verify other aspects (e.g., compatibility) through tests rather than pinning.
Applied to files:
requirements-test.txt
📚 Learning: 2026-02-24T00:04:04.187Z
Learnt from: nemesifier
Repo: openwisp/openwisp-firmware-upgrader PR: 377
File: openwisp_firmware_upgrader/static/firmware-upgrader/css/upgrade-options.css:11-13
Timestamp: 2026-02-24T00:04:04.187Z
Learning: In openwisp-firmware-upgrader, prefer hiding empty submit rows via CSS (e.g., `#upgradeoperation_form` .submit-row { display: none; }) rather than removing elements in templates. This keeps template logic simple and can improve rendering performance. Apply this pattern to CSS files in openwisp_firmware_upgrader/static/firmware-upgrader/css/*.css for consistent behavior across upgrade-option related styles.
Applied to files:
openwisp_firmware_upgrader/static/firmware-upgrader/css/upgrade-progress.cssopenwisp_firmware_upgrader/static/firmware-upgrader/css/upgrade-selected-confirmation.css
📚 Learning: 2026-02-27T19:08:56.218Z
Learnt from: nemesifier
Repo: openwisp/openwisp-firmware-upgrader PR: 383
File: openwisp_firmware_upgrader/tests/test_admin.py:50-50
Timestamp: 2026-02-27T19:08:56.218Z
Learning: In tests for openwisp-firmware-upgrader, derive app_label values from model meta instead of hard-coding. Specifically, let BaseTestAdmin compute app_label from Build._meta.app_label and config_app_label from Device._meta.app_label to ensure tests remain correct if models are swapped. This pattern should apply to all test files under openwisp_firmware_upgrader/tests.
Applied to files:
openwisp_firmware_upgrader/tests/test_notifications.pyopenwisp_firmware_upgrader/tests/base.pyopenwisp_firmware_upgrader/tests/test_admin.pyopenwisp_firmware_upgrader/tests/test_websockets.pyopenwisp_firmware_upgrader/tests/test_handlers.pyopenwisp_firmware_upgrader/tests/test_tasks.pyopenwisp_firmware_upgrader/tests/test_models.pyopenwisp_firmware_upgrader/tests/test_selenium.pyopenwisp_firmware_upgrader/tests/test_api.py
📚 Learning: 2026-03-07T01:16:15.471Z
Learnt from: atif09
Repo: openwisp/openwisp-firmware-upgrader PR: 387
File: openwisp_firmware_upgrader/admin.py:420-423
Timestamp: 2026-03-07T01:16:15.471Z
Learning: In openwisp-firmware-upgrader, UpgradeOperation and BatchUpgradeOperation always share the same organization (BatchUpgradeOperation.build.category.organization). A user who can view an UpgradeOperation can always view its related BatchUpgradeOperation. For the UpgradeOperationAdmin.change_view, model-level has_view_permission(request) suffices to show breadcrumbs for batch, and object-level permission checks are not required.
Applied to files:
openwisp_firmware_upgrader/admin.py
🪛 ast-grep (0.44.1)
tests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.py
[info] 69-80: use help_text to document model columns
Context: models.CharField(
choices=[
("in-progress", "in progress"),
("success", "success"),
("failed", "failed"),
("cancelled", "cancelled"),
("aborted", "aborted"),
("pending", "pending"),
],
default="in-progress",
max_length=12,
)
Note: [CWE-710] Improper Adherence to Coding Standards.
(model-help-text)
openwisp_firmware_upgrader/tests/test_notifications.py
[warning] 12-12: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("UpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
[warning] 13-13: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("BatchUpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
openwisp_firmware_upgrader/apps.py
[warning] 130-130: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("device_monitoring", "DeviceMonitoring", required=False)
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
[warning] 134-134: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("firmware_upgrader", "UpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
openwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_is_persistent_and_more.py
[info] 69-80: use help_text to document model columns
Context: models.CharField(
choices=[
("in-progress", "in progress"),
("success", "success"),
("failed", "failed"),
("cancelled", "cancelled"),
("aborted", "aborted"),
("pending", "pending"),
],
default="in-progress",
max_length=12,
)
Note: [CWE-710] Improper Adherence to Coding Standards.
(model-help-text)
openwisp_firmware_upgrader/tasks.py
[info] 175-175: use secrets package over random package
Context: random.uniform(0, jitter)
Note: [CWE-330] Use of Insufficiently Random Values.
(avoid-random-python)
[warning] 118-118: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("UpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
[warning] 160-160: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("UpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
[warning] 181-181: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("BatchUpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
openwisp_firmware_upgrader/tests/test_handlers.py
[warning] 14-14: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("UpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
openwisp_firmware_upgrader/base/models.py
[warning] 481-481: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("UpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
[warning] 666-666: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("BatchUpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
[warning] 987-987: Loading a Keras model from an untrusted file can execute arbitrary code via Lambda layers or custom objects. Load only trusted models and avoid deserializing custom objects from untrusted sources.
Context: load_model("UpgradeOperation")
Note: [CWE-502] Deserialization of Untrusted Data.
(keras-load-model-python)
[info] 1078-1078: use secrets package over random package
Context: random.uniform(1 - jitter, 1 + jitter)
Note: [CWE-330] Use of Insufficiently Random Values.
(avoid-random-python)
[info] 1098-1098: use secrets package over random package
Context: random.uniform(0, jitter)
Note: [CWE-330] Use of Insufficiently Random Values.
(avoid-random-python)
🔇 Additional comments (53)
docs/index.rst (1)
13-15: LGTM!Also applies to: 44-44
docs/user/intro.rst (1)
14-15: LGTM!docs/user/persistent-mass-upgrades.rst (1)
1-120: LGTM!docs/user/rest-api.rst (1)
222-223: LGTM!Also applies to: 323-324, 365-365, 381-384
docs/user/settings.rst (1)
39-46: LGTM!Also applies to: 57-121
docs/user/upgrade-status.rst (1)
36-62: LGTM!Also applies to: 139-147, 188-196, 224-233
docs/user/websocket-api.rst (1)
139-139: LGTM!Also applies to: 184-184
tests/openwisp2/settings.py (1)
3-3: LGTM!Also applies to: 191-201
openwisp_firmware_upgrader/migrations/0018_batchupgradeoperation_is_persistent_and_more.py (4)
3-8: Blocking-index concern already mitigated with documented maintenance-window warning.The
next_retry_atfield still usesdb_index=True, but the file now carries an explicit warning (added in response to the earlier review) that the non-concurrentCREATE INDEXcan block writes on large tables and recommends a maintenance window or pre-building the index concurrently. This directly addresses the earlier blocking-index concern raised by kilo-code-bot and nemesifier.Also applies to: 47-57
20-37: LGTM!
38-66: LGTM!
67-82: LGTM!openwisp_firmware_upgrader/migrations/0019_batchupgradeoperation_last_reminder_at.py (1)
13-22: LGTM! Thedb_index=Trueflagged by kilo-code-bot in an earlier revision has been removed, resolving the blocking-index concern for this field.tests/openwisp2/sample_firmware_upgrader/migrations/0005_batchupgradeoperation_is_persistent_and_more.py (1)
1-84: LGTM! Mirrors the main-app migration0018_batchupgradeoperation_is_persistent_and_more.py, including the backfill pattern and the maintenance-window warning, as previously requested.tests/openwisp2/sample_firmware_upgrader/migrations/0006_batchupgradeoperation_last_reminder_at.py (1)
1-27: LGTM! Correctly mirrors the main-app migration withoutdb_index=True, consistent with the resolved discussion on0019.openwisp_firmware_upgrader/tests/test_api.py (5)
671-694: LGTM! Correctly verifiesis_persistentdefaults toTruewhen omitted and honors explicit overrides, matchingbatch_upgrade'sis_persistent=Truedefault.
1044-1063: LGTM!
1494-1515: LGTM! Consistent withAbstractDeviceFirmware.save'sis_persistent=Falsedefault and the serializer'screate()poppingis_persistent.
1969-1992: LGTM!
2186-2237: LGTM! Cancellation and immutability-validation behavior match the upstreamcancel()/_validate_is_persistent_immutablecontracts.openwisp_firmware_upgrader/tests/test_selenium.py (4)
278-311: LGTM! Verifies the persistent checkbox default state and propagation to child operations, matching PR objective#417/#427.
378-411: LGTM!
499-538: LGTM! Polling the DB viaWebDriverWaitfor the async status transition is a sound pattern to avoid flakiness.
1102-1175: LGTM! The pending-count text switching and orange-indicator assertions are consistent withBatchUpgradeProgressPublisher.publish_batch_status'spendingparameter and theUpgradeOperation.save()trigger for batch recalculation.requirements-test.txt (1)
2-2: LGTM! Pinningopenwisp-notificationsto a specific commit SHA (rather than a floating branch ref) is good practice for reproducible test builds, needed for the new failure/pending-reminder notification tests.openwisp_firmware_upgrader/tests/test_models.py (2)
819-829: Use the loadedDeviceConnectionmodel for both patches.These concrete paths can miss a swapped connection model. Patch
DeviceConnection.get_working_connectionwithmock.patch.objectinstead.Based on learnings, model patches in
openwisp_firmware_upgrader/testsshould remain swap-safe rather than use hard-coded model paths.Also applies to: 924-927
Source: Learnings
10-27: LGTM!Also applies to: 594-818, 844-923, 928-1019, 1098-1107
openwisp_firmware_upgrader/tests/base.py (1)
2-2: LGTM!Also applies to: 11-11, 27-32
openwisp_firmware_upgrader/tests/test_tasks.py (1)
1-14: LGTM!Also applies to: 84-282
openwisp_firmware_upgrader/tests/test_websockets.py (1)
403-408: LGTM!Also applies to: 705-787
openwisp_firmware_upgrader/tests/test_notifications.py (1)
1-207: LGTM!openwisp_firmware_upgrader/tests/test_handlers.py (1)
1-210: LGTM!openwisp_firmware_upgrader/tests/test_admin.py (1)
203-283: LGTM!Also applies to: 2131-2177
openwisp_firmware_upgrader/base/models.py (5)
1132-1148: 🎯 Functional Correctness | ⚡ Quick winUntranslated log strings remain in
_recoverable_failure_handler.Both the recoverable branch (
"Detected a recoverable failure...","The upgrade operation will be retried soon.") and the persistent branch ("All immediate retries exhausted...","Scheduled persistent retry #...") still use raw f-strings, unlike the sibling fix already applied to the "No device connection available" message at Line 1163/1170. Past review comments flagged this exact gap without a recorded fix.🌐 Proposed fix
def _recoverable_failure_handler(self, recoverable, error): cause = str(error) if recoverable: - self.log_line(f"Detected a recoverable failure: {cause}.\n", save=False) - self.log_line("The upgrade operation will be retried soon.") + self.log_line( + _("Detected a recoverable failure: %(cause)s.\n") % {"cause": cause}, + save=False, + ) + self.log_line(_("The upgrade operation will be retried soon.")) raise error if self.is_persistent and isinstance(error, RecoverableFailure): self.status = "pending" self.retry_count += 1 self.next_retry_at = self._calculate_next_retry() - self.log_line(f"All immediate retries exhausted: {cause}.\n", save=False) - retry_time = self.next_retry_at.strftime("%Y-%m-%d %H:%M:%S %Z") - self.log_line( - f"Scheduled persistent retry #{self.retry_count} at {retry_time}.\n", - save=False, - ) + self.log_line( + _("All immediate retries exhausted: %(cause)s.\n") % {"cause": cause}, + save=False, + ) + retry_time = timezone.localtime(self.next_retry_at).strftime( + "%Y-%m-%d %H:%M:%S %Z" + ) + self.log_line( + _("Scheduled persistent retry #%(count)s at %(time)s.\n") + % {"count": self.retry_count, "time": retry_time}, + save=False, + ) returnAs per coding guidelines,
**/*.py: Mark user-facing strings for translation with Django i18n helpers in Django code.Source: Coding guidelines
2-3: LGTM!Also applies to: 17-17, 44-44, 175-210, 452-497
601-618: LGTM!Also applies to: 656-681, 746-770, 840-925
949-1000: LGTM!Also applies to: 1045-1051, 1071-1081, 1102-1130, 1161-1170, 1202-1202
1082-1100: 🗄️ Data Integrity & IntegrationSame-device concurrency race review — deferring to consolidated comment.
handle_health_status_changeddispatches retries for any operation withstatus="pending"without also constrainingis_persistent=True. This shares the same root cause as thesend_pending_upgrade_remindersfinding intasks.py— see the consolidated comment at the end of this review.openwisp_firmware_upgrader/tasks.py (4)
180-222: 🗄️ Data Integrity & IntegrationReminder queryset can match non-persistent batches — deferring to consolidated comment.
The batch queryset filters only on
status="in-progress"plus a pending child, withoutis_persistent=True. See the consolidated comment at the end of this review for the shared root cause and fix across files.
31-55: LGTM!
115-157: LGTM!
159-178: LGTM!openwisp_firmware_upgrader/settings.py (1)
27-41: LGTM!openwisp_firmware_upgrader/apps.py (1)
32-32: LGTM!Also applies to: 93-97, 125-140
openwisp_firmware_upgrader/static/firmware-upgrader/js/batch-upgrade-progress.js (1)
203-216: LGTM!openwisp_firmware_upgrader/static/firmware-upgrader/css/upgrade-progress.css (1)
52-55: LGTM!openwisp_firmware_upgrader/admin.py (1)
128-139: LGTM!Also applies to: 227-253, 397-404, 413-441, 517-528, 538-555, 809-809
openwisp_firmware_upgrader/api/serializers.py (1)
88-91: LGTM!Also applies to: 106-116, 121-133, 160-181
openwisp_firmware_upgrader/api/views.py (1)
98-106: LGTM!Also applies to: 157-157
openwisp_firmware_upgrader/templates/admin/upgrade_selected_confirmation.html (1)
50-55: LGTM!openwisp_firmware_upgrader/static/firmware-upgrader/js/upgrade-utils.js (1)
10-10: LGTM!Also applies to: 20-20, 35-35, 63-69, 86-86, 99-103
openwisp_firmware_upgrader/static/firmware-upgrader/js/upgrade-progress.js (1)
252-255: LGTM!Also applies to: 294-295
openwisp_firmware_upgrader/websockets.py (1)
254-259: LGTM!Also applies to: 268-268, 336-336, 506-512, 525-525
Eeshu-Yadav
moved this from In progress
to In review
in [GSoC26] Persistent & Scheduled Firmware Upgrades
- Surface the persistent-retry state (is_persistent, retry_count, next_retry_at) above the log on the operation page so the state is visible without scrolling past the log. - Make each persistent-retry log line stand on its own with a clean, local-time timestamp. - Verify the monitoring signal wiring by emission and use openwisp_utils.tests.catch_signal for signal-emission tests. - Merge the duplicate is-persistent-checkbox CSS rule on the mass-upgrade confirmation page. - Tidy the monitoring docstring and drop the manual changelog entry (the releaser manages it). Related to #379
87d1206 to
ee7a610
Compare
|
Kilo Code Review could not run — your account is out of credits. Add credits or switch to a free model to enable reviews on this change. |
nemesifier
left a comment
nemesifier
left a comment
There was a problem hiding this comment.
I found one remaining race that should be fixed before merging.
… race #417 upgrade_firmware() checked the operation status in the database, then called upgrade() on the already-loaded instance. A cancellation issued after that check but before upgrade() ran left the stale instance marked in-progress, so upgrade() proceeded into the flashing path and overwrote the cancellation. Refresh the instance immediately before calling upgrade() so its existing in-memory guard observes the cancellation, and drop the now-redundant status query. Related to #417
1deab78 to
49c90c2
Compare
…417 When no working connection existed, a non-persistent operation only logged the problem and returned, leaving it in-progress forever and blocking any future upgrade of the same device. Move it to the aborted status instead, since a device without a usable connection cannot be upgraded and its prerequisites are not met. Related to #417
nemesifier
left a comment
nemesifier
left a comment
There was a problem hiding this comment.
I am looking into upgrade operations not flagged as persistent, eg:
From an old test did in the staging environment (you have access to this).
When persistent is False, there's no point in showing retry count and next retry as it only confuses users, it should be possible to do so in the admin.py code without involving JS/CSS.
nemesifier
left a comment
nemesifier
left a comment
There was a problem hiding this comment.
There are two retry-state correctness issues that should be fixed before merging.
| if operation.device.is_deactivated(): | ||
| aborted = UpgradeOperation.objects.filter( | ||
| pk=operation_id, status="in-progress" | ||
| ).update(status="failed") |
There was a problem hiding this comment.
This stores the operation as failed while the user-facing log says the persistent retry was aborted. That is inconsistent in the admin/API and also triggers failed-persistent-upgrade behavior for a device deactivation path. Please either store aborted here, which matches the log and existing deactivated-device semantics, or change the log and downstream expectations if failed is intentional.
There was a problem hiding this comment.
PS: in master this case is flagged as aborted, let's maintain consistency.
| def retry_pending_upgrade(operation_id): | ||
| UpgradeOperation = load_model("UpgradeOperation") | ||
| updated = UpgradeOperation.objects.filter(pk=operation_id, status="pending").update( | ||
| status="in-progress" |
There was a problem hiding this comment.
When this pending retry is claimed, next_retry_at remains populated even though the operation is no longer pending and no retry is queued for that timestamp. The admin and REST API then show a stale next retry date while the operation is in-progress. Please clear next_retry_at when moving the operation back to in-progress.
There was a problem hiding this comment.
PS: added a failing test for this
There was a problem hiding this comment.
PPS: can we avoid showing retry_next at all if it's None? I think this would be better.
Test Failure in
|
Checklist
Reference to Existing Issue
Closes #417
Closes #422
Closes #423
Closes #424
Closes #425
Closes #426
Closes #427
Closes #428
Closes #429
Closes #430
Related to #379 (parent - Persistent Mass Upgrades, Phase 1).
Description of Changes
Today a mass upgrade that hits an offline device goes from
in-progresstofailedonce Celery's auto-retries run out (about 10 minutes), and the admin has to retry each failed device by hand. That is manageable for one device but not for hundreds.This PR adds a persistent mode. An offline device now moves to a new
pendingstatus with a backoff-scheduled retry time instead of failing; a periodic Beat task picks up due operations and re-dispatches the upgrade, and the batch stays active until every device is upgraded or cancelled. The four sub-issues land together because the new status is only coherent once its schema (#417), the failure handler that produces it (#422), the batch/conflict/cancel code that has to recognise it (#423), and the Beat scanner that retries it (#424) are all present. Shipping any one alone either leaves the status unused or breaks the existing failure path.Changes in this PR:
pendingwith a scheduled next-retry time. A Beat task runs every 10 minutes (configurable) and re-dispatches due retries using exponential backoff (10m, 20m, 40m, ... capped at 12h, with ±25% jitter) until the device returns or the admin cancels.pendingas a blocking state, so a second upgrade targeting the same device will not double-flash it.failed.is_persistentis locked once the operation leavesidle, so the policy cannot be changed mid-run.is_persistent, letting single-device upgrades opt into persistence (off by default).openwisp-monitoringinstalled, a device coming back online retries its pending operations immediately instead of waiting for the next Beat scan.Screenshots