fix: honor --s3-root-uri and --conflict-resolution in attachment CLI - #1292
fix: honor --s3-root-uri and --conflict-resolution in attachment CLI#1292crowecawcaw wants to merge 7 commits into
Conversation
When 'deadline attachment download' runs without --profile and without an explicit --conflict-resolution (relying on config defaults), all CLI options passed to _apply_cli_options_to_config are None, so the else-branch cleanup runs. That branch did not pop 'conflict_resolution', leaving it in args and tripping the 'not standard AWS Deadline Cloud CLI options' RuntimeError guard. Add 'conflict_resolution' to the cleanup list so it is consumed. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
'deadline attachment download'/'upload' unconditionally overwrote s3_root_uri with the queue's job-attachment settings whenever --profile was not passed, discarding any explicit --s3-root-uri. Only fall back to the queue settings when --s3-root-uri was not provided. Also adds arg-plumbing regression tests covering both the --conflict-resolution config-defaults workflow and the --s3-root-uri handling. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
Move the MissingJobAttachmentSettingsError guard inside the s3_root_uri fallback so an explicitly-supplied URI is honored even when the queue has no jobAttachmentSettings. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
…e modes Add coverage for: - --conflict-resolution SKIP reaching the _attachment_download call - explicit --s3-root-uri on upload when the queue lacks jobAttachmentSettings - MissingJobAttachmentSettingsError still raised when neither an explicit --s3-root-uri nor queue attachment settings are available Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
| # Only fall back to the queue's S3 settings when the caller did not provide an | ||
| # explicit --s3-root-uri. An explicitly-supplied value must always be honored, | ||
| # even when the queue has no attachment settings of its own. | ||
| if not s3_root_uri: |
There was a problem hiding this comment.
The new comment says an explicit --s3-root-uri must be honored "even when the queue has no attachment settings of its own", but get_queue(...) (lines 126-130) is still called unconditionally before this check. So when a caller supplies --s3-root-uri precisely to operate without relying on queue settings, they still incur a GetQueue call — and if that call fails (e.g. AccessDenied, or the queue does not exist), the command errors out before ever reaching this branch, defeating the intent. Consider moving the get_queue(...) lookup inside the if not s3_root_uri: block so it is only performed when the fallback is actually needed.
| # Only fall back to the queue's S3 settings when the caller did not provide an | ||
| # explicit --s3-root-uri. An explicitly-supplied value must always be honored, | ||
| # even when the queue has no attachment settings of its own. | ||
| if not s3_root_uri: |
There was a problem hiding this comment.
Same as in attachment_download: get_queue(...) (lines 244-248) is still called unconditionally, so an explicit --s3-root-uri does not actually let a caller bypass the queue lookup. If GetQueue fails (AccessDenied, missing queue), upload errors out before reaching this fallback branch, contradicting the "must always be honored, even when the queue has no attachment settings" comment. Move the get_queue(...) call inside if not s3_root_uri:.
Add two end-to-end tests that run the attachment download/upload CLI with no job-attachments internals mocked: the queue-role credential path (AssumeQueueRoleForUser via the deadline mock), real manifest decoding, and the real S3 transfer code all execute against moto S3. Both the queue's configured bucket and the explicitly-named bucket exist; the tests prove the transfer touches the explicit bucket (download reads its content instead of a same-key decoy in the queue bucket, upload writes CAS data there and leaves the queue bucket empty). Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
Fixes:
What was the problem/requirement? (What/Why)
Two arg-plumbing bugs in the
attachmentCLI group:attachment downloadraised aRuntimeErrorin the config-defaults workflow:--conflict-resolutionwas not stripped from the args in the else-branch cleanup of_apply_cli_options_to_config, so a leftover{"conflict_resolution": None}reached theif args: raise RuntimeError(...)guard.--s3-root-uriwas ignored/overwritten unless--profilewas also passed, because the no-profile path unconditionally recomputed the S3 root URI.What was the solution? (How)
conflict_resolutionto the standard-option cleanup list in_apply_cli_options_to_config.if not s3_root_uri:in bothattachment downloadandattachment upload(same shared option, same fix) so an explicit--s3-root-uriis honored regardless of--profile.What is the impact of this change?
attachment downloadworks in the config-defaults workflow, and--s3-root-uriis respected on its own.How was this change tested?
Added red-green tests: the config-defaults download no longer raises, and
--s3-root-uriis passed through without--profilefor both download and upload.test_cli_attachment_args.py(3 tests) plus regression runs oftest_cli_common.py/test_cli_job.py.Was this change documented?
Does this PR introduce new dependencies?
Is this a breaking change?
No.
Does this change impact security?
No. (The separate manifest-upload disclosure / CAS-poisoning concern (C4) is out of scope and tracked separately.)
Testing
Automated coverage (
test/unit/deadline_client/cli/test_cli_attachment_args.py, 9 tests, all invoking the real CLI viaCliRunner):--conflict-resolutionomitted in the config-defaults workflow no longer trips the_apply_cli_options_to_configRuntimeError (download).--conflict-resolution SKIPis threaded through to the_attachment_downloadcall (not just consumed).--s3-root-uriis honored without--profilefor bothdownloadandupload, including when it would otherwise be overwritten by the queue's settings.--s3-root-uriworks even when the queue has nojobAttachmentSettings(bothdownloadandupload).--s3-root-uriand no queue attachment settings, the command still fails withMissingJobAttachmentSettingsErrorand never reaches the download.Manual verification (scratch script driving
deadline attachment download/uploadend-to-end through click with mocked deadline/S3 clients) confirmed all of the above scenarios pass, including that the explicit URI (s3://manual-check-bucket/...) wins over the queue-provided bucket.End-to-end S3 verification (moto): two additional tests run
attachment download/uploadwith no job-attachments internals mocked — the queue-role credential path (AssumeQueueRoleForUser), real manifest decoding, and the real S3 transfer code all execute against moto S3 with both the queue's configured bucket and the explicitly-named bucket created. They prove the transfer actually touches the explicit bucket: download reads its content instead of a same-CAS-key decoy planted in the queue's bucket, and upload writes the CAS object + manifest to the explicit bucket while the queue's bucket stays empty.The only thing still not covered by automation is real-AWS IAM: whether a given queue role's policy actually grants access to a caller-chosen bucket. That's an account-configuration concern rather than client-code behavior, so no hands-on testing of this change is required.