diff --git a/docs/design/job-bundle-browser.md b/docs/design/job-bundle-browser.md new file mode 100644 index 000000000..9a14d91dc --- /dev/null +++ b/docs/design/job-bundle-browser.md @@ -0,0 +1,635 @@ +# Job Bundle Browser + +## Problem + +When using `deadline bundle gui-submit --browse` or the "Load a different job bundle" button, users are presented with a native OS folder picker. This is inadequate because: + +- Job bundles are not distinguishable from regular folders by name alone. +- Users must already know where their bundles live and navigate there manually. +- There is no preview of what a bundle contains — users pick blindly. +- The picker always starts at the job history directory or home, with no way to configure a default location. + +## Overview + +Replace the native folder picker with a custom job bundle browser dialog that: + +1. Provides a navigable directory tree showing only folders, archives, and job bundles. +2. Displays a preview panel with bundle metadata when a job bundle is selected. +3. Supports both local filesystem and S3 bucket browsing through a common backend abstraction. +4. Supports job bundles as directories or `.ojd` archives (zip format under the hood). +5. For S3, uses the selected queue's job attachment bucket with a `job-bundles/` prefix — no extra configuration needed. +6. Caches S3 archive bundles locally with ETag validation for fast repeated access. +7. Respects a configurable default local browse directory. + +## Design + +### Bundle Formats + +Job bundles can be either: + +- **Directories** — a folder containing `template.yaml` or `template.json` at the root, plus any scripts, data files, and `asset_references.yaml`. +- **Archives** — an `.ojd` file (zip format under the hood) containing a job bundle. The template can be at the archive root or inside a single wrapper directory. + +Both formats are supported for local browsing. S3 browsing only supports `.ojd` archives — this is the canonical sharing format. Archives are extracted to a local directory before submission. If an archive contains a single top-level wrapper directory (e.g. `my-bundle/template.yaml` instead of `template.yaml` at the root), the wrapper is detected and the inner directory is used as the bundle path. + +### Backend Abstraction + +To support both local and S3 browsing without coupling the UI to either, introduce a `BundleRepository` protocol: + +```python +@dataclass +class BundleInfo: + """Metadata extracted from a job bundle's template.""" + path: str # Local path, archive path, or s3:// URI + name: str # From template "name" field + description: str # From template "description" field, or "" + step_names: list[str] # Names of each step in the template + parameters: list[dict] # Parameter definitions from the template + +@dataclass +class BrowseEntry: + """A single item in the browser listing.""" + name: str # Display name (folder basename or archive name without extension) + path: str # Full path or S3 URI + is_bundle: bool # True if this is a valid job bundle + is_archive: bool # True if this is an archive file + +class BundleRepository(Protocol): + def list_entries(self, path: str) -> list[BrowseEntry]: + """List immediate children of `path`. Returns folders, archives, and bundles.""" + ... + + def get_bundle_info(self, path: str) -> Optional[BundleInfo]: + """Load and return metadata for the bundle at `path`, or None if invalid.""" + ... + + def root_path(self) -> str: + """The starting path for browsing.""" + ... +``` + +Two implementations: + +- `LocalBundleRepository` — walks the local filesystem. Lists directories and archive files. Directories are bundles if they contain `template.yaml`/`template.json`. Archives are always shown as bundles (validated on preview). Provides `extract_bundle()` for extracting archives to a local directory. +- `S3BundleRepository` — lists objects and prefixes under the queue's job attachment bucket at `{rootPrefix}/job-bundles/`. Only `.ojd` archives are recognized as bundles; subfolders are shown for navigation only. Provides `download_full_bundle()` which handles archive download+cache+extract. The `from_config()` classmethod encapsulates all initialization logic (session creation, queue lookup, settings extraction) to avoid duplicating this across callers. + +### S3 Bucket Convention + +The S3 bundle repository browses: + +``` +s3://{s3BucketName}/{rootPrefix}/job-bundles/ +``` + +Where `s3BucketName` and `rootPrefix` come from the selected queue's `jobAttachmentSettings`. This means: + +- No extra configuration is needed — the bucket is derived from the queue the user already has selected. +- Users (or admins) place job bundles as `.ojd` archives in the `job-bundles/` folder within the queue's attachment bucket. +- Subfolders within `job-bundles/` are supported for organization but only `.ojd` files are recognized as bundles. + +Example S3 layout: +``` +s3://my-farm-bucket/DeadlineCloud/job-bundles/ + blender-render.ojd + maya-arnold.ojd + rendering/ + custom-renderer.ojd +``` + +### Job History Source + +The Job History source browses the job history directory for the current AWS profile, as configured by `settings.job_history_dir` (default: `~/.deadline/job_history/{aws_profile_name}`). This directory contains bundles from previous submissions, organized by date. + +This is useful for: +- Re-submitting a previous job with modified parameters. +- Using a previously submitted bundle as a starting point for a new submission. +- Reviewing what was submitted in the past. + +The Job History source uses the same `LocalBundleRepository` as the Local source, just rooted at the job history directory instead of the user's home or configured default. + +### Archive Caching + +Archive bundles (both local `.ojd` files and S3 archives) are cached in a unified location to avoid redundant extraction and downloads. + +**Cache location**: `~/.deadline/cache/job-bundles/{hash}/{bundle-name}/` + +Where `{hash}` is a truncated SHA-256 of the source identifier: +- **S3 archives**: `hash(bucket/s3-key)` +- **Local archives**: `hash(local-file-path)` + +**Cache validation**: +- **S3**: A `head_object` call retrieves the archive's ETag. If it matches the cached ETag, the local copy is used. If it differs (or no cache exists), the archive is re-downloaded and re-extracted. +- **Local**: The file's mtime is compared to the cached mtime. If it differs, the archive is re-extracted in-place. + +**Cache metadata** (`.bundle_cache_meta.json`): +```json +{ + "etag": "\"d41d8cd98f00b204e9800998ecf8427e\"", + "last_modified": "2026-04-30T12:00:00+00:00" +} +``` + +For S3 archives, `etag` is used for validation. For local archives, `mtime` (float, seconds since epoch) is stored instead. The fields present indicate the source type. + +**Cleanup**: Extracted archives live in the local cache directory and can be removed manually if needed. There is no separate temp dir or `atexit` cleanup — all extracted archives live in the cache. + +### S3 Object Metadata for Preview + +When `deadline bundle upload` uploads an archive, it attaches bundle metadata as S3 user metadata on the object: + +- `ojd-name`: The template's `name` field (hard cap: 256 chars) +- `ojd-desc`: The template's `description` field, newlines collapsed to spaces (hard cap: 600 chars) +- `ojd-steps`: Comma-separated list of step names (dynamically allocated) +- `ojd-params`: Comma-separated `name:type` pairs (dynamically allocated) +- `ojd-step-count`: Total number of steps in the template (always included, enables accurate count display when steps are truncated) +- `ojd-param-count`: Total number of parameters in the template (always included, enables accurate count display when params are truncated) + +**Budget allocation**: S3 user-defined metadata is limited to 2 KB total (sum of all UTF-8 encoded keys and values, including the `x-amz-meta-` prefix). We reserve 256 bytes for customer-defined metadata on the same object, leaving ~1,792 bytes for bundle preview metadata. The budget is allocated dynamically with the following priority: + +1. **Name** (hard cap 256 chars) and **Description** (hard cap 600 chars) — always allocated first. +2. **Step count** and **Param count** — always included (tiny, ~10 bytes total). +3. **Steps** and **Params** — split the remaining budget evenly between them. If only one is present, it gets the full remainder. + +This means bundles with short names and descriptions get more space for steps/params, while bundles with long descriptions still get a fair split. Constants are defined in `repository.py` (`METADATA_LIMIT_NAME`, `METADATA_LIMIT_DESC`, `S3_METADATA_TOTAL_BUDGET`). See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata + +When truncation occurs, the CLI emits a yellow warning (e.g. `Warning: Bundle metadata 'ojd-params' truncated from 899 to 520 characters`) and the truncated value ends with `...` to make it visually obvious in the preview that information was cut off. The browser dialog detects truncation and uses the count metadata to show accurate totals (e.g. "Parameters (12)" even when only 8 fit in the metadata). When count metadata is unavailable (older uploads), the count displays as "N+" to indicate more exist. + +This metadata is returned by `head_object`, which is already called for ETag validation. This means preview of uploaded archives requires **zero downloads** — a single `head_object` provides both cache validation and all preview information. When a valid local cache exists (ETag matches), the cached template is preferred over metadata as it includes full parameter values and defaults. + +### Detection: What Is a Job Bundle? + +- **Directories** (local or S3 prefix): contains `template.yaml` or `template.json`. +- **Archives** (local file or S3 object): filename ends with `.ojd`. Validated by reading the template from inside the archive on preview. + +For `list_entries`, detection is kept fast: + +- **Local directories**: stat check for template file existence (no parsing). +- **Local archives**: matched by `.ojd` extension, then validated by checking for a template inside the archive. This prevents random files from appearing as bundles. Archive scanning can be disabled via `include_archives=False` on `LocalBundleRepository` (used via `--no-archives` in the CLI). The browser dialog shows both directory bundles and `.ojd` archives for a consistent experience across Local and Queue sources. +- **S3 folders**: shown for navigation only (expandable in the tree), never treated as bundles. +- **S3 archives**: matched by `.ojd` extension only (no API call). + +Full template parsing happens only in `get_bundle_info` when the user clicks a bundle for preview. + +### Browser Dialog UI + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Job Bundle Browser │ +├─────────────────────────────────────────────────────────────┤ +│ Source: (•) Queue ( ) Local ( ) History │ +│ Path: [/job-bundles/ ] │ +├────────────────────────────────┬────────────────────────────┤ +│ [Filter bundles... ] ☐ Hidden│ Name: Blender Render │ +│ 📁 my-bundles/ │ Description (1): │ +│ 📦 blender-render │ Renders a Blender scene │ +│ 📦 maya-arnold │ file... │ +│ 📁 wip/ │ │ +│ 📦 experimental-job │ Steps (1): │ +│ 📦 simple-job/ │ • RenderBlender │ +│ │ │ +│ │ Parameters (2): │ +│ │ ┌──────────┬──────┬─────┐ │ +│ │ │ Name │ Type │ Val │ │ +│ │ ├──────────┼──────┼─────┤ │ +│ │ │ Frames │ Text │ │ │ +│ │ │ OutputDir│ Path │ │ │ +│ │ └──────────┴──────┴─────┘ │ +│ │ [Download bundle]│ +├────────────────────────────────┴────────────────────────────┤ +│ [Cancel] [Select] │ +└─────────────────────────────────────────────────────────────┘ +``` + +**Top bar** — Source selection, path, and options: +- Radio toggle between Queue, Local, and History sources. Queue is selected by default when available; otherwise Local is selected. Queue option is disabled if the queue has no job attachment settings or access fails. When Queue is unavailable, an inline warning label appears below the radio buttons explaining why (e.g. "⚠ **Queue browsing unavailable:** AccessDeniedException..."). The warning uses theme-aware, WCAG-AA contrast colors derived from the Qt palette. +- Path display showing the current browse location (read-only, positioned under the source selector). +- "Show hidden" checkbox — grouped with the filter as a view control. Unchecked by default, toggling refreshes the tree to include/exclude hidden items. For Local/History sources, this means dot-prefixed directories. For the Queue source, this means bundles the user has hidden in their local, per-user view (see [Bundle Visibility](#bundle-visibility)). + +**Left panel** — Filter and navigable tree view: +- A text filter at the top that narrows the tree as you type. Case-insensitive, matches against entry names. Uses recursive filtering so parent folders remain visible when a child matches. The tree auto-expands when filtering to show results. +- Shows folders (📁) and job bundles (📦) with distinct icons. Both directory bundles and archive bundles use the 📦 icon. Local archives show the `.ojd` extension to distinguish them from directory bundles. +- Folders are listed before bundles at each level; both sorted alphabetically within their group. +- Clicking a folder clears any active filter, expands the folder to show its children, and scrolls it to the top of the view. This makes the search-then-navigate flow natural: search for a folder, click it, see its contents. +- Job bundles are leaf nodes (selectable, not expandable). +- Non-bundle, non-archive files are hidden. +- Hidden folders (names starting with `.`) and hidden S3 bundles are hidden by default; toggled via the "Show hidden" checkbox. +- **Context menu** (Queue source only): Right-clicking a visible bundle shows "Hide bundle"; right-clicking a hidden bundle (when "Show hidden" is checked) shows "Unhide bundle". Hidden bundles are rendered with a dimmed/grayed icon to distinguish them from visible ones. Hide/unhide operations happen in the background with automatic retry on conflict (see [Bundle Visibility](#bundle-visibility)). + +**Right panel** — Preview (shown when a bundle is selected, scrollable): +- When no bundle is selected, a centered empty state is shown with a bundle glyph and prompt. +- All bundle-derived text (name, description, step names) is rendered as **plain text** (`Qt.PlainText`), never rich text. Since a name/description/step value can come from attacker-influenced S3 metadata, this ensures a crafted value cannot inject HTML/CSS into the preview (Qt labels have no script engine, so this is injection hardening, not RCE prevention). Widget-level styling (fonts, colors, word-wrap) is unaffected. Parameter values render in a `QTableWidget`, which is plain-text by default. +- **Name**: From the template's `name` field, shown as-is (with `{{Param.X}}` references unresolved). +- **Description**: From the template's `description` field, if present. Multi-line descriptions are normalized to wrap to the panel width. Section label shows item count. Expanded by default. +- **Parameters**: Rendered as a table (indented under its section header) with Name, Type, and Value columns. Type uses friendly labels (Text, Path, Number instead of STRING, PATH, INT). Parameters with no default are marked `(required)`. Section label shows item count. Table is sized to content (no large empty void). If parameters were truncated in S3 metadata, the last garbled entry is dropped and a gray "… additional parameters not shown" row is appended. Ordered before Steps and **expanded by default**, since parameters are the most commonly inspected detail before submitting. +- **Steps**: List of step names from the template, in definition order. Section label shows item count. Expanded by default. +- The panel uses a raised, rounded surface with subtle gradient, visually distinct from the tree. Colors are theme-aware (derived from the Qt palette). +- **Download / Open bundle** button — pinned to the bottom-right of the preview panel, shown only while a bundle is being previewed (hidden in the empty and error states). For Queue bundles (fetched over the network) it reads **"Download bundle"** and shows the size (e.g. "Download bundle (12.3 MB)"), taken from the preview's `head_object` `ContentLength` so no extra call is made. For Local/History bundles (which open in place) it reads **"Open bundle"** with no size. It opens the bundle in the OS file explorer (Finder / Explorer / `xdg-open`) without closing the dialog. Queue bundles are downloaded — and archives extracted — to the local cache first (reusing the Select resolution flow, so the download progress dialog appears and is cancellable); Local and History bundles open in place. Failures are surfaced inline in the preview panel, not as popups. + +**Bottom bar**: +- Cancel and Select buttons. Select is enabled only when a valid bundle is highlighted. + +### Export Bundle + +The submitter dialog's "Export bundle" button replaces the previous separate "Export" and "Share" buttons with a unified flow. Clicking it opens an export dialog: + +``` +┌─ Export Bundle ─────────────────────────────┐ +│ │ +│ Name: [blender-render_________] │ +│ │ +│ Save to: │ +│ (•) Queue ( ) Local │ +│ │ +│ ⚠ Queue unavailable: AccessDenied... │ +│ (inline warning, shown only when Queue │ +│ is disabled) │ +│ │ +│ Location: [s3://bucket/DC/job-bundles/] │ +│ (read-only for Queue, editable for Local) │ +│ │ +│ [Cancel] [Export] │ +└─────────────────────────────────────────────┘ +``` + +**Name** — defaults to the job name with `{{Param.X}}` references resolved using current parameter values. Editable. Used as the `.ojd` filename for Queue or the directory name for Local. + +**Save to** — Queue or Local: +- **Queue**: archives the bundle as `.ojd` and uploads to the queue's S3 `job-bundles/` folder. S3 user metadata (name, description, steps, parameters) is attached for zero-download preview. If a bundle with the same name already exists, the user is prompted to confirm overwrite. When Queue is unavailable (no permissions, no farm/queue configured, no job attachment settings), the radio button is disabled and an inline warning label explains why. +- **Local**: saves the bundle as a directory to the specified location. Defaults to `settings.job_bundle_default_directory` — the same path the browser's Local source browses. The exported bundle immediately appears when browsing Local. + +**Location** — always visible, updates based on the selected source: +- **Queue selected**: shows the S3 path (e.g. `s3://bucket/DeadlineCloud/job-bundles/`), read-only. +- **Local selected**: shows the local directory path, editable with a folder picker button for override. + +Queue export is enabled when the API is available and a farm and queue are configured. Local export is always available. + +Note: the job history directory (`settings.job_history_dir`) is still used internally during Submit to record what was submitted, but Export now targets user-visible locations (Local browse path or Queue) rather than the history directory. + +### Lazy Loading + +The tree is populated lazily — only the children of expanded nodes are fetched. This keeps the initial load fast and avoids scanning deep directory trees or making excessive S3 API calls. + +As an exception, the **immediate children of each top-level folder are preloaded one level deep** (`_preload_first_level`, sharing `_load_children` with the on-expand handler). This lets the text filter match one level below the root without the user first expanding folders, while keeping the up-front cost bounded (a directory read, or one S3 list per top-level folder). Deeper levels remain lazy — the filter only searches what has been loaded (root + the preloaded level + any folders the user has since expanded). + +### Preview Prefetch (background, Queue source) + +The Queue listing itself is fast: a single paginated `list_objects_v2` plus one local read of the per-user visibility file (see [Bundle Visibility](#bundle-visibility)) — no per-bundle S3 calls are on the critical path, so the tree renders immediately regardless of how many bundles the queue holds. + +Once the listing is on screen, the dialog warms a **preview prefetch cache** off the UI thread (`S3BundleRepository.prefetch_previews()` run in a `QThread` from `_on_s3_refresh_done`). It issues a `head_object` per `.ojd` object in parallel (bounded by `PREVIEW_PREFETCH_MAX_WORKERS`, default 16) and stashes each response — ETag, S3 user metadata, and `ContentLength` — in `_head_cache`. The preview (`_get_archive_bundle_info`), size (`get_bundle_size`), and download (`_resolve_archive_bundle`) paths consult that cache first, so clicking a bundle whose HEAD has already been warmed needs **zero** extra S3 round-trips. + +Key properties: +- **Decoupled from visibility.** Visibility is a local file read; the prefetch is a pure preview optimization. Neither depends on the other. +- **Off the critical path.** The prefetch never blocks the listing — a slow or failing prefetch just means the first preview falls back to an on-demand `head_object`. Individual HEAD failures are logged and ignored. +- **Rebuilt per refresh.** A full prefetch clears `_head_cache` and repopulates it, dropping entries for bundles that no longer exist. The S3 client's connection pool is sized to `max(settings.s3_max_pool_connections, PREVIEW_PREFETCH_MAX_WORKERS)` so the parallel HEADs don't exhaust it. + +### Configuration + +Add a new setting for the default local browse directory: + +```python +# In SETTINGS dict in config_file.py +"settings.job_bundle_default_directory": { + "default": "", + "description": ( + "The default local directory to open when browsing for job bundles. " + "If empty, defaults to the user's home directory." + ), +} +``` + +This setting is also exposed in the Deadline Cloud settings dialog (Settings → General settings) as a "Job bundle directory" picker, alongside the existing "Job history directory" setting. + +### CLI Integration + +The `--browse` flag on `deadline bundle gui-submit` opens this new dialog instead of `QFileDialog.getExistingDirectory()`. No new flags needed. When `--browse` is used, the browser dialog opens before the submitter dialog. If the user cancels the browser, the command exits. Additionally, a "Load Bundle" button is added to the submitter dialog's button bar, allowing users to switch bundles mid-session by reopening the browser. + +The "Load a different job bundle" button inside the submitter dialog (`JobBundleSettingsWidget.on_load_bundle`) also uses the new browser dialog, giving users the same browsing experience when switching bundles mid-session. + +### Bundle Resolution Flow + +After the user selects a bundle in the browser, it must be resolved to a local directory for the existing submission pipeline: + +| Source | Format | Resolution | Cleanup | +|---|---|---|---| +| Local | Directory | Used directly (no copy) | None needed | +| Local | Archive (.ojd) | Extracted to cache dir (`hash(path)` + mtime validation) | Local cache dir | +| S3 | Archive (.ojd) | Downloaded to cache dir (`hash(bucket/key)` + ETag validation), extracted | Local cache dir | + +The CLI `deadline bundle download` command downloads the `.ojd` archive, caches it locally with ETag validation, and extracts it to the output directory. + +Once resolved to a local directory, the standard submission flow takes over: `read_job_bundle_parameters()` parses the template and resolves relative PATH defaults against the bundle directory, `apply_job_parameters()` processes asset references, and the job is submitted normally. + +Bundled assets (scripts, data files) with relative paths resolve correctly against the extracted/downloaded directory because the existing path resolution logic operates on the `bundle_dir` path regardless of its origin. + +### Changes to Existing Code + +| File | Change | +|---|---| +| `config/config_file.py` | Add `settings.job_bundle_default_directory` to `SETTINGS` | +| `cli/_groups/bundle_group.py` | Add `deadline bundle list`, `deadline bundle upload`, `deadline bundle download`, `deadline bundle info`, `deadline bundle hide`, and `deadline bundle unhide` commands | +| `ui/dialogs/job_bundle_browser_dialog.py` | **New file.** The browser dialog with filter, Queue/Local/History sources, "Show hidden" toggle, parameter table preview, and right-click context menu for hide/unhide (Queue source). Constructor takes keyword-only args: `queue_source`, `queue_error`, `local_source`, `history_source`. | +| `ui/dialogs/deadline_config_dialog.py` | Add "Job bundle directory" picker to the settings dialog | +| `ui/dialogs/submit_job_to_deadline_dialog.py` | Replace "Export" and "Share" buttons with unified "Export bundle" button that opens the export dialog | +| `ui/dialogs/export_bundle_dialog.py` | **New file.** Export dialog with Queue/Local destination, name override, and location display | +| `ui/widgets/job_bundle_settings_tab.py` | `on_load_bundle` opens the new browser dialog instead of `QFileDialog` | +| `ui/job_bundle_submitter.py` | `show_job_bundle_submitter` uses the new browser dialog when `browse=True`; handles archive extraction and S3 resolution | +| `job_bundle/loader.py` | Add `is_job_bundle_dir(path) -> bool` helper for quick detection | +| `job_bundle/repository.py` | **New file.** `BundleRepository` protocol, `LocalBundleRepository`, `S3BundleRepository` (with `from_config()` factory), archive helpers, cache management, metadata constants, and a local per-user visibility store (`_LocalBundleVisibility`) | + +### CLI Commands + +#### `deadline bundle list [path]` + +Lists job bundles in a local directory or the queue's S3 `job-bundles/` folder. + +- With no arguments, lists bundles in the configured default local directory (`settings.job_bundle_default_directory`, or home if not set). No AWS config needed. +- With `path`, lists bundles in that local directory. +- With `--queue`, lists bundles shared on the queue (requires farm and queue). Hidden bundles are excluded by default. +- With `--queue --show-hidden`, includes hidden bundles in the output (marked with `(hidden)` in plain text, `"hidden": true` in JSON). +- Default output is one bundle name per line, suitable for piping. +- `--output json`: JSON array with name, format (archive/folder), and path. + +``` +$ deadline bundle list +blender-render +maya-arnold + +$ deadline bundle list ./my-bundles +simple-job + +$ deadline bundle list --queue +blender-render +maya-arnold +monte_carlo_simulation + +$ deadline bundle list --queue --show-hidden +blender-render +maya-arnold +monte_carlo_simulation +old-maya-job (hidden) + +$ deadline bundle list --queue --output json +[{"name": "blender-render", "path": "s3://bucket/prefix/job-bundles/blender-render.ojd", "format": "archive"}, ...] + +$ deadline bundle list | head -1 | xargs deadline bundle gui-submit --browse +``` + +The plain-text output enables chaining with other commands — e.g. selecting a bundle interactively with `fzf`: + +``` +$ deadline bundle submit $(deadline bundle download $(deadline bundle list | fzf) -o /tmp/bundles) +``` + +Use `jq` with JSON output to filter by format or extract paths: + +``` +$ deadline bundle list --output json | jq -r '.[] | select(.format == "archive") | .name' +blender-render +maya-arnold + +$ deadline bundle list --output json | jq -r '.[0].path' +s3://my-farm-bucket/DeadlineCloud/job-bundles/blender-render.ojd +``` + +#### `deadline bundle upload ` + +Uploads a local job bundle to share on the queue as an `.ojd` archive. Accepts either a bundle directory or an existing `.ojd` archive file. + +- When given a directory, archives it as `.ojd` and uploads. Symlinks are skipped (not followed). +- When given an `.ojd` file, uploads it directly without re-archiving. +- `--name`: Override the bundle name (defaults to the directory/file name). +- `--profile`, `--farm-id`, `--queue-id`: Standard config overrides. +- If a bundle with the same name already exists, prompts for confirmation before overwriting. + +``` +$ deadline bundle upload ./my-render-job +Uploaded bundle to s3://my-farm-bucket/DeadlineCloud/job-bundles/my-render-job.ojd + +$ deadline bundle upload ./my-render-job.ojd +Uploaded bundle to s3://my-farm-bucket/DeadlineCloud/job-bundles/my-render-job.ojd + +$ deadline bundle upload ./my-render-job --name custom-name +Uploaded bundle to s3://my-farm-bucket/DeadlineCloud/job-bundles/custom-name.ojd + +$ deadline bundle upload ./my-render-job +Bundle 'my-render-job' already exists on the queue. Overwrite? [y/N]: y +Uploaded bundle to s3://my-farm-bucket/DeadlineCloud/job-bundles/my-render-job.ojd +``` + +#### `deadline bundle download ` + +Downloads a shared job bundle from the queue. + +- Finds the `.ojd` archive matching the given name. +- Uses the ETag cache (same as the browser dialog) — repeated downloads are instant if the archive hasn't changed. +- Copies the resolved bundle to the output directory (cache is used internally but the user gets a clean copy at their requested location). +- `-o, --output-dir`: Local directory to download to (defaults to `.`). +- `--profile`, `--farm-id`, `--queue-id`: Standard config overrides. + +``` +$ deadline bundle download blender-render +Downloaded bundle to: ./blender-render + +$ deadline bundle download blender-render -o /tmp/bundles +Downloaded bundle to: /tmp/bundles/blender-render +``` + +#### `deadline bundle hide ` + +Hides a shared bundle in **your** view. The bundle stays on the queue for everyone else; it is just no longer shown in your browser or your `deadline bundle list` by default. + +- Records the name in your local per-user view file (no S3 calls, no permissions needed). +- `--profile`, `--farm-id`, `--queue-id`: Standard config overrides (used to identify which queue's view to update). +- No-op if the bundle is already hidden in your view. + +``` +$ deadline bundle hide blender-render +Hidden bundle: blender-render + +$ deadline bundle hide blender-render +Bundle already hidden: blender-render +``` + +#### `deadline bundle unhide ` + +Unhides a bundle in your view, making it visible again in your browser and `deadline bundle list`. + +- Removes the name from your local per-user view file (no S3 calls). +- `--profile`, `--farm-id`, `--queue-id`: Standard config overrides. +- No-op if the bundle is not hidden in your view. + +``` +$ deadline bundle unhide blender-render +Unhidden bundle: blender-render + +$ deadline bundle unhide blender-render +Bundle is not hidden: blender-render +``` + +#### `deadline bundle info ` + +Shows detailed information about a job bundle — the same data shown in the browser's preview panel. + +- Without `--queue`, `bundle_name` is treated as a local path to a job bundle directory. If the path doesn't exist, searches by name in the current directory and then the configured `settings.job_bundle_default_directory`. +- With `--queue`, looks up the named bundle on the queue (uses S3 metadata for zero-download preview when available). +- Output always includes the resolved path so the user knows where the bundle was found. +- `--output json`: JSON object with path, name, description, steps, and parameters. +- `--profile`, `--farm-id`, `--queue-id`: Standard config overrides (with `--queue`). + +``` +$ deadline bundle info ./my-render-job +Path: /home/user/my-render-job +Name: Blender Render +Description: Renders a Blender scene file using Cycles +Steps: + • RenderBlender +Parameters: + Frames (STRING) = 1-100 + OutputDir (PATH) = /tmp/output + +$ deadline bundle info blender-render --queue +Path: s3://my-farm-bucket/DeadlineCloud/job-bundles/blender-render.ojd +Name: Blender Render +Description: Renders a Blender scene file using Cycles +Steps: + • RenderBlender +Parameters: + Frames (STRING) = 1-100 + OutputDir (PATH) = /tmp/output + +$ deadline bundle info blender-render --queue --output json +{ + "path": "s3://my-farm-bucket/DeadlineCloud/job-bundles/blender-render.ojd", + "name": "Blender Render", + "description": "Renders a Blender scene file using Cycles", + "steps": ["RenderBlender"], + "parameters": [ + {"name": "Frames", "type": "STRING", "_display_value": "1-100"}, + {"name": "OutputDir", "type": "PATH", "_display_value": "/tmp/output"} + ] +} +``` + +### Bundle Visibility + +A shared bundle can be "hidden" so it doesn't clutter the browser or `deadline bundle list`. Hiding is a **private, per-user view preference** — it does not change anything on S3 and does not affect what other users see. Showing or hiding a bundle only changes your own listing. + +**Mechanism — local per-user view file:** + +Hidden bundle names are stored in a small local JSON file, one **per queue** (in that queue's folder inside the bundle cache, keyed by bucket + prefix). The file is hidden (dot-prefixed) — it's a private view store, not meant for manual editing: + +``` +~/.deadline/cache/job-bundles/{hash(bucket/prefix)}/.visibility.json +``` + +```json +{ + "version": 1, + "hidden": ["blender-render", "rendering/old-maya-job"] +} +``` + +The `hidden` array holds bundle names relative to `job-bundles/` (subfolder paths preserved, e.g. `"rendering/old-maya-job"`), without the `.ojd` extension. + +- `get_hidden_set()` reads that file and returns the set of hidden names (empty if the file is missing, unreadable, or malformed). **No S3 calls.** +- `set_bundle_visibility(name, hidden=...)` reads the current set, adds/removes the name, and writes it back atomically (temp file + `os.replace`). **No S3 calls.** + +The file lives in the queue's own folder inside the bundle cache as `.visibility.json` — one per queue. Visibility is a per-queue, per-user concept, so it sits alongside that queue's cached data. + +**Why local instead of on S3?** +- **Per-user, not shared.** One person hiding a bundle from their own view shouldn't remove it from everyone else's. Visibility is a personal UI preference, like collapsing a folder. +- **No write permissions or object rewrites needed.** Hiding requires nothing on S3 — no `s3:PutObject`, no object rewrite, no ETag churn, and no 5 GB copy limit. Read-only users can hide/show freely. +- **Instant.** Reading and writing the hidden set is a local file operation; it never scales with the number of bundles on the queue. + +**Browser behavior:** + +- When "Show hidden" is unchecked (default), locally-hidden bundles are excluded from the tree. +- When "Show hidden" is checked, hidden bundles appear with a dimmed/grayed 📦 icon. +- Right-click context menu on bundles (Queue source only): + - Visible bundle → "Hide bundle" + - Hidden bundle → "Unhide bundle" +- After hide/unhide, the tree refreshes to reflect the change (a local file write). + +**Sync behavior:** Because the hidden set is local and per-user, there is nothing to sync. If a bundle is deleted from the queue, a stale entry may remain in the local file; it is simply ignored during listing (it matches no existing bundle) and is harmless. + +**Permissions:** Browsing/reading and previewing bundles requires `s3:ListBucket`, `s3:GetObject` (and `s3:GetObjectAttributes`/HeadObject) on the bucket. Hiding/unhiding requires **no** S3 permissions at all — it only writes a local file. + +### Progress Indication + +Operations that involve network I/O show progress to the user: + +- **Browser dialog**: Selecting an S3 bundle and clicking "Select" opens a cancellable progress dialog (`QDialog` + `QProgressBar`) showing download progress via the S3 `download_fileobj` callback delivering updates over Qt signals from the background transfer thread. The progress bar is scaled to KiB and the size signal is typed as `qlonglong` to avoid 32-bit int overflow for large (multi-GiB) bundles, and the label is formatted with `human_readable_file_size` (e.g. "1.2 GB / 2.1 GB"). +- **Save to Queue (GUI)**: Archiving and upload run on a background `QThread`. A `QProgressDialog` shows two phases: + 1. "Archiving bundle... X MB / Y MB" — progress updates per file (small files use fast `zf.write()`, files >8MB use chunked 4MB writes for smoother progress). + 2. "Uploading bundle... X MB / Y MB" — byte-level progress via `upload_fileobj` callback. + On completion, the dialog shows "Bundle saved to queue." with a Close button. All values are scaled to KB to avoid 32-bit int overflow for large bundles. +- **CLI**: `deadline bundle upload` shows a click progress bar with bytes uploaded. `deadline bundle download` shows a progress bar with bytes downloaded. `deadline bundle hide`/`unhide` complete fast enough to not need progress. + +### Error Handling + +Errors are displayed inline rather than as popup dialogs: + +- **Queue unavailable** (no farm/queue, no JA settings, auth failure): The Queue radio button is disabled and a styled inline warning label appears below the source selector showing the reason (e.g. "⚠ **Queue browsing unavailable:** AccessDeniedException..."). +- **Listing failure** (network error, permissions): The preview panel shows "⚠ Error" in red with the error message. +- **Expand failure** (subfolder listing fails): A disabled `⚠ Error: {message}` entry appears in the tree under that folder. +- **Preview failure** (malformed template, missing fields): The preview panel shows "⚠ Error" with "Could not read bundle template" and the tree entry icon changes from 📦 to ⚠. +- **Double-click**: Double-clicking a bundle selects it and accepts the dialog. Double-clicking a folder does nothing. +- **Hide/unhide failure** (insufficient permissions or conflict): The context menu action is always shown. If the operation fails (e.g. `AccessDeniedException`), an inline warning appears below the tree: "⚠ Could not hide bundle: AccessDeniedException". + +### Archive Safety + +Archives are validated before extraction to prevent path traversal attacks: + +- All entry paths are checked for absolute paths and `../` traversal using `os.path.commonpath()` with `os.path.realpath()` — this handles mixed path separators on Windows. The entire archive is rejected if any entry would extract outside the target directory. + +Resource-exhaustion / zip-bomb protection is enforced before extraction. Checks read the zip central directory (`infolist()`), so nothing is decompressed to validate; `zipfile` also caps actual output at each entry's declared `file_size`, making these central-directory values a sound upper bound. The archive is rejected if: + +- It has more than `MAX_ARCHIVE_ENTRIES` (100,000) entries. +- Its declared uncompressed size exceeds `max(MAX_ARCHIVE_UNCOMPRESSED_FLOOR, compressed × MAX_ARCHIVE_COMPRESSION_RATIO)` — i.e. `max(256 MB, compressed × 200)`. A generous absolute floor is always permitted; above it the expansion ratio must be plausible for real data. There is no absolute size ceiling, so legitimately large (low-ratio) bundles still extract. +- The extracted payload wouldn't fit in the destination filesystem's free space. +- The template entry is larger than `MAX_TEMPLATE_BYTES` (16 MB), refused when reading the template (a template, even with embedded scripts, is small). + +Symlink protection during upload: + +- `os.walk(followlinks=False)` is used when archiving bundles. Symlinked files and directories are skipped to prevent unintended inclusion of files outside the bundle directory. + +### Bundle Name Validation + +Upload rejects bundles with invalid names: + +- Empty names or names consisting only of whitespace/slashes are rejected with an error directing the user to `--name`. +- The full S3 key (prefix + name + `.ojd`) is validated against S3's 1024-character key limit. +- Control characters (0x00–0x1F, 0x7F) are considered invalid. + +On download, the bundle name is sanitized for the local filesystem in a platform-specific manner: + +- **POSIX** (macOS/Linux): `/` and control characters (0x00–0x1F, 0x7F) are replaced with `_`. While most control characters are technically valid in POSIX filenames, they cause issues in terminals and scripts. +- **Windows**: `\ / : * ? " < > |` and control characters are replaced with `_`. + +This means the S3 key preserves the original name as-is (all characters are valid in S3 keys), and only the local directory name is adjusted for the user's OS. + +### S3 Considerations + +- **Authentication**: S3 browsing and CLI commands use `api.get_boto3_session()` which respects the configured AWS profile in `~/.deadline/config`. The `S3BundleRepository.from_config()` factory method encapsulates session creation, queue lookup, and settings extraction in one place. No separate auth flow. +- **Permissions**: Requires `s3:ListBucket` and `s3:GetObject` on the queue's attachment bucket for browsing/preview/download. Upload additionally requires `s3:PutObject`. Hiding/unhiding requires **no** S3 permissions — it is a local, per-user view preference (see [Bundle Visibility](#bundle-visibility)). If access is denied, show an error rather than crashing. +- **Performance**: Listing is a single paginated `list_objects_v2` call with delimiter. Archive preview with S3 metadata is 1 `head_object` (no download). Cached archive selection is 1 `head_object`. +- **S3 object metadata**: `deadline bundle upload` attaches bundle name, description, steps, and parameters as S3 user metadata. This enables zero-download preview via `head_object`. Archives uploaded by other means fall back to downloading the archive for preview. +- **Bundled assets**: Scripts, data files, and other assets within the bundle are included in the archive. Relative PATH parameters resolve against the extracted copy. + +### MCP Server Integration + +The MCP server exposes bundle sharing operations as tools for AI assistants: + +- **list_shared_bundles** — Lists bundles on the queue (respects visibility, supports `show_hidden`). +- **upload_bundle** — Uploads a local job bundle (directory or `.ojd` archive) to the queue. Archives are uploaded directly without re-archiving. +- **download_bundle** — Downloads a shared bundle from the queue to a local directory. + +These tools use the same `S3BundleRepository` as the CLI and GUI, so behavior is consistent. Hide/unhide is not exposed via MCP — it's a management action better suited to direct user intent via CLI or GUI. + +## Out of Scope (Future) + +- Favoriting or pinning frequently used bundles. +- Browsing bundles from a Deadline Cloud service API (e.g. farm-level bundle registry). +- Configurable S3 bucket/prefix (currently always derived from the queue). +- Cache size limits or TTL-based eviction. +- **Persisting the visibility/preview scan across sessions.** Today the hidden-set scan (list + parallel HEADs) and its `_head_cache` prefetch live only for the lifetime of a repository instance and are rebuilt on every listing/refresh. A future improvement could persist this data — e.g. a local on-disk index keyed by bucket/prefix with ETag-based validation, or an `If-Modified-Since`/conditional-HEAD refresh that only re-reads changed objects — so repeated browses avoid re-HEADing every bundle. This would recover most of the manifest's single-round-trip snappiness while keeping visibility state on the objects themselves. diff --git a/src/deadline/_mcp/registry.py b/src/deadline/_mcp/registry.py index 83dce456e..25085c16e 100644 --- a/src/deadline/_mcp/registry.py +++ b/src/deadline/_mcp/registry.py @@ -5,7 +5,7 @@ from typing import Any, Callable, List, Optional, TypedDict, Dict from ..client import api -from .tools import job, logs +from .tools import job, logs, bundles class ToolDefinition(TypedDict): @@ -115,4 +115,17 @@ def get_all_tool_names() -> List[str]: "item_offset", ], }, + # Bundle sharing tools + "list_shared_bundles": { + "func": bundles.list_shared_bundles, + "param_names": ["farm_id", "queue_id", "show_hidden"], + }, + "upload_bundle": { + "func": bundles.upload_bundle, + "param_names": ["job_bundle", "name", "farm_id", "queue_id"], + }, + "download_bundle": { + "func": bundles.download_bundle, + "param_names": ["bundle_name", "output_dir", "farm_id", "queue_id"], + }, } diff --git a/src/deadline/_mcp/tools/bundles.py b/src/deadline/_mcp/tools/bundles.py new file mode 100644 index 000000000..e9ab2aaed --- /dev/null +++ b/src/deadline/_mcp/tools/bundles.py @@ -0,0 +1,100 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +""" +Deadline Cloud Bundle sharing tools for MCP. +""" + +import json +from typing import Any, Dict, Optional + +from click.testing import CliRunner + +from ...client.cli import main + + +def list_shared_bundles( + farm_id: Optional[str] = None, + queue_id: Optional[str] = None, + show_hidden: bool = False, +) -> Dict[str, Any]: + """List job bundles shared on the queue. + + Returns a list of bundles with their name and format. + Hidden bundles are excluded by default unless show_hidden is True. + """ + args = ["bundle", "list", "--queue", "--output", "json"] + if farm_id: + args.extend(["--farm-id", farm_id]) + if queue_id: + args.extend(["--queue-id", queue_id]) + if show_hidden: + args.append("--show-hidden") + + runner = CliRunner() + result = runner.invoke(main, args) + + if result.exit_code != 0: + return {"success": False, "error": result.output.strip()} + return {"bundles": json.loads(result.output)} + + +def upload_bundle( + job_bundle: str, + name: Optional[str] = None, + farm_id: Optional[str] = None, + queue_id: Optional[str] = None, +) -> Dict[str, Any]: + """Upload a local job bundle to the queue as a shared .ojd archive. + + Args: + job_bundle: Path to a job bundle directory or .ojd archive file. + name: Override the bundle name (defaults to directory/file name). + farm_id: The farm ID (uses default if not specified). + queue_id: The queue ID (uses default if not specified). + """ + args = ["bundle", "upload", job_bundle] + if name: + args.extend(["--name", name]) + if farm_id: + args.extend(["--farm-id", farm_id]) + if queue_id: + args.extend(["--queue-id", queue_id]) + + runner = CliRunner() + result = runner.invoke(main, args, input="y\n") + + if result.exit_code != 0: + return {"success": False, "error": result.output.strip()} + return {"success": True, "message": result.output.strip()} + + +def download_bundle( + bundle_name: str, + output_dir: Optional[str] = None, + farm_id: Optional[str] = None, + queue_id: Optional[str] = None, +) -> Dict[str, Any]: + """Download a shared bundle from the queue to a local directory. + + Args: + bundle_name: Name of the bundle to download. + output_dir: Local directory to download to (uses cache if not specified). + farm_id: The farm ID (uses default if not specified). + queue_id: The queue ID (uses default if not specified). + """ + args = ["bundle", "download", bundle_name, "--output", "json"] + if output_dir: + args.extend(["-o", output_dir]) + if farm_id: + args.extend(["--farm-id", farm_id]) + if queue_id: + args.extend(["--queue-id", queue_id]) + + runner = CliRunner() + result = runner.invoke(main, args) + + if result.exit_code != 0: + return {"success": False, "error": result.output.strip()} + + data = json.loads(result.output) + return {"success": True, "path": data["path"]} diff --git a/src/deadline/client/cli/_groups/bundle_group.py b/src/deadline/client/cli/_groups/bundle_group.py index a087ec360..b12b0bbc7 100644 --- a/src/deadline/client/cli/_groups/bundle_group.py +++ b/src/deadline/client/cli/_groups/bundle_group.py @@ -22,11 +22,26 @@ from ... import api from ...config import config_file from ...dataclasses import SubmitterInfo +from ...job_bundle.loader import is_job_bundle_dir +from ...job_bundle.repository import ( + BundleRepository, + LocalBundleRepository, + S3BundleRepository, + S3_JOB_BUNDLES_PREFIX, + _parse_template, + archive_bundle_dir, + build_bundle_metadata, + extract_bundle_info, + get_bundle_dir_size, + read_template_from_archive, + sanitize_bundle_name, +) from ....job_attachments.exceptions import ( AssetSyncError, AssetSyncCancelledError, MisconfiguredInputsError, ) +from ....job_attachments._aws.deadline import get_queue from ....job_attachments.models import JobAttachmentsFileSystem from ...exceptions import DeadlineOperationError, CreateJobWaiterCanceled @@ -468,6 +483,16 @@ def bundle_gui_submit( from ...ui._utils import tr with gui_context_for_cli(automatically_install_dependencies=install_gui) as app: + # Pre-warm boto3 session + Deadline client (lru_cached, reused by background thread) + if browse: + try: + from ...api import get_boto3_session, get_boto3_client + + get_boto3_session() + get_boto3_client("deadline") + except Exception: + pass # Non-fatal — background thread will handle it + from ...ui.job_bundle_submitter import show_job_bundle_submitter if not job_bundle_dir and not browse: @@ -525,3 +550,438 @@ def _print_response( click.echo(f"Job ID: {job_id}") else: click.echo("Job submission canceled.") + + +def _get_queue_s3_settings(config): + """Get the queue's job attachment S3 settings from config.""" + farm_id = config_file.get_setting("defaults.farm_id", config=config) + queue_id = config_file.get_setting("defaults.queue_id", config=config) + if not farm_id or not queue_id: + raise DeadlineOperationError( + "A default farm and queue must be configured. Run 'deadline config set defaults.farm_id ' and 'deadline config set defaults.queue_id '." + ) + boto3_session = api.get_boto3_session(config=config) + queue = get_queue(farm_id=farm_id, queue_id=queue_id, session=boto3_session) + if not queue.jobAttachmentSettings: + raise DeadlineOperationError( + f"Queue {queue_id} does not have job attachment settings configured." + ) + # Use queue role credentials for S3 access (required for DCM profiles) + deadline_client = api.get_boto3_client("deadline", config=config) + s3_session = api.get_queue_user_boto3_session( + deadline=deadline_client, config=config, farm_id=farm_id, queue_id=queue_id + ) + return queue.jobAttachmentSettings, s3_session + + +@cli_bundle.command(name="list") +@click.argument("path", required=False) +@click.option( + "--queue", + "use_queue", + is_flag=True, + help="List bundles shared on the queue.", +) +@click.option( + "--show-hidden", + is_flag=True, + help="Include hidden bundles in the output (queue only).", +) +@click.option( + "--no-archives", + is_flag=True, + help="Skip archive files when listing local bundles.", +) +@click.option("--profile", help="The AWS profile to use.") +@click.option("--farm-id", help="The farm to use.") +@click.option("--queue-id", help="The queue to use.") +@click.option( + "--output", + type=click.Choice(["text", "json"], case_sensitive=False), + default="text", + help="Output format. TEXT prints one name per line, JSON prints full details.", +) +@_handle_error +def bundle_list(path, use_queue, show_hidden, no_archives, output, **args): + """ + List job bundles. + + \b + With no arguments, lists bundles in the configured default local directory + (settings.job_bundle_default_directory, or home if not set). + With PATH, lists bundles in that local directory. + With --queue, lists bundles shared on the queue. + """ + + hidden_set: set[str] = set() + if use_queue: + config = _apply_cli_options_to_config(required_options={"farm_id", "queue_id"}, **args) + repo: BundleRepository = S3BundleRepository.from_config(config) + hidden_set = repo.get_hidden_set() # type: ignore[attr-defined] + else: + if path: + local_root = os.path.abspath(path) + else: + local_root = config_file.get_setting("settings.job_bundle_default_directory") + if not local_root: + local_root = os.path.expanduser("~") + local_root = os.path.expanduser(local_root) + repo = LocalBundleRepository(root=local_root, include_archives=not no_archives) + + entries = repo.list_entries(repo.root_path()) + bundles = [e for e in entries if e.is_bundle] + + # Filter hidden bundles unless --show-hidden + if use_queue and not show_hidden: + bundles = [e for e in bundles if e.name not in hidden_set] + + if output == "json": + result = [ + { + "name": e.name, + "path": e.path, + "format": "archive" if e.is_archive else "folder", + **({"hidden": True} if e.name in hidden_set else {}), + } + for e in bundles + ] + click.echo(json.dumps(result, indent=2)) + else: + for e in bundles: + suffix = " (hidden)" if e.name in hidden_set else "" + click.echo(f"{e.name}{suffix}") + + +@cli_bundle.command(name="upload") +@click.argument("job_bundle_dir") +@click.option("--profile", help="The AWS profile to use.") +@click.option("--farm-id", help="The farm to use.") +@click.option("--queue-id", help="The queue to use.") +@click.option( + "--name", + help="Name for the shared archive on the queue. Defaults to the bundle directory name.", +) +@_handle_error +def bundle_upload(job_bundle_dir, name, **args): + """ + Upload a job bundle to share on the queue as an .ojd archive. + """ + config = _apply_cli_options_to_config(required_options={"farm_id", "queue_id"}, **args) + s3_settings, boto3_session = _get_queue_s3_settings(config) + + job_bundle_dir = os.path.abspath(job_bundle_dir) + + # Determine if input is an .ojd archive or a directory + is_archive_input = os.path.isfile(job_bundle_dir) and job_bundle_dir.endswith(".ojd") + + if not is_archive_input and not is_job_bundle_dir(job_bundle_dir): + raise DeadlineOperationError( + f"Directory does not appear to be a job bundle (no template.yaml or template.json): {job_bundle_dir}" + ) + + # Parse the template to extract metadata for S3 object metadata + bundle_metadata = {} + if is_archive_input: + result = read_template_from_archive(job_bundle_dir) + if result: + raw, fname = result + template = _parse_template(raw, fname) + if template: + info = extract_bundle_info(template, job_bundle_dir) + bundle_metadata = build_bundle_metadata(bundle_info=info) + else: + bundle_metadata = build_bundle_metadata(job_bundle_dir) + + bundle_name = name or os.path.basename(job_bundle_dir) + if is_archive_input and bundle_name.endswith(".ojd"): + bundle_name = bundle_name[:-4] + # Sanitize the name the same way the GUI export path does: replace characters + # that are unsafe in a filename and reject empty or path-traversal names. This + # keeps CLI and GUI behavior consistent and prevents a name like "../foo" or + # "a/b" from being written to an arbitrary S3 sub-prefix (or with ".." + # segments) that the browser and `bundle list` would not find consistently. + try: + bundle_name = sanitize_bundle_name(bundle_name) + except ValueError: + raise DeadlineOperationError( + "Bundle name is empty or invalid. Use --name to specify a valid name." + ) + prefix = f"{s3_settings.rootPrefix.rstrip('/')}/{S3_JOB_BUNDLES_PREFIX}" + s3_key = f"{prefix}/{bundle_name}.ojd" + if len(s3_key) > 1024: + raise DeadlineOperationError( + f"Bundle name is too long. S3 key would be {len(s3_key)} characters (max 1024)." + ) + + s3 = boto3_session.client("s3") + + # Check if bundle already exists + try: + s3.head_object(Bucket=s3_settings.s3BucketName, Key=s3_key) + if not click.confirm(f"Bundle '{bundle_name}' already exists on the queue. Overwrite?"): + click.echo("Upload canceled.") + return + except ClientError as e: + if e.response["Error"]["Code"] != "404": + raise + + # Archive and upload + # Advertise the archive's true type as a courtesy hint for other consumers. + # The download/browse paths do NOT trust ContentType (it's set by the + # uploader) and validate by parsing the zip; this just keeps the object + # correctly typed. Kept consistent with S3BundleRepository.upload_archive. + extra_args: dict = {"ContentType": "application/zip"} + if bundle_metadata: + extra_args["Metadata"] = bundle_metadata + if is_archive_input: + # Already an .ojd — upload directly + file_size = os.path.getsize(job_bundle_dir) + with ( + open(job_bundle_dir, "rb") as f, + click.progressbar(length=file_size, label="Uploading") as bar, # type: ignore[var-annotated] + ): + s3.upload_fileobj( + f, + s3_settings.s3BucketName, + s3_key, + ExtraArgs=extra_args, + Callback=lambda bytes_sent: bar.update(bytes_sent), + ) + else: + total_size = get_bundle_dir_size(job_bundle_dir) + with click.progressbar(length=total_size, label="Archiving") as bar: # type: ignore[var-annotated] + buf = archive_bundle_dir(job_bundle_dir, progress_callback=lambda n: bar.update(n)) + + file_size = buf.getbuffer().nbytes + with click.progressbar(length=file_size, label="Uploading") as bar: # type: ignore[var-annotated] + s3.upload_fileobj( + buf, + s3_settings.s3BucketName, + s3_key, + ExtraArgs=extra_args, + Callback=lambda bytes_sent: bar.update(bytes_sent), + ) + click.echo(f"Uploaded bundle to s3://{s3_settings.s3BucketName}/{s3_key}") + + +@cli_bundle.command(name="download") +@click.argument("bundle_name") +@click.option("--profile", help="The AWS profile to use.") +@click.option("--farm-id", help="The farm to use.") +@click.option("--queue-id", help="The queue to use.") +@click.option( + "-o", + "--output-dir", + default=None, + help="Local directory to copy the bundle to. If not specified, uses the local cache.", +) +@click.option("--output", default="text", help="Output format: text or json.") +@_handle_error +def bundle_download(bundle_name, output_dir, output, **args): + """ + Download a shared job bundle from the queue. + + BUNDLE_NAME is the name of the bundle (e.g. 'blender-render'). + """ + + config = _apply_cli_options_to_config(required_options={"farm_id", "queue_id"}, **args) + repo = S3BundleRepository.from_config(config) + + if output_dir: + output_dir = os.path.abspath(output_dir) + os.makedirs(output_dir, exist_ok=True) + + # List entries to find the bundle by name + entries = repo.list_entries(repo.root_path()) + match = None + for entry in entries: + if entry.name == bundle_name and entry.is_bundle: + match = entry + break + + if not match: + available = [e.name for e in entries if e.is_bundle] + msg = f"Bundle '{bundle_name}' not found in {repo.root_path()}" + if available: + msg += f"\nAvailable bundles: {', '.join(available)}" + raise DeadlineOperationError(msg) + + # Get file size for progress bar + file_size = repo.get_bundle_size(match.path) + + # Download and extract are sequential inside download_full_bundle. + _bars: dict = {} + + def _dl_callback(n): + if "dl" not in _bars: + _bars["dl"] = click.progressbar(length=file_size, label="Downloading") + _bars["dl_ctx"] = _bars["dl"].__enter__() + _bars["dl_ctx"].update(n) + + def _ex_callback(n): + if "dl" in _bars and "dl_closed" not in _bars: + _bars["dl_closed"] = True + _bars["dl"].__exit__(None, None, None) + if "ex" not in _bars: + _bars["ex"] = click.progressbar( + length=_bars.get("ex_size", file_size), label="Extracting" + ) + _bars["ex_ctx"] = _bars["ex"].__enter__() + _bars["ex_ctx"].update(n) + + def _ex_size_callback(total): + _bars["ex_size"] = total + + local_path = repo.download_full_bundle( + match.path, + output_dir, + progress_callback=_dl_callback, + extract_callback=_ex_callback, + extract_size_callback=_ex_size_callback, + ) + if "dl" in _bars and "dl_closed" not in _bars: + _bars["dl"].__exit__(None, None, None) + if "ex" in _bars: + _bars["ex"].__exit__(None, None, None) + # download_full_bundle resolves to cache; copy to user's output_dir if specified + if output_dir: + dest_path = os.path.join(output_dir, sanitize_bundle_name(bundle_name)) + if os.path.exists(dest_path): + shutil.rmtree(dest_path) + shutil.copytree(local_path, dest_path) + result_path = dest_path + else: + result_path = local_path + + if output.lower() == "json": + click.echo(json.dumps({"path": result_path})) + else: + click.echo(f"Downloaded bundle to: {result_path}") + + +@cli_bundle.command(name="hide") +@click.argument("bundle_name") +@click.option("--profile", help="The AWS profile to use.") +@click.option("--farm-id", help="The farm to use.") +@click.option("--queue-id", help="The queue to use.") +@_handle_error +def bundle_hide(bundle_name, **args): + """ + Hide a shared bundle on the queue. + + The bundle remains in S3 but is no longer shown in the browser or + `deadline bundle list` by default. Use --show-hidden to see it. + """ + config = _apply_cli_options_to_config(required_options={"farm_id", "queue_id"}, **args) + repo = S3BundleRepository.from_config(config) + + hidden_set = repo.get_hidden_set() + if bundle_name in hidden_set: + click.echo(f"Bundle already hidden: {bundle_name}") + return + + repo.set_bundle_visibility(bundle_name, hidden=True) + click.echo(f"Hidden bundle: {bundle_name}") + + +@cli_bundle.command(name="unhide") +@click.argument("bundle_name") +@click.option("--profile", help="The AWS profile to use.") +@click.option("--farm-id", help="The farm to use.") +@click.option("--queue-id", help="The queue to use.") +@_handle_error +def bundle_unhide(bundle_name, **args): + """ + Unhide a previously hidden bundle on the queue. + + Makes the bundle visible again in the browser and `deadline bundle list`. + """ + config = _apply_cli_options_to_config(required_options={"farm_id", "queue_id"}, **args) + repo = S3BundleRepository.from_config(config) + + hidden_set = repo.get_hidden_set() + if bundle_name not in hidden_set: + click.echo(f"Bundle is not hidden: {bundle_name}") + return + + repo.set_bundle_visibility(bundle_name, hidden=False) + click.echo(f"Unhidden bundle: {bundle_name}") + + +@cli_bundle.command(name="info") +@click.argument("bundle_name") +@click.option( + "--queue", + "use_queue", + is_flag=True, + help="Inspect a bundle shared on the queue.", +) +@click.option( + "--output", + type=click.Choice(["verbose", "json"], case_sensitive=False), + default="verbose", + help="Output format.", +) +@click.option("--profile", help="The AWS profile to use.") +@click.option("--farm-id", help="The farm to use.") +@click.option("--queue-id", help="The queue to use.") +@_handle_error +def bundle_info(bundle_name, use_queue, output, **args): + """ + Show details about a job bundle (name, description, steps, parameters). + + BUNDLE_NAME is either a local path to a job bundle directory, or the name + of a shared bundle on the queue (when used with --queue). For local bundles, + if the path doesn't exist, searches by name in the current directory and then + the configured job bundle default directory. + """ + if use_queue: + config = _apply_cli_options_to_config(required_options={"farm_id", "queue_id"}, **args) + repo: BundleRepository = S3BundleRepository.from_config(config) + # Find the bundle by name in the listing + entries = repo.list_entries(repo.root_path()) + match = next((e for e in entries if e.name == bundle_name and e.is_bundle), None) + if not match: + available = [e.name for e in entries if e.is_bundle] + msg = f"Bundle '{bundle_name}' not found on queue." + if available: + msg += f"\nAvailable bundles: {', '.join(available)}" + raise DeadlineOperationError(msg) + info = repo.get_bundle_info(match.path) + else: + bundle_path = os.path.abspath(bundle_name) + if not os.path.isdir(bundle_path): + # Search by name in cwd, then configured default directory + for search_dir in [ + os.getcwd(), + os.path.expanduser( + config_file.get_setting("settings.job_bundle_default_directory") or "" + ), + ]: + if not search_dir: + continue + candidate = os.path.join(search_dir, bundle_name) + if os.path.isdir(candidate) and is_job_bundle_dir(candidate): + bundle_path = candidate + break + else: + raise DeadlineOperationError( + f"Bundle '{bundle_name}' not found as a path, in current directory, " + "or in the configured job bundle default directory." + ) + repo = LocalBundleRepository(root=os.path.dirname(bundle_path)) + info = repo.get_bundle_info(bundle_path) + + if not info: + raise DeadlineOperationError( + f"Could not read bundle template for '{bundle_name}'. " + "The template may be missing or malformed." + ) + + if output == "json": + result = info.to_dict() + result["path"] = info.path + click.echo(json.dumps(result, indent=2)) + else: + click.echo(f"Path: {info.path}") + click.echo(info.format_text()) diff --git a/src/deadline/client/config/config_file.py b/src/deadline/client/config/config_file.py index 56709e7d4..f07de0d85 100644 --- a/src/deadline/client/config/config_file.py +++ b/src/deadline/client/config/config_file.py @@ -301,6 +301,13 @@ def get_deadline_regions(config: Optional[ConfigParser] = None) -> List[str]: "default": "20", "description": "The default maximum number of tasks that can fail before the job is marked as failed.", }, + "settings.job_bundle_default_directory": { + "default": "", + "description": ( + "The default local directory to open when browsing for job bundles. " + "If empty, defaults to the user's home directory." + ), + }, } diff --git a/src/deadline/client/job_bundle/loader.py b/src/deadline/client/job_bundle/loader.py index 5094a92e7..ee2ca869f 100644 --- a/src/deadline/client/job_bundle/loader.py +++ b/src/deadline/client/job_bundle/loader.py @@ -3,6 +3,7 @@ from __future__ import annotations __all__ = [ + "is_job_bundle_dir", "parse_yaml_or_json_content", "read_yaml_or_json", "read_yaml_or_json_object", @@ -19,6 +20,14 @@ from ..exceptions import DeadlineOperationError +def is_job_bundle_dir(path: str) -> bool: + """Returns True if the directory contains a template.yaml or template.json file.""" + if not os.path.isdir(path): + return False + template_prefix = os.path.join(path, "template") + return os.path.isfile(template_prefix + ".yaml") or os.path.isfile(template_prefix + ".json") + + def validate_directory_symlink_containment(job_bundle_dir: str) -> None: """ Validates the integrity of the job bundle, validating that all files diff --git a/src/deadline/client/job_bundle/repository.py b/src/deadline/client/job_bundle/repository.py new file mode 100644 index 000000000..969aef863 --- /dev/null +++ b/src/deadline/client/job_bundle/repository.py @@ -0,0 +1,1491 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +""" +Bundle repository abstraction for browsing job bundles from local filesystem or S3. +Supports both directory-based bundles and .ojd archive bundles (zip format). +""" + +from __future__ import annotations + +import base64 +import hashlib +import io +import json +import os +import re +import shutil +import sys +import tempfile +import zipfile +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass, field +from email.header import decode_header +from logging import getLogger +from typing import Optional, Protocol + +import yaml + +from ..config import config_file +from ..config.config_file import get_cache_directory +from ..exceptions import DeadlineOperationError + +logger = getLogger(__name__) + +TEMPLATE_FILENAMES = ("template.yaml", "template.json") +S3_JOB_BUNDLES_PREFIX = "job-bundles" +ARCHIVE_EXTENSION = ".ojd" +CACHE_META_FILENAME = ".bundle_cache_meta.json" + +# ── Bundle visibility (local, per-user) ────────────────────── +# +# "Hiding" a shared bundle is a private, per-user view preference stored in a +# local JSON file — it never touches S3 and does not affect other users. +# Show/hide only changes the local user's own listing. Each queue gets its own +# hidden (dot-prefixed) .visibility.json in a per-queue folder inside the bundle +# cache — not intended for manual editing. +VISIBILITY_FILENAME = ".visibility.json" +VISIBILITY_VERSION = 1 + +# Max concurrent head_object calls issued when warming the preview prefetch cache. +# This is a background optimization (off the listing critical path), not visibility. +PREVIEW_PREFETCH_MAX_WORKERS = 16 + + +# S3 user-defined metadata is limited to 2 KB total (keys + values, UTF-8 encoded). +# Keys include the "x-amz-meta-" prefix (12 bytes) added by S3. +# We reserve 256 bytes for customer-defined metadata on the same object. +# See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata +METADATA_KEY_NAME = "ojd-name" +METADATA_KEY_DESC = "ojd-desc" +METADATA_KEY_STEPS = "ojd-steps" +METADATA_KEY_PARAMS = "ojd-params" +METADATA_KEY_STEP_COUNT = "ojd-step-count" +METADATA_KEY_PARAM_COUNT = "ojd-param-count" +# Hard caps to prevent any single field from consuming the entire budget +METADATA_LIMIT_NAME = 256 +METADATA_LIMIT_DESC = 600 +S3_METADATA_TOTAL_BUDGET = 2048 - 256 # Reserve 256 bytes for customer metadata + +# Preview display caps. +# +# A job bundle is fully readable once it is local, but a maliciously constructed +# template could declare enormous names/descriptions or huge numbers of steps and +# parameters that would freeze the preview UI or exhaust memory. We always cap +# what the preview surfaces, regardless of the source. The caps are set at (and +# never above) the OpenJD 2023-09 spec's documented maxima so any spec-valid +# bundle previews in full: +# - name / identifier: 512 chars (spec max with the FEATURE_BUNDLE_1 extension) +# - description: 2048 chars +# - parameter string value: 1024 chars +# - parameterDefinitions count: 200 (spec max with FEATURE_BUNDLE_1) +# The OpenJD spec does NOT bound the number of steps, so PREVIEW_MAX_STEPS is our +# own defensive limit for display purposes only. +PREVIEW_MAX_NAME_LEN = 512 +PREVIEW_MAX_DESC_LEN = 2048 +PREVIEW_MAX_PARAM_NAME_LEN = 512 +PREVIEW_MAX_PARAM_VALUE_LEN = 1024 +PREVIEW_MAX_PARAMS = 200 +PREVIEW_MAX_STEPS = 500 + +# Archive extraction safety limits (always enforced, independent of any UI +# warning) to defend against maliciously constructed .ojd archives: +# - too many entries (resource exhaustion), +# - decompression bombs (tiny compressed size expanding to a huge payload). +# Uncompressed sizes come from the zip central directory via infolist(), which +# does not decompress anything; zipfile also caps actual extraction output at the +# declared file_size, so this is a sound upper bound. A generous absolute floor is +# always allowed; above it, the expansion ratio must be plausible for real data +# (a bomb has a tiny compressed size relative to its uncompressed size). There is +# no absolute size ceiling here — the physical limit is free disk space, checked +# separately — so legitimately large (low-ratio) bundles still extract. +MAX_ARCHIVE_ENTRIES = 100_000 +MAX_ARCHIVE_COMPRESSION_RATIO = 200 +MAX_ARCHIVE_UNCOMPRESSED_FLOOR = 256 * 1024 * 1024 # 256 MB always permitted +# A template (even with embedded scripts) is small; refuse to read a giant one. +MAX_TEMPLATE_BYTES = 16 * 1024 * 1024 # 16 MB + +# POSIX only forbids / and null; Windows also forbids \ : * ? " < > | +# Control characters (0x00-0x1F, 0x7F) are problematic on all platforms +_WINDOWS_UNSAFE_CHARS = re.compile(r'[\\/:*?"<>|\x00-\x1f\x7f]+') +_POSIX_UNSAFE_CHARS = re.compile(r"[/\x00-\x1f\x7f]+") + + +def sanitize_bundle_name(name: str) -> str: + """Sanitize a bundle name for use as a local directory name. + + Only replaces characters illegal on the current OS, preserving the + original name as closely as possible. Rejects path traversal attempts. + """ + pattern = _WINDOWS_UNSAFE_CHARS if sys.platform == "win32" else _POSIX_UNSAFE_CHARS + sanitized = pattern.sub("_", name) + # Reject names that are empty or consist solely of replaced unsafe characters + # (e.g. "///" -> "___"), as well as path-traversal components. Underscores that + # result from replacing real content (e.g. a trailing "<") are preserved so the + # sanitized name stays distinct from other bundles. + if ( + not sanitized.strip("_") + or sanitized in (".", "..") + or ".." in re.split(r"[/\\]", sanitized) + ): + raise ValueError("Bundle name is empty or unsafe after sanitization") + return sanitized + + +def _is_archive(name: str) -> bool: + """Check if a filename is an .ojd archive.""" + return name.endswith(ARCHIVE_EXTENSION) + + +def _strip_archive_ext(name: str) -> str: + """Remove the .ojd extension from a filename.""" + if name.endswith(ARCHIVE_EXTENSION): + return name[: -len(ARCHIVE_EXTENSION)] + return name + + +def _safe_zip_extract( + zf: zipfile.ZipFile, dest_dir: str, progress_callback=None, size_callback=None +) -> None: + """Extract a zip file, rejecting archives with entries that would escape dest_dir.""" + dest = os.path.realpath(dest_dir) + + for member in zf.namelist(): + # A zip entry with a leading separator is rooted/absolute. os.path.isabs() + # alone is insufficient: on Windows (Python 3.13+) it returns False for a + # leading-slash path that has no drive letter, so check separators explicitly + # to classify such entries consistently across platforms. + if os.path.isabs(member) or member.startswith(("/", "\\")): + raise ValueError(f"Archive contains absolute path: {member}") + target = os.path.realpath(os.path.join(dest, member)) + try: + common = os.path.commonpath([dest, target]) + except ValueError: + # On Windows, different drives have no common path + raise ValueError(f"Archive entry would extract outside target directory: {member}") + if common != dest: + raise ValueError(f"Archive entry would extract outside target directory: {member}") + + _check_archive_extraction_safety(zf, dest) + + if progress_callback: + if size_callback: + size_callback(sum(info.file_size for info in zf.infolist())) + for info in zf.infolist(): + zf.extract(info, dest_dir) + progress_callback(info.file_size) + else: + zf.extractall(dest_dir) + + +def _check_archive_extraction_safety(zf: zipfile.ZipFile, dest: str) -> None: + """Reject archives that would exhaust resources on extraction. + + Uses the zip central directory (infolist) — no decompression — to check the + entry count and the declared uncompressed size against a bomb-detection ratio, + then verifies the payload fits on the destination filesystem. Raises ValueError + if the archive is unsafe to extract. + """ + infos = zf.infolist() + if len(infos) > MAX_ARCHIVE_ENTRIES: + raise ValueError( + f"Archive has too many entries ({len(infos)} > {MAX_ARCHIVE_ENTRIES}); " + "refusing to extract" + ) + + total_uncompressed = sum(i.file_size for i in infos) + total_compressed = sum(i.compress_size for i in infos) + max_uncompressed = max( + MAX_ARCHIVE_UNCOMPRESSED_FLOOR, + total_compressed * MAX_ARCHIVE_COMPRESSION_RATIO, + ) + if total_uncompressed > max_uncompressed: + raise ValueError( + f"Archive expands to {total_uncompressed} bytes from {total_compressed} " + f"compressed, exceeding the safe limit of {max_uncompressed} bytes " + "(possible zip bomb); refusing to extract" + ) + + # Refuse if the extracted payload wouldn't fit on the target filesystem. + try: + free = shutil.disk_usage(dest if os.path.exists(dest) else os.path.dirname(dest)).free + except OSError: + free = None + if free is not None and total_uncompressed > free: + raise ValueError( + f"Not enough free disk space to extract archive: needs {total_uncompressed} " + f"bytes, {free} available" + ) + + +def _extract_archive(archive_path: str, dest_dir: str) -> None: + """Extract an .ojd archive to dest_dir. + + Raises ValueError with a clear message if the file is not a valid zip archive + (an ``.ojd`` is a zip under the hood), so callers surface a friendly error + instead of a raw ``zipfile.BadZipFile`` for a corrupt/renamed/non-zip file. + """ + try: + with zipfile.ZipFile(archive_path, "r") as zf: + _safe_zip_extract(zf, dest_dir) + except zipfile.BadZipFile as e: + raise ValueError( + f"{os.path.basename(archive_path)!r} is not a valid .ojd archive (expected a zip file)" + ) from e + + +def read_template_from_archive(archive_path: str) -> Optional[tuple[str, str]]: + """Read a template file from a local .ojd archive. Returns (contents, filename) or None.""" + try: + with zipfile.ZipFile(archive_path, "r") as zf: + return _read_template_from_zip(zf) + except Exception: + logger.debug("Failed to read template from archive %s", archive_path, exc_info=True) + return None + + +def _read_template_from_zip(zf: zipfile.ZipFile) -> Optional[tuple[str, str]]: + """Read a template file from an open ZipFile. Returns (contents, filename) or None.""" + names = zf.namelist() + for fname in TEMPLATE_FILENAMES: + matches = [n for n in names if n == fname or n.endswith("/" + fname)] + matches.sort(key=lambda n: n.count("/")) + if matches: + # Refuse to read an implausibly large template (bomb defense). The + # declared size bounds the actual read: zipfile caps output at + # file_size, so this check on the central-directory value is sound. + info = zf.getinfo(matches[0]) + if info.file_size > MAX_TEMPLATE_BYTES: + raise ValueError( + f"Template '{matches[0]}' is too large to read " + f"({info.file_size} > {MAX_TEMPLATE_BYTES} bytes)" + ) + return zf.read(matches[0]).decode("utf-8"), fname + return None + + +def _extract_archive_from_fileobj( + fileobj, dest_dir: str, progress_callback=None, size_callback=None +) -> None: + """Extract an .ojd archive from a seekable binary file object to dest_dir. + + Operating on a file object (rather than a bytes blob) lets callers stream a + large download into a temp file and extract from it without ever holding the + whole compressed archive in memory. + + Raises ValueError with a clear message if the object is not a valid zip + archive, so a corrupt or non-`.ojd` download surfaces a friendly error rather + than a raw ``zipfile.BadZipFile``. + """ + fileobj.seek(0) + try: + with zipfile.ZipFile(fileobj, "r") as zf: + _safe_zip_extract( + zf, dest_dir, progress_callback=progress_callback, size_callback=size_callback + ) + except zipfile.BadZipFile as e: + raise ValueError( + "Downloaded object is not a valid .ojd archive (expected a zip file)" + ) from e + + +# Downloads at or below this size are buffered in memory; larger archives spill +# to a temp file so we never hold a multi-GB archive (in several copies) in RAM. +_DOWNLOAD_SPOOL_THRESHOLD = 64 * 1024 * 1024 # 64 MB + + +def _open_download_sink(size_hint: int): + """Return a seekable binary sink for a download of approximately size_hint bytes. + + Small downloads stay in memory (fast); larger ones spill to a temporary file + created inside the bundle cache directory. Keeping the temp file in the cache + dir — rather than the shared system temp dir — co-locates the spill with the + extraction target's filesystem and keeps it out of world-writable /tmp. The + temp file is unlinked on close (immediately, on POSIX), so no other process + can reference or tamper with it. + + ``SpooledTemporaryFile`` is intentionally avoided: before Python 3.11 it does + not implement ``seekable()``, which ``zipfile`` requires. ``io.BytesIO`` and + ``tempfile.TemporaryFile`` are fully seekable on all supported versions. + """ + if size_hint and size_hint > _DOWNLOAD_SPOOL_THRESHOLD: + cache_root = get_bundle_cache_dir() + os.makedirs(cache_root, exist_ok=True) + return tempfile.TemporaryFile(dir=cache_root) + return io.BytesIO() + + +_LARGE_FILE_THRESHOLD = 8 * 1024 * 1024 # 8MB +_CHUNK_SIZE = 4 * 1024 * 1024 # 4MB + + +def archive_bundle_dir(source_dir: str, progress_callback=None) -> io.BytesIO: + """Archive a job bundle directory into an in-memory .ojd (zip) buffer. + + Args: + source_dir: Path to the bundle directory to archive. + progress_callback: Optional callable(bytes_archived) called after each file/chunk. + + Returns: + A BytesIO buffer positioned at the start, containing the zip archive. + """ + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED, allowZip64=True) as zf: + for root, dirs, files in os.walk(source_dir, followlinks=False): + dirs[:] = [d for d in dirs if not os.path.islink(os.path.join(root, d))] + for fname in files: + local_path = os.path.join(root, fname) + if os.path.islink(local_path): + logger.warning("Skipping symlink: %s", local_path) + continue + arcname = os.path.relpath(local_path, source_dir) + fsize = os.path.getsize(local_path) + if fsize <= _LARGE_FILE_THRESHOLD: + zf.write(local_path, arcname) + if progress_callback: + progress_callback(fsize) + else: + with ( + zf.open(arcname.replace(os.sep, "/"), "w", force_zip64=True) as dest, + open(local_path, "rb") as src, + ): + while True: + chunk = src.read(_CHUNK_SIZE) + if not chunk: + break + dest.write(chunk) + if progress_callback: + progress_callback(len(chunk)) + buf.seek(0) + return buf + + +def get_bundle_dir_size(source_dir: str) -> int: + """Calculate total size of archivable files in a bundle directory.""" + total = 0 + for root, dirs, files in os.walk(source_dir, followlinks=False): + dirs[:] = [d for d in dirs if not os.path.islink(os.path.join(root, d))] + for fname in files: + fpath = os.path.join(root, fname) + if not os.path.islink(fpath): + total += os.path.getsize(fpath) + return total + + +def build_bundle_metadata( + source_dir: Optional[str] = None, + bundle_name: Optional[str] = None, + bundle_info: Optional["BundleInfo"] = None, +) -> dict[str, str]: + """Build S3 user metadata dict for a bundle. + + Extracts name, description, steps, and parameters from the template, + fitting them within the S3 2KB metadata budget. Returns an empty dict + if no template is found. + + Provide either source_dir (to auto-extract info) or bundle_info (pre-extracted). + + Args: + source_dir: Path to the bundle directory. + bundle_name: Override for the metadata name field (defaults to template name). + bundle_info: Pre-extracted BundleInfo (skips template parsing if provided). + """ + metadata: dict[str, str] = {} + + if bundle_info: + info = bundle_info + elif source_dir: + template = None + for tname in TEMPLATE_FILENAMES: + tpath = os.path.join(source_dir, tname) + if os.path.isfile(tpath): + try: + with open(tpath, encoding="utf-8") as f: + template = _parse_template(f.read(), tname) + if template: + break + except OSError: + pass # Unreadable template file — try next candidate + if not template: + return metadata + pv = LocalBundleRepository.read_parameter_values(source_dir) + info = extract_bundle_info(template, source_dir, pv) + else: + return metadata + + name_value = bundle_name or info.name + metadata[METADATA_KEY_NAME] = _truncate_s3_value( + name_value, METADATA_LIMIT_NAME, METADATA_KEY_NAME + ) + if info.description: + desc = " ".join(info.description.split()) + metadata[METADATA_KEY_DESC] = _truncate_s3_value( + desc, METADATA_LIMIT_DESC, METADATA_KEY_DESC + ) + if info.step_names: + metadata[METADATA_KEY_STEP_COUNT] = str(len(info.step_names)) + if info.parameters: + metadata[METADATA_KEY_PARAM_COUNT] = str(len(info.parameters)) + + # Dynamically allocate remaining budget to steps and params + steps_str = ",".join(info.step_names) if info.step_names else "" + param_strs = ( + ",".join(f"{p.get('name', '?')}:{p.get('type', '?')}" for p in info.parameters) + if info.parameters + else "" + ) + + used = sum(12 + len(k.encode("utf-8")) + len(v.encode("utf-8")) for k, v in metadata.items()) + remaining = S3_METADATA_TOTAL_BUDGET - used + keys_needed = 0 + if steps_str: + keys_needed += 12 + len(METADATA_KEY_STEPS) + if param_strs: + keys_needed += 12 + len(METADATA_KEY_PARAMS) + remaining -= keys_needed + + if remaining > 0: + if steps_str and param_strs: + steps_budget = remaining // 2 + params_budget = remaining - steps_budget + metadata[METADATA_KEY_STEPS] = _truncate_s3_value( + steps_str, steps_budget, METADATA_KEY_STEPS + ) + metadata[METADATA_KEY_PARAMS] = _truncate_s3_value( + param_strs, params_budget, METADATA_KEY_PARAMS + ) + elif steps_str: + metadata[METADATA_KEY_STEPS] = _truncate_s3_value( + steps_str, remaining, METADATA_KEY_STEPS + ) + elif param_strs: + metadata[METADATA_KEY_PARAMS] = _truncate_s3_value( + param_strs, remaining, METADATA_KEY_PARAMS + ) + + return metadata + + +# RFC 2047 base64 encoded-word for UTF-8: "=?utf-8?b?" + base64 + "?=". +# The wrapper is a constant 12 characters and base64 expands N bytes to exactly +# 4*ceil(N/3) characters, so the encoded length is fully predictable — see +# _truncate_s3_value. +_S3_ENCODED_WORD_PREFIX = "=?utf-8?b?" +_S3_ENCODED_WORD_SUFFIX = "?=" +_S3_ENCODED_WORD_OVERHEAD = len(_S3_ENCODED_WORD_PREFIX) + len(_S3_ENCODED_WORD_SUFFIX) # 12 + + +def _encode_s3_value(value: str) -> str: + """Encode a metadata value so it is safe to send as S3 user metadata. + + S3 user metadata is transmitted as ``x-amz-meta-*`` HTTP headers, and botocore + rejects any non-US-ASCII value with a ``ParamValidationError`` ("S3 metadata + can only contain ASCII characters"). ASCII values are returned unchanged, so + existing uploads and readers are unaffected. Values containing non-ASCII + characters (e.g. Japanese, accented Latin, emoji) are encoded as an RFC 2047 + base64 encoded-word so the original text can be recovered on read via + ``_decode_s3_value``. + + The encoded-word is built explicitly (rather than via ``email.header``) so the + encoded length is exactly ``12 + 4*ceil(N/3)`` for ``N`` UTF-8 bytes — this + guarantee lets ``_truncate_s3_value`` size a truncation in one shot without a + retry loop, and produces a single, unfolded word. + + The authoritative values still live inside the bundle template; this metadata + is only a zero-download preview hint. + """ + if value.isascii(): + return value + body = base64.b64encode(value.encode("utf-8")).decode("ascii") + return f"{_S3_ENCODED_WORD_PREFIX}{body}{_S3_ENCODED_WORD_SUFFIX}" + + +def _decode_s3_value(value: str) -> str: + """Inverse of ``_encode_s3_value``: decode RFC 2047 encoded-words if present. + + Decoding is intentionally permissive and uses the stdlib ``decode_header`` so + it handles *any* valid encoded-word, not just this module's exact output: + upper/lowercase charset (``UTF-8``/``utf-8``), both Base64 (``B``) and + quoted-printable (``Q``) encodings, and values folded into multiple + space-separated encoded-words. This makes previews robust to bundles uploaded + by other clients or earlier builds. Plain ASCII values (no encoded-word + marker) are returned unchanged, and a trailing truncation marker ("...") is + preserved. + """ + if "=?" not in value: + return value + try: + return "".join( + fragment.decode(charset or "utf-8") if isinstance(fragment, bytes) else fragment + for fragment, charset in decode_header(value) + ) + except Exception: + # Never let a malformed value break preview — fall back to the raw text. + return value + + +def _truncate_to_utf8_bytes(value: str, max_bytes: int) -> str: + """Truncate a string to at most ``max_bytes`` UTF-8 bytes, on a char boundary.""" + encoded = value.encode("utf-8") + if len(encoded) <= max_bytes: + return value + return encoded[:max_bytes].decode("utf-8", errors="ignore") + + +def _truncate_s3_value(value: str, limit: int, field: str = "") -> str: + """Encode to an S3-safe (US-ASCII) form and fit it within a byte ``limit``. + + The byte limit applies to the *encoded* value (that is what is sent to S3 and + counts against the metadata budget). Because a base64 encoded-word cannot be + sliced without corrupting it, we instead size how much *raw* text to keep: + + * ASCII values are stored verbatim, so we slice to ``limit - 3`` (reserving 3 + bytes for the "..." marker). + * Non-ASCII values become a base64 encoded-word of length + ``12 + 4*ceil(N/3)`` for ``N`` UTF-8 bytes. Solving ``12 + 4*ceil(N/3) + 3 + <= limit`` gives ``N <= 3 * ((limit - 15) // 4)`` — an exact bound, so we + truncate the raw value to that many UTF-8 bytes and encode once. + """ + encoded = _encode_s3_value(value) + if len(encoded) <= limit: # encoded form is ASCII: one byte per character + return encoded + if limit <= 3: + return "" + + if value.isascii(): + truncated_raw = value[: limit - 3] + else: + # 12 bytes wrapper + 3 bytes for "..." = 15 bytes of fixed overhead. + max_groups = (limit - _S3_ENCODED_WORD_OVERHEAD - 3) // 4 + truncated_raw = _truncate_to_utf8_bytes(value, max_groups * 3) if max_groups > 0 else "" + + if field: + logger.warning( + "Bundle metadata '%s' truncated from %d to %d bytes", + field, + len(encoded), + limit, + ) + return _encode_s3_value(truncated_raw) + "..." + + +@dataclass +class BundleInfo: + """Metadata extracted from a job bundle's template.""" + + path: str + name: str + description: str = "" + step_names: list[str] = field(default_factory=list) + parameters: list[dict] = field(default_factory=list) + total_steps: Optional[int] = None # Actual count (when metadata was truncated) + total_parameters: Optional[int] = None # Actual count (when metadata was truncated) + size_bytes: Optional[int] = None # Archive/download size in bytes, when known + + def to_dict(self) -> dict: + """Serialize to a dict suitable for JSON output.""" + return { + "name": self.name, + "description": self.description, + "steps": self.step_names, + "parameters": self.parameters, + "sizeBytes": self.size_bytes, + } + + def format_text(self) -> str: + """Format as human-readable text.""" + lines = [f"Name: {self.name}"] + if self.description: + lines.append(f"Description: {self.description}") + if self.step_names: + lines.append("Steps:") + for step in self.step_names: + lines.append(f" \u2022 {step}") + if self.parameters: + lines.append("Parameters:") + for p in self.parameters: + name = p.get("name", "?") + ptype = p.get("type", "?") + value = p.get("_display_value", "") + line = f" {name} ({ptype})" + if value: + line += f" = {value}" + lines.append(line) + return "\n".join(lines) + + +@dataclass +class BrowseEntry: + """A single item in the browser listing.""" + + name: str + path: str + is_bundle: bool + is_archive: bool = False + + +class BundleRepository(Protocol): + def list_entries(self, path: str) -> list[BrowseEntry]: + """List immediate children of `path`. Returns folders and bundles.""" + ... + + def get_bundle_info(self, path: str) -> Optional[BundleInfo]: + """Load and return metadata for the bundle at `path`, or None if invalid.""" + ... + + def root_path(self) -> str: + """The starting path for browsing.""" + ... + + +def _parse_template(raw: str, filename: str) -> Optional[dict]: + """Parse a template file's contents, returning the dict or None on failure.""" + try: + if filename.endswith(".json"): + return json.loads(raw) + else: + return yaml.safe_load(raw) + except Exception: + logger.debug("Failed to parse template %s", filename, exc_info=True) + return None + + +def extract_bundle_info( + template: dict, path: str, parameter_values: Optional[dict] = None +) -> BundleInfo: + """Extract BundleInfo from a parsed template dict. + + If parameter_values is provided, merges values into the parameter definitions. + + All fields are capped to the preview limits (see PREVIEW_MAX_* constants) and + every access is defensive: the template comes from a job bundle that may have + been authored by another user (e.g. a shared queue bundle), so it could be + malformed or maliciously oversized. We never let it crash or DoS the preview. + """ + raw_params = template.get("parameterDefinitions", []) + if not isinstance(raw_params, list): + raw_params = [] + + # Build a lookup from parameter_values file + pv_map: dict[str, str] = {} + if isinstance(parameter_values, dict): + for pv in parameter_values.get("parameterValues", []) or []: + if isinstance(pv, dict) and "name" in pv and "value" in pv: + pv_map[pv["name"]] = pv["value"] + + # Cap the number of parameters; record the true total so the preview can show + # "… N more" via the existing truncation UI. + total_parameters = len(raw_params) if len(raw_params) > PREVIEW_MAX_PARAMS else None + params: list[dict] = [] + for p in raw_params[:PREVIEW_MAX_PARAMS]: + if not isinstance(p, dict): + continue + capped = dict(p) # copy so we never mutate the caller's template + name = capped.get("name", "") + capped["name"] = str(name)[:PREVIEW_MAX_PARAM_NAME_LEN] if name else "" + if capped.get("type"): + capped["type"] = str(capped["type"])[:PREVIEW_MAX_PARAM_NAME_LEN] + # Attach resolved value: parameter_values > default > (unset) + if name in pv_map: + capped["_display_value"] = str(pv_map[name])[:PREVIEW_MAX_PARAM_VALUE_LEN] + elif "default" in capped: + capped["_display_value"] = str(capped["default"])[:PREVIEW_MAX_PARAM_VALUE_LEN] + params.append(capped) + + raw_steps = template.get("steps", []) + if not isinstance(raw_steps, list): + raw_steps = [] + total_steps = len(raw_steps) if len(raw_steps) > PREVIEW_MAX_STEPS else None + step_names = [ + str(s.get("name", ""))[:PREVIEW_MAX_NAME_LEN] + for s in raw_steps[:PREVIEW_MAX_STEPS] + if isinstance(s, dict) + ] + + raw_name = template.get("name") + if not isinstance(raw_name, str) or not raw_name: + raw_name = os.path.basename(path.rstrip("/")) + name = raw_name[:PREVIEW_MAX_NAME_LEN] + + raw_desc = template.get("description", "") + description = raw_desc[:PREVIEW_MAX_DESC_LEN] if isinstance(raw_desc, str) else "" + + return BundleInfo( + path=path, + name=name, + description=description, + step_names=step_names, + parameters=params, + total_steps=total_steps, + total_parameters=total_parameters, + ) + + +class LocalBundleRepository: + """Browse job bundles on the local filesystem. Supports directories and archives.""" + + def __init__(self, root: str = "", include_archives: bool = True): + self._root = root or os.path.expanduser("~") + self._include_archives = include_archives + + def root_path(self) -> str: + return self._root + + def list_entries(self, path: str) -> list[BrowseEntry]: + entries: list[BrowseEntry] = [] + try: + with os.scandir(path) as it: + children = sorted(it, key=lambda e: e.name) + except OSError: + return entries + for entry in children: + if entry.is_dir(follow_symlinks=False): + is_bundle = self._is_dir_bundle(entry.path) + entries.append(BrowseEntry(name=entry.name, path=entry.path, is_bundle=is_bundle)) + elif ( + self._include_archives + and entry.is_file(follow_symlinks=False) + and _is_archive(entry.name) + ): + # Only show archives that actually contain a template + if read_template_from_archive(entry.path) is not None: + entries.append( + BrowseEntry( + name=_strip_archive_ext(entry.name), + path=entry.path, + is_bundle=True, + is_archive=True, + ) + ) + return entries + + def get_bundle_info(self, path: str) -> Optional[BundleInfo]: + if os.path.isfile(path) and _is_archive(path): + return self._get_archive_bundle_info(path) + return self._get_dir_bundle_info(path) + + def extract_bundle(self, path: str, dest_dir: str) -> str: + """Extract an archive bundle, using mtime-based cache to avoid redundant extraction. + + Returns path to the extracted bundle directory.""" + cache_dir = os.path.join(get_bundle_cache_dir(), _local_cache_key(path)) + meta = _read_cache_meta(cache_dir) + current_mtime = os.path.getmtime(path) + + # Cache hit — mtime unchanged + if meta and meta.get("mtime") == current_mtime and os.path.isdir(cache_dir): + bundle_dir = self._find_bundle_root(cache_dir) + if bundle_dir: + logger.info("Using cached bundle: %s", cache_dir) + return bundle_dir + + # Cache miss or stale — extract + if os.path.exists(cache_dir): + shutil.rmtree(cache_dir) + os.makedirs(cache_dir, exist_ok=True) + _extract_archive(path, cache_dir) + + # Write mtime to cache meta + meta_path = os.path.join(cache_dir, CACHE_META_FILENAME) + with open(meta_path, "w", encoding="utf-8") as f: + json.dump({"mtime": current_mtime}, f) + + return self._find_bundle_root(cache_dir) or cache_dir + + @staticmethod + def _find_bundle_root(extract_dir: str) -> Optional[str]: + """If the archive has a single top-level wrapper dir, return it; otherwise extract_dir.""" + contents = [c for c in os.listdir(extract_dir) if c != CACHE_META_FILENAME] + if len(contents) == 1 and os.path.isdir(os.path.join(extract_dir, contents[0])): + return os.path.join(extract_dir, contents[0]) + return extract_dir + + def _get_dir_bundle_info(self, path: str) -> Optional[BundleInfo]: + for fname in TEMPLATE_FILENAMES: + fpath = os.path.join(path, fname) + if os.path.isfile(fpath): + try: + with open(fpath, encoding="utf-8") as f: + raw = f.read() + except OSError: + return None + template = _parse_template(raw, fname) + if template: + pv = self.read_parameter_values(path) + return extract_bundle_info(template, path, pv) + return None + + @staticmethod + def read_parameter_values(path: str) -> Optional[dict]: + """Read parameter_values.yaml or .json from a bundle directory.""" + for pvname in ("parameter_values.yaml", "parameter_values.json"): + pvpath = os.path.join(path, pvname) + if os.path.isfile(pvpath): + try: + with open(pvpath, encoding="utf-8") as f: + return _parse_template(f.read(), pvname) + except OSError: + pass + return None + + def _get_archive_bundle_info(self, path: str) -> Optional[BundleInfo]: + result = read_template_from_archive(path) + if result: + raw, fname = result + template = _parse_template(raw, fname) + if template: + return extract_bundle_info(template, path) + return None + + @staticmethod + def _is_dir_bundle(path: str) -> bool: + for fname in TEMPLATE_FILENAMES: + if os.path.isfile(os.path.join(path, fname)): + return True + return False + + +# ── S3 Cache ───────────────────────────────────────────────── + + +def get_bundle_cache_dir() -> str: + """Get the root cache directory for S3 bundle archives.""" + return os.path.join(get_cache_directory(), "job-bundles") + + +def _cache_key(bucket: str, s3_key: str) -> str: + """Deterministic cache subdirectory from bucket + key.""" + h = hashlib.sha256(f"{bucket}/{s3_key}".encode()).hexdigest()[:16] + name = _strip_archive_ext(s3_key.rstrip("/").rsplit("/", 1)[-1]) + return os.path.join(h, name) + + +def _local_cache_key(path: str) -> str: + """Deterministic cache subdirectory from a local file path.""" + h = hashlib.sha256(os.path.abspath(path).encode()).hexdigest()[:16] + name = _strip_archive_ext(os.path.basename(path)) + return os.path.join(h, name) + + +def _read_cache_meta(cache_dir: str) -> Optional[dict]: + meta_path = os.path.join(cache_dir, CACHE_META_FILENAME) + if os.path.isfile(meta_path): + try: + with open(meta_path, encoding="utf-8") as f: + return json.load(f) + except Exception: + pass # Corrupt or unreadable cache meta — treat as cache miss + return None + + +def _normalize_etag(etag: Optional[str]) -> str: + """Strip surrounding quotes from an ETag for consistent comparison.""" + if not etag: + return "" + return etag.strip('"') + + +def _write_cache_meta(cache_dir: str, etag: str, last_modified: str) -> None: + meta_path = os.path.join(cache_dir, CACHE_META_FILENAME) + with open(meta_path, "w", encoding="utf-8") as f: + json.dump({"etag": etag, "last_modified": last_modified}, f) + + +# ── S3 Repository ──────────────────────────────────────────── + + +def _safe_int(value: Optional[str]) -> Optional[int]: + """Parse a string as int, returning None on failure or empty/None input.""" + if not value: + return None + try: + return int(value) + except ValueError: + return None + + +def _bundle_info_from_s3_metadata(metadata: dict, path: str) -> Optional[BundleInfo]: + """Try to construct BundleInfo from S3 user metadata set during upload. + Returns None if the required 'ojd-name' key is missing. + + For bundles uploaded outside this tool the metadata is attacker-influenced, + so every value is capped to the same PREVIEW_MAX_* limits used for templates. + S3 already bounds total user metadata to ~2 KB, but we defend in depth and + stay consistent with the template preview path so a crafted object can't + bloat the preview. + """ + name = metadata.get(METADATA_KEY_NAME) + if not name: + return None + name = _decode_s3_value(name)[:PREVIEW_MAX_NAME_LEN] + params = [] + params_str = _decode_s3_value(metadata.get(METADATA_KEY_PARAMS, "")) + if params_str: + for p in params_str.split(",")[:PREVIEW_MAX_PARAMS]: + parts = p.split(":", 1) + if len(parts) == 2: + params.append( + { + "name": parts[0][:PREVIEW_MAX_PARAM_NAME_LEN], + "type": parts[1][:PREVIEW_MAX_PARAM_NAME_LEN], + "_from_metadata": True, + } + ) + + step_count_str = metadata.get(METADATA_KEY_STEP_COUNT) + param_count_str = metadata.get(METADATA_KEY_PARAM_COUNT) + + step_names = [ + s[:PREVIEW_MAX_NAME_LEN] + for s in _decode_s3_value(metadata.get(METADATA_KEY_STEPS, "")).split(",") + if s + ][:PREVIEW_MAX_STEPS] + + return BundleInfo( + path=path, + name=name, + description=_decode_s3_value(metadata.get(METADATA_KEY_DESC, ""))[:PREVIEW_MAX_DESC_LEN], + step_names=step_names, + parameters=params, + total_steps=_safe_int(step_count_str), + total_parameters=_safe_int(param_count_str), + ) + + +def get_bundle_queue_cache_dir(bucket: str, prefix: str) -> str: + """Per-queue folder inside the bundle cache (keyed by bucket + prefix). + + Holds this queue's local, per-user data such as ``.visibility.json``. + """ + key = hashlib.sha256(f"{bucket}/{prefix.rstrip('/')}".encode()).hexdigest()[:16] + return os.path.join(get_bundle_cache_dir(), key) + + +class _LocalBundleVisibility: + """Per-user "hidden" view for a queue's shared bundles, stored locally. + + Hiding a bundle is a private view preference written to a ``.visibility.json`` + in this queue's cache folder. It never touches S3, so showing or hiding a + bundle only changes this user's own listing, not anyone else's. + """ + + def __init__(self, bucket: str, prefix: str): + self._bucket = bucket + self._prefix = prefix + + def _view_path(self) -> str: + return os.path.join( + get_bundle_queue_cache_dir(self._bucket, self._prefix), VISIBILITY_FILENAME + ) + + def get_hidden_set(self) -> set[str]: + """Read this user's hidden bundle names for the queue (empty if none).""" + try: + with open(self._view_path(), encoding="utf-8") as f: + data = json.load(f) + return set(data.get("hidden", [])) + except (OSError, ValueError): + # No file yet, unreadable, or malformed — treat as nothing hidden. + return set() + + def set_bundle_visibility(self, bundle_name: str, *, hidden: bool) -> None: + """Hide or unhide a bundle in this user's local view (no S3 calls).""" + hidden_set = self.get_hidden_set() + if hidden: + if bundle_name in hidden_set: + return # Already hidden + hidden_set.add(bundle_name) + else: + if bundle_name not in hidden_set: + return # Already visible + hidden_set.discard(bundle_name) + + path = self._view_path() + os.makedirs(os.path.dirname(path), exist_ok=True) + body = json.dumps({"version": VISIBILITY_VERSION, "hidden": sorted(hidden_set)}, indent=2) + # Atomic write so a crash can't corrupt the view file. + tmp = f"{path}.tmp" + with open(tmp, "w", encoding="utf-8") as f: + f.write(body) + os.replace(tmp, path) + + +def _make_s3_client(session): + """Create an S3 client for a bundle repository. + + Sizes the connection pool to cover the parallel ``head_object`` calls the + background preview prefetch issues (and the managed upload/download + transfers) so urllib3 does not log "Connection pool is full". + """ + from ..api._session import get_default_client_config + + try: + configured = int(config_file.get_setting("settings.s3_max_pool_connections")) + except (ValueError, TypeError): + configured = 0 + max_pool = max(configured, PREVIEW_PREFETCH_MAX_WORKERS) + return session.client("s3", config=get_default_client_config(max_pool_connections=max_pool)) + + +class S3BundleRepository: + """Browse .ojd job bundles in an S3 bucket under {rootPrefix}/job-bundles/. + Only .ojd archives are supported. Subfolders are shown for navigation only. + Archive bundles are cached locally with ETag validation.""" + + def __init__(self, bucket_name: str, root_prefix: str, session=None): + import boto3 as _boto3 + + self._bucket = bucket_name + base = root_prefix.rstrip("/") + self._prefix = f"{base}/{S3_JOB_BUNDLES_PREFIX}/" + self._session = session or _boto3.Session() + self._s3 = _make_s3_client(self._session) + self._last_head: Optional[tuple[str, dict]] = None + # Preview prefetch cache of head_object responses keyed by S3 key. Warmed + # in the background by prefetch_previews() after the listing is shown, and + # consulted by the preview/size/download paths to avoid a second HEAD. + self._head_cache: dict[str, dict] = {} + + @classmethod + def from_config(cls, config=None) -> "S3BundleRepository": + """Create an S3BundleRepository from the user's Deadline Cloud configuration. + + Handles session creation, queue lookup, and attachment settings extraction. + Uses queue role credentials for S3 access (required for DCM profiles). + Raises DeadlineOperationError if farm/queue is not configured or has no attachments. + """ + from concurrent.futures import ThreadPoolExecutor, Future + + from ..api import get_boto3_client, get_boto3_session, get_queue_user_boto3_session + + farm_id = config_file.get_setting("defaults.farm_id", config=config) + queue_id = config_file.get_setting("defaults.queue_id", config=config) + if not farm_id or not queue_id: + raise DeadlineOperationError("A default farm and queue must be configured.") + + # Ensure session is cached (used internally by get_boto3_client and get_queue_user_boto3_session) + get_boto3_session(config=config) + + # Create one Deadline client — reused for both get_queue and AssumeQueueRole + deadline_client = get_boto3_client("deadline", config=config) + + # Set up queue role session (just wires up credential provider, no API call) + s3_session = get_queue_user_boto3_session( + deadline=deadline_client, + config=config, + farm_id=farm_id, + queue_id=queue_id, + queue_display_name=None, + ) + + # Run get_queue API call and S3 client creation in parallel. + # get_queue gives us the bucket name; S3 client creation triggers AssumeRole. + def _call_get_queue(): + return deadline_client.get_queue(farmId=farm_id, queueId=queue_id) + + with ThreadPoolExecutor(max_workers=2) as executor: + queue_future: Future = executor.submit(_call_get_queue) + s3_client_future: Future = executor.submit(_make_s3_client, s3_session) + + s3_client = s3_client_future.result() + queue_response = queue_future.result() + + # Extract job attachment settings + ja_settings = queue_response.get("jobAttachmentSettings") + if not ja_settings or not ja_settings.get("s3BucketName"): + raise DeadlineOperationError( + f"Queue {queue_id} does not have job attachment settings configured." + ) + + repo = cls.__new__(cls) + base = ja_settings["rootPrefix"].rstrip("/") + repo._bucket = ja_settings["s3BucketName"] + repo._prefix = f"{base}/{S3_JOB_BUNDLES_PREFIX}/" + repo._session = s3_session + repo._s3 = s3_client + repo._last_head = None + repo._head_cache = {} + return repo + + def root_path(self) -> str: + return f"s3://{self._bucket}/{self._prefix}" + + def list_entries(self, path: str) -> list[BrowseEntry]: + prefix = self._to_s3_prefix(path) + entries: list[BrowseEntry] = [] + child_prefixes: list[tuple[str, str, str]] = [] # (name, child_prefix, child_path) + try: + paginator = self._s3.get_paginator("list_objects_v2") + for page in paginator.paginate(Bucket=self._bucket, Prefix=prefix, Delimiter="/"): + # Subfolders (for navigation only, not bundles) + for cp in page.get("CommonPrefixes", []): + child_prefix = cp["Prefix"] + name = child_prefix.rstrip("/").rsplit("/", 1)[-1] + child_path = f"s3://{self._bucket}/{child_prefix}" + child_prefixes.append((name, child_prefix, child_path)) + # .ojd archive bundles + for obj in page.get("Contents", []): + key = obj["Key"] + name = key.rsplit("/", 1)[-1] if "/" in key else key + if _is_archive(name): + s3_path = f"s3://{self._bucket}/{key}" + entries.append( + BrowseEntry( + name=_strip_archive_ext(name), + path=s3_path, + is_bundle=True, + is_archive=True, + ) + ) + except Exception: + logger.warning("Failed to list S3 prefix %s", prefix, exc_info=True) + raise + + # Subfolders are shown for navigation but never as bundles + for name, child_prefix, child_path in child_prefixes: + entries.append(BrowseEntry(name=name, path=child_path, is_bundle=False)) + + entries.sort(key=lambda e: e.name.lower()) + return entries + + def get_bundle_info(self, path: str) -> Optional[BundleInfo]: + return self._get_archive_bundle_info(path) + + def download_full_bundle( + self, + path: str, + dest_dir: str, + progress_callback=None, + extract_callback=None, + extract_size_callback=None, + ) -> str: + """Download a complete S3 .ojd bundle to a local directory. + Uses the ETag cache for repeated access.""" + return self._resolve_archive_bundle( + path, + progress_callback=progress_callback, + extract_callback=extract_callback, + extract_size_callback=extract_size_callback, + ) + + def prefetch_previews(self) -> None: + """Warm the preview cache by issuing ``head_object`` for every ``.ojd`` + object under the prefix, in parallel. + + Each response (ETag + user metadata + ContentLength) is stashed in + ``_head_cache`` so the preview/size/download paths can reuse it instead of + issuing their own HEAD — making previews effectively instant once warmed. + + This is a background optimization and is safe to call off the UI thread + after the listing is displayed; it never blocks the initial listing. + Individual HEAD failures are ignored. The cache is rebuilt from scratch, + dropping entries for bundles that no longer exist. + """ + keys = self._list_all_bundle_keys() + self._head_cache.clear() + if not keys: + return + + def _head(key: str) -> None: + try: + # dict setitem is atomic under the GIL, so this is safe to do + # from the worker threads while the UI reads the cache. + self._head_cache[key] = self._s3.head_object(Bucket=self._bucket, Key=key) + except Exception: + logger.debug("prefetch head_object failed for %s", key, exc_info=True) + + max_workers = min(PREVIEW_PREFETCH_MAX_WORKERS, len(keys)) + with ThreadPoolExecutor(max_workers=max_workers) as ex: + list(ex.map(_head, keys)) + + def _list_all_bundle_keys(self) -> list[str]: + """List every ``.ojd`` object key under the prefix, recursively.""" + keys: list[str] = [] + paginator = self._s3.get_paginator("list_objects_v2") + for page in paginator.paginate(Bucket=self._bucket, Prefix=self._prefix): + for obj in page.get("Contents", []): + key = obj["Key"] + if key.endswith(ARCHIVE_EXTENSION): + keys.append(key) + return keys + + def _head_object(self, key: str) -> dict: + """Return the prefetched head for ``key`` if the cache warmed it, else + issue a ``head_object``. Raises on a genuine HEAD failure — callers that + tolerate failure wrap this in try/except.""" + cached = self._head_cache.get(key) + if cached is not None: + return cached + return self._s3.head_object(Bucket=self._bucket, Key=key) + + def get_bundle_size(self, path: str) -> int: + """Get the size in bytes of a bundle archive on S3. + Caches the result so a subsequent download_full_bundle doesn't repeat the call.""" + key = self._to_s3_key(path) + head = self._head_object(key) + self._last_head = (key, head) + return head.get("ContentLength", 0) + + def bundle_exists(self, bundle_name: str) -> bool: + """Check if a bundle with the given name exists on S3.""" + key = f"{self._prefix}{bundle_name}.ojd" + try: + self._s3.head_object(Bucket=self._bucket, Key=key) + return True + except Exception: + return False + + def upload_archive( + self, + buf: io.BytesIO, + bundle_name: str, + metadata: Optional[dict[str, str]] = None, + progress_callback=None, + ) -> str: + """Upload an in-memory archive buffer to S3 as an .ojd bundle. + + Returns the S3 URI of the uploaded bundle. + """ + key = f"{self._prefix}{bundle_name}.ojd" + # Advertise the archive's true type. This is a courtesy hint for other + # consumers/tools; the download path deliberately does NOT trust it (an + # object's ContentType is set by whoever uploaded it), and validates by + # actually parsing the zip. See _extract_archive_from_fileobj. + extra_args: dict = {"ContentType": "application/zip"} + if metadata: + extra_args["Metadata"] = metadata + kwargs: dict = { + "Bucket": self._bucket, + "Key": key, + "ExtraArgs": extra_args, + } + if progress_callback: + kwargs["Callback"] = progress_callback + self._s3.upload_fileobj(buf, **kwargs) + return f"s3://{self._bucket}/{key}" + + def clear_cache_for(self, path: str) -> None: + """Remove cached data for a specific bundle path.""" + key = self._to_s3_key(path) + cache_dir = os.path.join(get_bundle_cache_dir(), _cache_key(self._bucket, key)) + if os.path.exists(cache_dir): + shutil.rmtree(cache_dir, ignore_errors=True) + + # ── Archive bundles ────────────────────────────────────── + + def _get_archive_bundle_info(self, path: str) -> Optional[BundleInfo]: + key = self._to_s3_key(path) + cache_dir = os.path.join(get_bundle_cache_dir(), _cache_key(self._bucket, key)) + meta = _read_cache_meta(cache_dir) + + # Always do a head_object first — it's cheap and gives us both + # ETag (for cache validation) and user metadata (for preview without download). + # Reuse the background prefetch cache if it already warmed this object. + head = None + try: + head = self._head_object(key) + except Exception: + pass # head_object failure is non-fatal; we fall through to download + + # The archive/download size, reused for the preview (shown on the Download + # button) without an extra call. May be refined from the GET below if the + # HEAD failed. + content_length = head.get("ContentLength") if head else None + + def _stamp(info: Optional[BundleInfo]) -> Optional[BundleInfo]: + if info is not None and content_length is not None: + info.size_bytes = content_length + return info + + if head: + # Check local cache validity + cache_valid = meta and _normalize_etag(head.get("ETag")) == _normalize_etag( + meta.get("etag") + ) + + # Prefer cached template (source of truth) over S3 metadata (fast hint) + if cache_valid: + cached_info = self._read_info_from_cache(cache_dir, path) + if cached_info: + return _stamp(cached_info) + + # No valid cache — use S3 user metadata as a fast preview hint + # (avoids downloading the archive just for preview) + s3_metadata = head.get("Metadata", {}) + info = _bundle_info_from_s3_metadata(s3_metadata, path) + if info: + return _stamp(info) + + # Cache miss or stale — download, cache, and parse. Stream into a + # memory/temp-file sink so a large archive is never held whole in RAM. + buf = _open_download_sink(head.get("ContentLength", 0) if head else 0) + try: + try: + resp = self._s3.get_object(Bucket=self._bucket, Key=key) + shutil.copyfileobj(resp["Body"], buf) + etag = resp.get("ETag", "") + last_modified = str(resp.get("LastModified", "")) + # Refine the size if the earlier HEAD was unavailable. + if content_length is None: + content_length = resp.get("ContentLength") + except Exception: + logger.debug("Failed to download S3 archive %s", key, exc_info=True) + return None + + # Extract to cache so resolve_bundle can reuse it + if os.path.exists(cache_dir): + shutil.rmtree(cache_dir) + os.makedirs(cache_dir, exist_ok=True) + try: + _extract_archive_from_fileobj(buf, cache_dir) + _write_cache_meta(cache_dir, etag, last_modified) + return _stamp(self._read_info_from_cache(cache_dir, path)) + except Exception: + logger.debug("Failed to cache S3 archive %s", key, exc_info=True) + + # Caching failed — still try to read the template for a preview. + try: + buf.seek(0) + with zipfile.ZipFile(buf, "r") as zf: + result = _read_template_from_zip(zf) + except Exception: + return None + if result: + raw, fname = result + template = _parse_template(raw, fname) + if template: + return _stamp(extract_bundle_info(template, path)) + return None + finally: + buf.close() + + def _resolve_archive_bundle( + self, path: str, progress_callback=None, extract_callback=None, extract_size_callback=None + ) -> str: + key = self._to_s3_key(path) + cache_dir = os.path.join(get_bundle_cache_dir(), _cache_key(self._bucket, key)) + + # Single head_object for both cache validation and metadata. Reuse a prior + # head from the get_bundle_size hand-off (_last_head) or the prefetch cache. + head = None + if self._last_head is not None and self._last_head[0] == key: + head = self._last_head[1] + self._last_head = None + else: + try: + head = self._head_object(key) + except Exception: + pass # head_object failure is non-fatal; proceeds without cache validation + + # Check if cache is valid + meta = _read_cache_meta(cache_dir) + if meta and head: + if _normalize_etag(head.get("ETag")) == _normalize_etag(meta.get("etag")): + bundle_path = self._find_bundle_in_cache(cache_dir) + if bundle_path: + logger.info("Using cached bundle: %s", bundle_path) + return bundle_path + + # Download, extract, and cache + etag = head.get("ETag", "") if head else "" + last_modified = str(head.get("LastModified", "")) if head else "" + + # Stream small archives through memory and large ones through a temp file + # in the cache dir, so a large bundle is never held whole (in several + # copies) in RAM. + buf = _open_download_sink(head.get("ContentLength", 0) if head else 0) + try: + download_kwargs: dict = {"Bucket": self._bucket, "Key": key} + if progress_callback: + download_kwargs["Callback"] = progress_callback + self._s3.download_fileobj(Fileobj=buf, **download_kwargs) + + # Clear old cache and extract + if os.path.exists(cache_dir): + shutil.rmtree(cache_dir) + os.makedirs(cache_dir, exist_ok=True) + + _extract_archive_from_fileobj( + buf, + cache_dir, + progress_callback=extract_callback, + size_callback=extract_size_callback, + ) + _write_cache_meta(cache_dir, etag, last_modified) + finally: + buf.close() + + bundle_path = self._find_bundle_in_cache(cache_dir) + if bundle_path: + return bundle_path + return cache_dir + + def _read_info_from_cache(self, cache_dir: str, original_path: str) -> Optional[BundleInfo]: + """Read bundle info from an already-extracted cache directory.""" + bundle_dir = self._find_bundle_in_cache(cache_dir) + if not bundle_dir: + return None + for fname in TEMPLATE_FILENAMES: + fpath = os.path.join(bundle_dir, fname) + if os.path.isfile(fpath): + try: + with open(fpath, encoding="utf-8") as f: + raw = f.read() + except OSError: + return None + template = _parse_template(raw, fname) + if template: + pv = LocalBundleRepository.read_parameter_values(bundle_dir) + return extract_bundle_info(template, original_path, pv) + return None + + @staticmethod + def _find_bundle_in_cache(cache_dir: str) -> Optional[str]: + """Find the actual bundle directory within a cache dir. + Handles both flat extraction and single-directory-wrapped archives.""" + # Check if template is directly in cache_dir + for fname in TEMPLATE_FILENAMES: + if os.path.isfile(os.path.join(cache_dir, fname)): + return cache_dir + # Check one level deep (single wrapper directory) + try: + contents = [ + d + for d in os.listdir(cache_dir) + if os.path.isdir(os.path.join(cache_dir, d)) and d != CACHE_META_FILENAME + ] + except OSError: + return None + for d in contents: + subdir = os.path.join(cache_dir, d) + for fname in TEMPLATE_FILENAMES: + if os.path.isfile(os.path.join(subdir, fname)): + return subdir + return None + + # ── Helpers ────────────────────────────────────────────── + + def _to_s3_key(self, path: str) -> str: + """Convert an s3:// URI to a raw S3 key.""" + if path.startswith("s3://"): + _, _, key = path.partition(f"s3://{self._bucket}/") + return key + return path + + def _to_s3_prefix(self, path: str) -> str: + """Convert an s3:// URI or prefix to a raw S3 prefix ending with /.""" + key = self._to_s3_key(path) + return key if key.endswith("/") else key + "/" + + # ── Visibility (local, per-user) ───────────────────────── + + def _visibility(self) -> "_LocalBundleVisibility": + """Per-user local view for this queue's bundles (keyed by bucket + prefix).""" + return _LocalBundleVisibility(self._bucket, self._prefix) + + def get_hidden_set(self) -> set[str]: + """Fetch this user's locally hidden bundle names for the queue.""" + return self._visibility().get_hidden_set() + + def set_bundle_visibility(self, bundle_name: str, *, hidden: bool) -> None: + """Hide or unhide a bundle in this user's local view (no S3 calls).""" + self._visibility().set_bundle_visibility(bundle_name, hidden=hidden) diff --git a/src/deadline/client/ui/_utils.py b/src/deadline/client/ui/_utils.py index 148234055..cbf07744d 100644 --- a/src/deadline/client/ui/_utils.py +++ b/src/deadline/client/ui/_utils.py @@ -49,6 +49,28 @@ def tr(text: TranslationKey) -> str: return _get_translations().get(text, text) +def warning_banner_qss(widget) -> str: + """Stylesheet for an inline warning banner that adapts to the active theme. + + The submitter has no in-app theme toggle — it inherits the OS/Qt palette — so a + hardcoded light-amber banner looks out of place in dark mode. This derives a + light- or dark-amber treatment from the widget's palette. Both pass WCAG AA + contrast for the body text (light 7.35:1, dark 9.35:1). + """ + from qtpy.QtGui import QPalette # type: ignore + + is_dark = widget.palette().color(QPalette.Window).lightness() < 128 + if is_dark: + text, bg, border = "#ffcc80", "#3a2a10", "#8a5a20" + else: + text, bg, border = "#7a4200", "#fff3e0", "#e0a040" + return ( + f"QLabel {{ color: {text}; background-color: {bg};" + f" border: 1px solid {border}; border-radius: 4px;" + " padding: 4px 8px; }" + ) + + @contextmanager def block_signals(element): """ diff --git a/src/deadline/client/ui/controllers/_async_task.py b/src/deadline/client/ui/controllers/_async_task.py index 4c21c074f..f61602530 100644 --- a/src/deadline/client/ui/controllers/_async_task.py +++ b/src/deadline/client/ui/controllers/_async_task.py @@ -7,11 +7,15 @@ with proper Qt signal integration and automatic cancellation handling. """ +from logging import getLogger from typing import Any, Callable, Iterator, Optional from qtpy.QtCore import QObject, QRunnable, Signal +logger = getLogger(__name__) + + class WorkerSignals(QObject): """ Signals for QRunnable workers. @@ -104,6 +108,28 @@ def is_canceled(self) -> bool: """Check if this task has been canceled.""" return self._is_canceled + def _safe_emit(self, signal_name: str, *args: Any) -> None: + """ + Emit ``self.signals.`` unless the task was canceled or the + signal source has already been deleted. + + ``run()`` executes in a background thread and may still be in-flight when + the owning runner (or the widget it is parented to) is destroyed. When that + happens the underlying ``WorkerSignals`` C++ object is deleted out from under + us, and touching/emitting it raises ``RuntimeError("Signal source has been + deleted")``. A deleted source has no live listeners, so there is nothing to + deliver - we log at debug level and move on instead of letting the exception + cascade through the result/error/finished emissions. + """ + if self._is_canceled: + return + try: + getattr(self.signals, signal_name).emit(*args) + except RuntimeError: + # WorkerSignals was deleted while the task was still running; the + # results are no longer needed by anyone. + logger.debug("Skipping '%s' emit; signal source has been deleted", signal_name) + def run(self) -> None: """ Execute the task in the thread pool. @@ -116,20 +142,21 @@ def run(self) -> None: 2. Execute the function 3. Check if canceled before emitting result/error 4. Emit finished signal (if not canceled) + + Emissions are routed through :meth:`_safe_emit`, which additionally + tolerates the signal source being deleted mid-flight (e.g. when the + owning runner/widget is torn down before a slow task returns). """ if self._is_canceled: return try: result = self.fn(*self.args, **self.kwargs) - if not self._is_canceled: - self.signals.result.emit(result) + self._safe_emit("result", result) except Exception as e: - if not self._is_canceled: - self.signals.error.emit(e) + self._safe_emit("error", e) finally: - if not self._is_canceled: - self.signals.finished.emit() + self._safe_emit("finished") class StreamingAsyncTask(AsyncTask): @@ -159,7 +186,9 @@ def run(self) -> None: Runs in a background thread. Emits ``progress`` per yielded item, a single terminal ``result`` once exhausted, ``error`` if the generator raises, and - ``finished`` at the end. All emissions are guarded by cancellation checks. + ``finished`` at the end. All emissions are guarded by cancellation checks + and tolerate the signal source being deleted mid-flight (see + :meth:`AsyncTask._safe_emit`). """ if self._is_canceled: return @@ -169,13 +198,10 @@ def run(self) -> None: for item in iterator: if self._is_canceled: return - self.signals.progress.emit(item) - if not self._is_canceled: - # Terminal result (no aggregate payload; progress already delivered each item). - self.signals.result.emit(None) + self._safe_emit("progress", item) + # Terminal result (no aggregate payload; progress already delivered each item). + self._safe_emit("result", None) except Exception as e: - if not self._is_canceled: - self.signals.error.emit(e) + self._safe_emit("error", e) finally: - if not self._is_canceled: - self.signals.finished.emit() + self._safe_emit("finished") diff --git a/src/deadline/client/ui/dialogs/bundle_progress_dialog.py b/src/deadline/client/ui/dialogs/bundle_progress_dialog.py new file mode 100644 index 000000000..a57ca35a0 --- /dev/null +++ b/src/deadline/client/ui/dialogs/bundle_progress_dialog.py @@ -0,0 +1,122 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +""" +Two-phase progress dialog for bundle operations (archive/upload, download/extract). +Matches the visual style of the job attachments progress in SubmitJobProgressDialog. +""" + +from __future__ import annotations + +from qtpy.QtCore import Qt # type: ignore +from qtpy.QtWidgets import ( # type: ignore + QDialog, + QDialogButtonBox, + QFormLayout, + QGroupBox, + QLabel, + QProgressBar, + QVBoxLayout, + QWidget, +) + + +class _PhaseWidget(QGroupBox): + """A single phase with a progress bar and status message.""" + + def __init__(self, title: str, parent=None): + super().__init__(title=title, parent=parent) + layout = QFormLayout(self) + layout.setFieldGrowthPolicy(QFormLayout.AllNonFixedFieldsGrow) + self.progress_bar = QProgressBar() + self.status_label = QLabel("") + layout.addWidget(self.progress_bar) + layout.addWidget(self.status_label) + + +class BundleProgressDialog(QDialog): + """A modal dialog showing two-phase progress for bundle operations. + + Each phase has a titled group box with a progress bar and status label, + matching the style of the job attachments submission progress. + + Example:: + + dialog = BundleProgressDialog( + "Saving bundle to queue", + phase1_title="Archiving", + phase2_title="Uploading", + parent=self, + ) + dialog.show() + dialog.set_phase1_progress(50, 100, "50 MB / 100 MB") + dialog.set_phase2_progress(0, 100, "Waiting...") + dialog.set_complete("Bundle saved to queue.") + """ + + def __init__( + self, + window_title: str, + *, + phase1_title: str = "Phase 1", + phase2_title: str = "Phase 2", + parent: QWidget | None = None, + ): + super().__init__(parent=parent) + self.setWindowTitle(window_title) + self.setWindowModality(Qt.WindowModal) + self.setMinimumWidth(450) + self.setWindowFlags( + (self.windowFlags() & ~Qt.WindowContextHelpButtonHint) | Qt.WindowCloseButtonHint + ) + + layout = QVBoxLayout(self) + layout.setContentsMargins(10, 10, 10, 10) + + self._phase1 = _PhaseWidget(phase1_title) + layout.addWidget(self._phase1) + + self._phase2 = _PhaseWidget(phase2_title) + layout.addWidget(self._phase2) + + self._button_box = QDialogButtonBox(QDialogButtonBox.Cancel) + self._button_box.rejected.connect(self.reject) + layout.addWidget(self._button_box) + + self._layout = layout + + def set_phase1_progress(self, value: int, maximum: int, message: str = ""): + self._phase1.progress_bar.setMaximum(maximum) + self._phase1.progress_bar.setValue(value) + if message: + self._phase1.status_label.setText(message) + + def set_phase2_progress(self, value: int, maximum: int, message: str = ""): + self._phase2.progress_bar.setMaximum(maximum) + self._phase2.progress_bar.setValue(value) + if message: + self._phase2.status_label.setText(message) + + def set_complete(self, message: str = "Complete"): + """Mark operation as complete — hide progress, show checkmark, change Cancel to Close.""" + self._phase1.setVisible(False) + self._phase2.setVisible(False) + + if not hasattr(self, "_complete_label"): + self._complete_label = QLabel() + self._complete_label.setAlignment(Qt.AlignCenter) + self._complete_label.setStyleSheet("font-size: 14px; padding: 20px;") + self._layout.insertWidget(0, self._complete_label) + + self._complete_label.setText(f"\u2705 {message}") + self._complete_label.setVisible(True) + + self._button_box.clear() + self._button_box.addButton(QDialogButtonBox.Close) + self._button_box.rejected.connect(self.accept) + + def set_error(self, message: str): + """Show error state.""" + self._phase2.status_label.setText(f"\u26a0 {message}") + self._button_box.clear() + self._button_box.addButton(QDialogButtonBox.Close) + self._button_box.rejected.connect(self.reject) diff --git a/src/deadline/client/ui/dialogs/deadline_config_dialog.py b/src/deadline/client/ui/dialogs/deadline_config_dialog.py index e0a320b6c..5b2824841 100644 --- a/src/deadline/client/ui/dialogs/deadline_config_dialog.py +++ b/src/deadline/client/ui/dialogs/deadline_config_dialog.py @@ -210,7 +210,7 @@ def __init__(self, parent: Optional[QWidget] = None): super().__init__(parent) def sizeHint(self): - return QSize(500, 400) + return QSize(500, 500) class DeadlineWorkstationConfigWidget(QWidget): @@ -248,7 +248,7 @@ def __init__(self, parent: Optional[QWidget] = None): self.refresh() def minimumSizeHint(self): - return QSize(500, 700) + return QSize(500, 800) def _build_ui(self): # Ensure the widget expands horizontally @@ -336,6 +336,18 @@ def _build_profile_settings_ui(self, group, layout): layout.addRow(job_history_dir_label, self.job_history_dir_edit) self.job_history_dir_edit.path_changed.connect(self.job_history_dir_changed) + self.job_bundle_dir_edit = DirectoryPickerWidget( + initial_directory="", + directory_label=tr("Job bundle directory"), + parent=group, + collapse_user_dir=True, + ) + job_bundle_dir_label = self.labels["settings.job_bundle_default_directory"] = QLabel( + tr("Job bundle directory") + ) + layout.addRow(job_bundle_dir_label, self.job_bundle_dir_edit) + self.job_bundle_dir_edit.path_changed.connect(self.job_bundle_dir_changed) + self.default_farm_box = DeadlineFarmListComboBoxController(parent=group) default_farm_box_label = self.labels["defaults.farm_id"] = QLabel(tr("Default farm")) self.default_farm_box.box.currentIndexChanged.connect(self.default_farm_changed) @@ -497,7 +509,8 @@ def refresh_locale_message(): self.labels["settings.known_asset_paths"] = known_paths_label known_paths_widget = QWidget(parent=group) - known_paths_widget.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Minimum) + known_paths_widget.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Expanding) + known_paths_widget.setMinimumHeight(120) known_paths_layout = QVBoxLayout(known_paths_widget) known_paths_layout.setContentsMargins(0, 0, 0, 0) @@ -849,6 +862,15 @@ def refresh(self): ) self.job_history_dir_edit.setText(job_history_dir) + with block_signals(self.job_bundle_dir_edit): + job_bundle_dir = self.changes.get( + "settings.job_bundle_default_directory", + config_file.get_setting( + "settings.job_bundle_default_directory", config=self.config + ), + ) + self.job_bundle_dir_edit.setText(job_bundle_dir) + self.default_farm_box.refresh_selected_id() for refresh_callback in self._refresh_callbacks: @@ -931,6 +953,14 @@ def job_history_dir_changed(self): self.changes["settings.job_history_dir"] = job_history_dir self.refresh() + def job_bundle_dir_changed(self): + job_bundle_dir = self.job_bundle_dir_edit.text() + if job_bundle_dir != config_file.get_setting( + "settings.job_bundle_default_directory", config=self.config + ): + self.changes["settings.job_bundle_default_directory"] = job_bundle_dir + self.refresh() + def default_farm_changed(self, index): if index < 0: return diff --git a/src/deadline/client/ui/dialogs/export_bundle_dialog.py b/src/deadline/client/ui/dialogs/export_bundle_dialog.py new file mode 100644 index 000000000..c1327052b --- /dev/null +++ b/src/deadline/client/ui/dialogs/export_bundle_dialog.py @@ -0,0 +1,140 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +"""Dialog for exporting a job bundle to Queue (S3) or a local directory.""" + +from __future__ import annotations + +import os +from typing import Optional + +from qtpy.QtCore import Qt # type: ignore +from qtpy.QtWidgets import ( # type: ignore + QDialog, + QDialogButtonBox, + QFileDialog, + QHBoxLayout, + QLabel, + QLineEdit, + QPushButton, + QRadioButton, + QVBoxLayout, +) + +from .._utils import tr, warning_banner_qss +from ...job_bundle.repository import S3BundleRepository + + +class ExportBundleDialog(QDialog): + """Dialog to choose where to export a bundle (Queue or Local) with a name override.""" + + def __init__( + self, + *, + default_name: str = "", + queue_repo: Optional[S3BundleRepository] = None, + queue_error: str = "", + local_dir: str = "", + parent=None, + ): + super().__init__(parent=parent) + self.setWindowTitle(tr("Save bundle as")) + self.setMinimumWidth(500) + + self._queue_repo = queue_repo + self._queue_available = queue_repo is not None + self._queue_error = queue_error + self._local_dir = local_dir or os.path.expanduser("~") + + self._build_ui(default_name) + + @property + def bundle_name(self) -> str: + return self._name_edit.text().strip() + + @property + def export_to_queue(self) -> bool: + return self._radio_queue.isChecked() + + @property + def local_directory(self) -> str: + return self._location_edit.text() + + def _build_ui(self, default_name: str): + layout = QVBoxLayout(self) + + # Name + name_row = QHBoxLayout() + name_row.addWidget(QLabel(f"{tr('Name')}:")) + self._name_edit = QLineEdit(default_name) + name_row.addWidget(self._name_edit) + layout.addLayout(name_row) + + # Save to + source_row = QHBoxLayout() + source_row.addWidget(QLabel(f"{tr('Save to')}:")) + self._radio_queue = QRadioButton(tr("Queue")) + self._radio_queue.setEnabled(self._queue_available) + self._radio_queue.setFocusPolicy(Qt.StrongFocus) + self._radio_queue.toggled.connect(self._on_source_changed) + source_row.addWidget(self._radio_queue) + self._radio_local = QRadioButton(tr("Local")) + self._radio_local.setFocusPolicy(Qt.StrongFocus) + self._radio_local.toggled.connect(self._on_source_changed) + source_row.addWidget(self._radio_local) + source_row.addStretch() + layout.addLayout(source_row) + + # Warning label for unavailable queue + self._queue_warning = QLabel() + self._queue_warning.setWordWrap(True) + self._queue_warning.setTextFormat(Qt.RichText) + self._queue_warning.setStyleSheet(warning_banner_qss(self)) + if not self._queue_available and self._queue_error: + self._queue_warning.setText(f"\u26a0 Queue unavailable: {self._queue_error}") + self._queue_warning.setVisible(True) + else: + self._queue_warning.setVisible(False) + layout.addWidget(self._queue_warning) + + # Location + location_row = QHBoxLayout() + location_row.addWidget(QLabel(f"{tr('Location')}:")) + self._location_edit = QLineEdit() + location_row.addWidget(self._location_edit) + self._browse_button = QPushButton("...") + self._browse_button.setFixedWidth(30) + self._browse_button.clicked.connect(self._on_browse) + location_row.addWidget(self._browse_button) + layout.addLayout(location_row) + + # Buttons + button_box = QDialogButtonBox(QDialogButtonBox.Cancel) + self._export_button = QPushButton(tr("Save bundle as")) + button_box.addButton(self._export_button, QDialogButtonBox.AcceptRole) + self._export_button.clicked.connect(self.accept) + button_box.rejected.connect(self.reject) + layout.addWidget(button_box) + + # Default selection + if self._queue_available: + self._radio_queue.setChecked(True) + else: + self._radio_local.setChecked(True) + + def _on_source_changed(self): + if self._radio_queue.isChecked() and self._queue_repo: + s3_path = self._queue_repo.root_path() + self._location_edit.setText(s3_path) + self._location_edit.setReadOnly(True) + self._browse_button.setVisible(False) + else: + self._location_edit.setText(self._local_dir) + self._location_edit.setReadOnly(False) + self._browse_button.setVisible(True) + + def _on_browse(self): + directory = QFileDialog.getExistingDirectory( + self, tr("Select directory"), self._location_edit.text() + ) + if directory: + self._location_edit.setText(directory) diff --git a/src/deadline/client/ui/dialogs/job_bundle_browser_dialog.py b/src/deadline/client/ui/dialogs/job_bundle_browser_dialog.py new file mode 100644 index 000000000..84be3dc00 --- /dev/null +++ b/src/deadline/client/ui/dialogs/job_bundle_browser_dialog.py @@ -0,0 +1,1507 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +""" +Custom job bundle browser dialog that replaces the native folder picker. +Shows a navigable tree of directories/bundles with a preview panel. +""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +from logging import getLogger +from typing import Optional + +from deadline.job_attachments.api import human_readable_file_size + +from qtpy.QtCore import Qt, QModelIndex, QSize, QSortFilterProxyModel, QThread, QTimer, Signal # type: ignore +from qtpy.QtGui import QColor, QPalette, QStandardItemModel, QStandardItem # type: ignore +from qtpy.QtWidgets import ( # type: ignore + QApplication, + QCheckBox, + QDialog, + QDialogButtonBox, + QFrame, + QHBoxLayout, + QHeaderView, + QLabel, + QLineEdit, + QMenu, + QProgressBar, + QPushButton, + QRadioButton, + QGraphicsOpacityEffect, + QScrollArea, + QSplitter, + QStackedWidget, + QTableWidget, + QTableWidgetItem, + QTreeView, + QVBoxLayout, + QWidget, +) + +from .._utils import tr, warning_banner_qss +from ..widgets.expandable_section import ExpandableSection +from ...job_bundle.repository import ( + BrowseEntry, + BundleRepository, + LocalBundleRepository, + S3BundleRepository, +) + +logger = getLogger(__name__) + +# Custom data roles +ROLE_PATH = Qt.UserRole + 1 +ROLE_IS_BUNDLE = Qt.UserRole + 2 +ROLE_LOADED = Qt.UserRole + 3 +ROLE_IS_ARCHIVE = Qt.UserRole + 4 +ROLE_IS_HIDDEN = Qt.UserRole + 5 + +# Semantic warning/accent color, matched to the queue-unavailable banner used +# elsewhere in this dialog so the panel reads as part of the same UI. Legible on +# both light and dark themes. +REQUIRED_COLOR = QColor("#b35900") + + +class _DownloadCancelled(Exception): + """Raised inside a download progress callback to abort a transfer cooperatively. + + Preferred over ``QThread.terminate()``, which would kill the thread mid-request + and could leave the boto3 client/socket in an inconsistent state. + """ + + +# Shared "quiet section label" style — small, bold, sentence-case, muted. + + +class _WrappingLabel(QLabel): + """QLabel that word-wraps without expanding its parent's width. + + Standard QLabel with wordWrap reports a minimumSizeHint equal to the full + single-line width, which pushes splitter panes and scroll areas wider. + This subclass overrides that to allow shrinking. + """ + + def minimumSizeHint(self): + return QSize(0, 0) + + +# Color is set per-widget from the palette so it adapts to the theme. +_SECTION_LABEL_QSS = "font-size: 13px; font-weight: bold;" + +# Friendly, artist-facing labels for the OpenJD parameter type enums. +_FRIENDLY_PARAM_TYPES = { + "STRING": "Text", + "PATH": "Path", + "INT": "Number", + "FLOAT": "Number", +} + + +def _friendly_param_type(raw_type: str) -> str: + """Map an OpenJD parameter type enum to an artist-facing label.""" + return _FRIENDLY_PARAM_TYPES.get(raw_type.upper(), raw_type.title() if raw_type else "?") + + +def _folders_first(entries: list) -> list: + """Order entries folders-first, preserving the repo's name-sort within each group.""" + folders = [e for e in entries if not e.is_bundle] + bundles = [e for e in entries if e.is_bundle] + return folders + bundles + + +def _steps_list_text(step_names: list[str]) -> str: + """Render step names as a plain bulleted list.""" + return "\n".join(f" • {name}" for name in step_names if name) + + +def _normalize_description(text: str) -> str: + """Collapse hard line breaks within a paragraph so the label can word-wrap to + the panel width, while preserving intentional blank-line paragraph breaks. + + Template descriptions are often authored as multi-line YAML blocks, which would + otherwise display with awkward breaks mid-sentence. + """ + paragraphs = re.split(r"\n\s*\n", text.strip()) + return "\n\n".join(" ".join(p.split()) for p in paragraphs if p.strip()) + + +class _BundleFilterProxy(QSortFilterProxyModel): + """Proxy that filters by text and optionally hides items marked as hidden.""" + + def __init__(self, parent=None): + super().__init__(parent) + self._show_hidden = False + + def set_show_hidden(self, show: bool): + self._show_hidden = show + self.invalidateFilter() + + def filterAcceptsRow(self, source_row, source_parent) -> bool: # type: ignore[override] + index = self.sourceModel().index(source_row, 0, source_parent) + if not self._show_hidden and index.data(ROLE_IS_HIDDEN): + return False + return super().filterAcceptsRow(source_row, source_parent) + + +class JobBundleBrowserDialog(QDialog): + """ + A dialog for browsing and selecting job bundles from local filesystem or queue. + + Args: + local_root: Default local directory to browse. + s3_bucket_name: The queue's job attachment S3 bucket name (optional). + s3_root_prefix: The queue's job attachment S3 root prefix (optional). + parent: Parent widget. + """ + + bundle_selected = Signal(str) # Emits the selected bundle path + + def __init__( + self, + *, + queue_source: Optional[S3BundleRepository] = None, + queue_error: str = "", + queue_loading: bool = False, + local_source: str = "", + history_source: str = "", + parent: Optional[QWidget] = None, + ): + super().__init__(parent=parent) + self.setWindowTitle(tr("Browse Job Bundles")) + self.setMinimumSize(750, 550) + self.resize(850, 620) + + self._s3_repo: Optional[S3BundleRepository] = queue_source + self._s3_error = queue_error + self._s3_available = self._s3_repo is not None + self._s3_loading = queue_loading + + self._local_repo = LocalBundleRepository(root=local_source, include_archives=True) + + self._history_repo: Optional[LocalBundleRepository] = None + if history_source and os.path.isdir(history_source): + self._history_repo = LocalBundleRepository(root=history_source, include_archives=True) + + self._current_repo: BundleRepository = self._local_repo + self._selected_path: Optional[str] = None + self._selected_is_s3 = False + self._selected_is_archive = False + self._cached_root_entries: list[BrowseEntry] = [] + self._hidden_set: set[str] = set() + self._last_preview_path: Optional[str] = None + self._tree_states: dict[str, set[str]] = {} # repo root -> expanded paths + self._tree_selections: dict[str, str] = {} # repo root -> selected path + self._s3_refresh_worker: Optional[QThread] = None + self._prefetch_worker: Optional[QThread] = None + self._ready = False + + self._build_ui() + self._ready = True + self._populate_root() + + # Set initial focus to the source radio group + if self._radio_s3.isChecked(): + self._radio_s3.setFocus() + elif self._radio_local.isChecked(): + self._radio_local.setFocus() + + @property + def selected_path(self) -> Optional[str]: + return self._selected_path + + @property + def selected_is_s3(self) -> bool: + return self._selected_is_s3 + + @property + def selected_is_archive(self) -> bool: + return self._selected_is_archive + + @property + def s3_repo(self) -> Optional[S3BundleRepository]: + return self._s3_repo + + def set_queue_source(self, repo, error: str, entries: list = None, hidden_set: set = None): + """Called when background S3 initialization completes.""" + self._s3_loading = False + if repo: + self._s3_repo = repo + self._s3_available = True + self._s3_error = "" + else: + self._s3_available = False + self._s3_error = error + self._radio_s3.setEnabled(False) + # Surface the reason (e.g. not logged in / expired credentials) in the + # inline banner rather than silently disabling the Queue option. + self._show_queue_warning(error) + # Switch to Local if Queue was selected but failed + if self._radio_s3.isChecked(): + self._radio_local.setChecked(True) + return + + # If Queue is currently selected, populate now + if self._radio_s3.isChecked(): + self._current_repo = self._s3_repo + if entries is not None: + # Use pre-fetched data to avoid blocking the main thread + self._cached_root_entries = _folders_first(entries) + self._hidden_set = hidden_set or set() + self._populate_tree_from_cache() + else: + self._populate_root() + + def resolve_selection(self) -> Optional[str]: + """Resolve the selected bundle to a local directory path. + + Handles S3 download/cache, archive extraction, and direct directory paths. + Returns None if no selection. + """ + if not self._selected_path: + return None + + if self._selected_is_s3 and self._s3_repo: + + class _DownloadWorker(QThread): + progress = Signal(int) + # Emitted once the archive size is known (from a head_object made + # on this worker thread — see below). Declared as ``qlonglong`` + # (64-bit) because archive sizes routinely exceed INT_MAX (2 GiB); + # a plain ``Signal(int)`` maps to a 32-bit C++ int, so a >2 GiB + # size overflows (shiboken raises OverflowError / clamps the value, + # which is why the reported total came out as a garbage "1 KB"). + size_ready = Signal("qlonglong") # type: ignore[arg-type] + # NOTE: named ``done`` rather than ``finished`` to avoid shadowing + # QThread's built-in ``finished`` signal. + done = Signal(str) + error = Signal(str) + + def __init__(self, repo, path): + super().__init__() + self._repo = repo + self._path = path + self._sent = 0 + self._cancelled = False + + def cancel(self): + # Cooperative cancel: the next progress callback aborts the + # transfer, letting boto3 unwind cleanly. + self._cancelled = True + + def run(self): + try: + # Determine the archive size here rather than on the main + # (Qt) thread: get_bundle_size() makes a synchronous + # head_object network call, which would block the UI. + # The result is cached in the repo's _last_head and reused + # by the subsequent download, so this adds no extra call. + total = self._repo.get_bundle_size(self._path) + self.size_ready.emit(total) + + def _cb(n): + if self._cancelled: + # Abort the transfer cooperatively instead of + # terminating the thread, which would risk an + # inconsistent boto3 client/socket. + raise _DownloadCancelled() + self._sent += n + self.progress.emit(self._sent // 1024) + + result = self._repo.download_full_bundle( + self._path, "", progress_callback=_cb + ) + self.done.emit(result) + except _DownloadCancelled: + # User cancelled — nothing to report; the main thread + # clears any partial cache after the worker unwinds. + pass + except Exception as e: + self.error.emit(str(e)) + + progress = QDialog(self) + progress.setWindowFlags(Qt.Dialog | Qt.CustomizeWindowHint | Qt.WindowTitleHint) + progress.setWindowTitle("Downloading Bundle") + progress.setWindowModality(Qt.ApplicationModal) + progress.setMinimumWidth(350) + _dlg_layout = QVBoxLayout(progress) + _progress_label = QLabel("Downloading bundle...") + _progress_label.setAlignment(Qt.AlignCenter) + _progress_bar = QProgressBar() + # Start indeterminate (busy) until the worker reports the size. + _progress_bar.setRange(0, 0) + _cancel_btn = QPushButton("Cancel") + _cancel_btn.clicked.connect(progress.reject) + _dlg_layout.addWidget(_progress_label) + _dlg_layout.addWidget(_progress_bar) + _dlg_layout.addWidget(_cancel_btn, alignment=Qt.AlignRight) + + worker = _DownloadWorker(self._s3_repo, self._selected_path) + download_result = [None] + download_error = [] + # The true archive size in bytes. Tracked on the Python side rather + # than read back from the progress bar's maximum, because the bar + # works in KiB (a 32-bit int, so multi-GiB byte counts would overflow + # it) and reconstructing bytes from it loses precision. + total_bytes = [0] + + def _on_size(total): + total_bytes[0] = total + try: + # Progress bar tracks KiB to stay within QProgressBar's + # 32-bit int range for large (multi-GiB) archives. + _progress_bar.setRange(0, max(1, total // 1024)) + except RuntimeError: + return + + def _on_progress(n): + try: + _progress_bar.setValue(n) + except RuntimeError: + return + total = total_bytes[0] + current = n * 1024 + if total > 0: + _progress_label.setText( + f"Downloading bundle... {human_readable_file_size(current)}" + f" / {human_readable_file_size(total)}" + ) + + def _on_finished(path): + download_result[0] = path + progress.close() + + def _on_error(msg): + download_error.append(msg) + progress.close() + + worker.size_ready.connect(_on_size, Qt.QueuedConnection) + worker.progress.connect(_on_progress, Qt.QueuedConnection) + worker.done.connect(_on_finished, Qt.QueuedConnection) + worker.error.connect(_on_error, Qt.QueuedConnection) + # Cancelling the dialog (Cancel button or window close) flags the + # worker so its download callback aborts cooperatively. + progress.rejected.connect(worker.cancel) + worker.start() + progress.exec_() + + if not download_result[0]: + # Cancelled or error — stop the worker cooperatively (the flag is + # checked in the download callback) and wait for it to unwind so + # the boto3 client/socket close cleanly, then remove partial cache. + worker.cancel() + worker.wait() + self._s3_repo.clear_cache_for(self._selected_path) + return None + + worker.wait() + return download_result[0] + elif self._selected_is_archive: + QApplication.setOverrideCursor(Qt.WaitCursor) + try: + return self._local_repo.extract_bundle(self._selected_path, "") + finally: + QApplication.restoreOverrideCursor() + else: + return self._selected_path + + # ── UI Construction ────────────────────────────────────────── + + def _build_ui(self): + layout = QVBoxLayout(self) + + # Source toggle row — at the top so users select source before browsing + source_row = QHBoxLayout() + source_label = QLabel(tr("Source:")) + source_row.addWidget(source_label) + self._radio_s3 = QRadioButton(tr("Queue")) + self._radio_s3.setEnabled(self._s3_available or self._s3_loading) + self._radio_s3.setFocusPolicy(Qt.StrongFocus) + self._radio_s3.toggled.connect(self._on_source_changed) + source_row.addWidget(self._radio_s3) + self._radio_local = QRadioButton(tr("Local")) + self._radio_local.setFocusPolicy(Qt.StrongFocus) + self._radio_local.toggled.connect(self._on_source_changed) + source_row.addWidget(self._radio_local) + self._radio_history = QRadioButton(tr("History")) + self._radio_history.setFocusPolicy(Qt.StrongFocus) + self._radio_history.setEnabled(self._history_repo is not None) + self._radio_history.toggled.connect(self._on_source_changed) + source_row.addWidget(self._radio_history) + source_row.addStretch() + layout.addLayout(source_row) + + # Inline warning when queue source is unavailable + self._queue_warning = QLabel() + self._queue_warning.setWordWrap(True) + # Let users select/copy the error text (e.g. to paste an expired-token or + # AccessDenied message into a ticket). QLabels aren't selectable by default. + self._queue_warning.setTextInteractionFlags( + Qt.TextSelectableByMouse | Qt.TextSelectableByKeyboard + ) + self._queue_warning.setStyleSheet(warning_banner_qss(self)) + if not self._s3_available and self._s3_error: + self._queue_warning.setText( + f"\u26a0 Queue browsing unavailable: {self._s3_error}" + ) + self._queue_warning.setTextFormat(Qt.RichText) + self._queue_warning.setVisible(True) + else: + self._queue_warning.setVisible(False) + layout.addWidget(self._queue_warning) + + # #7 — Path display sits just under the source selection (both describe + # "where am I browsing"), rather than at the bottom by the buttons. + path_row = QHBoxLayout() + path_row.addWidget(QLabel(tr("Path:"))) + self._path_display = QLineEdit() + self._path_display.setReadOnly(True) + # Look like static text, not an editable field: drop the frame and fill so + # it blends into the dialog. ClickFocus (rather than NoFocus) is kept so the + # user can still click-drag to select and copy the path. + self._path_display.setFrame(False) + self._path_display.setFocusPolicy(Qt.ClickFocus) + self._path_display.setStyleSheet("QLineEdit { background: transparent; border: none; }") + path_row.addWidget(self._path_display) + layout.addLayout(path_row) + + # Default to Queue if available or loading, otherwise Local + if self._s3_available or self._s3_loading: + self._radio_s3.setChecked(True) + if self._s3_repo: + self._current_repo = self._s3_repo + else: + self._current_repo = None # Will be set by set_queue_source + else: + self._radio_local.setChecked(True) + + # Main splitter: tree on left, preview on right + splitter = QSplitter(Qt.Horizontal) + layout.addWidget(splitter, stretch=1) + + # Left: tree view with filter + left_widget = QWidget() + left_layout = QVBoxLayout(left_widget) + left_layout.setContentsMargins(0, 0, 0, 0) + + # Row 1: filter + "Show hidden" (both are list-view controls, grouped above + # the tree). Show hidden is a filter toggle, so it lives with the filter. + filter_row = QHBoxLayout() + self._filter_edit = QLineEdit() + self._filter_edit.setPlaceholderText(tr("Filter bundles...")) + self._filter_edit.setClearButtonEnabled(True) + self._filter_edit.textChanged.connect(self._on_filter_changed) + filter_row.addWidget(self._filter_edit, stretch=1) + + self._show_hidden_cb = QCheckBox(tr("Show hidden"), parent=self) + self._show_hidden_cb.setChecked(False) + self._show_hidden_cb.toggled.connect(self._on_hidden_toggled) + filter_row.addSpacing(8) + filter_row.addWidget(self._show_hidden_cb) + left_layout.addLayout(filter_row) + + self._model = QStandardItemModel() + self._model.setHorizontalHeaderLabels([tr("Name")]) + + self._proxy = _BundleFilterProxy() + self._proxy.setSourceModel(self._model) + self._proxy.setRecursiveFilteringEnabled(True) + self._proxy.setFilterCaseSensitivity(Qt.CaseInsensitive) + + self._tree = QTreeView() + self._tree.setModel(self._proxy) + self._tree.setHeaderHidden(True) + self._tree.setEditTriggers(QTreeView.NoEditTriggers) + self._tree.setContextMenuPolicy(Qt.CustomContextMenu) + self._tree.customContextMenuRequested.connect(self._on_context_menu) + self._tree.expanded.connect(self._on_expanded) + self._tree.clicked.connect(self._on_clicked) + self._tree.doubleClicked.connect(self._on_double_clicked) + self._tree.selectionModel().currentChanged.connect(self._on_selection_changed) + # Overlay label for empty tree state (no bundles or no filter match) + self._tree_empty_label = QLabel("No bundles found", self._tree.viewport()) + self._tree_empty_label.setAlignment(Qt.AlignCenter) + self._tree_empty_label.setStyleSheet( + "color: palette(text); font-size: 13px; background: transparent;" + ) + self._tree_empty_label.setAttribute(Qt.WA_TransparentForMouseEvents) + self._tree_empty_label.setVisible(False) + self._tree.viewport().installEventFilter(self) + self._tree.installEventFilter(self) + left_layout.addWidget(self._tree) + + splitter.addWidget(left_widget) + + # Right: preview panel. A QStackedWidget switches between an empty-state + # page (centered prompt) and the detail page (scrollable bundle info). + self._muted_hex = self.palette().color(QPalette.PlaceholderText).name() + muted_qss = f"color: {self._muted_hex};" + + self._preview_stack = QStackedWidget() + + # Empty-state page — icon + prompt + hint, centered both ways. + empty_page = QWidget() + empty_layout = QVBoxLayout(empty_page) + empty_layout.setSpacing(4) + + # Large, low-opacity bundle glyph as a backdrop (matches the tree's 📦). + empty_icon = QLabel("\U0001f4e6") + empty_icon.setAlignment(Qt.AlignCenter) + empty_icon.setStyleSheet("font-size: 44px;") + empty_icon.setGraphicsEffect(self._make_opacity(0.35)) + + empty_prompt = QLabel(tr("Select a job bundle")) + empty_prompt.setAlignment(Qt.AlignCenter) + empty_prompt.setStyleSheet("font-size: 15px; font-weight: bold;") + + self._empty_label = QLabel(tr("Choose one from the list to preview its details")) + self._empty_label.setAlignment(Qt.AlignCenter) + self._empty_label.setWordWrap(True) + self._empty_label.setStyleSheet(f"font-size: 12px; {muted_qss}") + + empty_layout.addStretch(1) + empty_layout.addWidget(empty_icon) + empty_layout.addWidget(empty_prompt) + empty_layout.addWidget(self._empty_label) + empty_layout.addStretch(1) + self._preview_stack.addWidget(empty_page) # index 0 + + # Detail page — scrollable bundle info. + preview_widget = QWidget() + preview_layout = QVBoxLayout(preview_widget) + # One consistent vertical rhythm instead of scattered per-widget margins. + preview_layout.setSpacing(6) + + # Title — top of a 3-step scale (title / body / section-label). + self._preview_name = _WrappingLabel() + self._preview_name.setWordWrap(True) + # Plain text: the bundle-derived name must never be interpreted as markup. + # Wrapping and all visual styling come from setWordWrap + the stylesheet + # below (widget-level styling is independent of the text format); we only + # forgo CSS break-word for a pathological unbroken name, which is bounded + # by PREVIEW_MAX_NAME_LEN and shrinks via _WrappingLabel. + self._preview_name.setTextFormat(Qt.PlainText) + self._preview_name.setStyleSheet("font-weight: bold; font-size: 18px;") + preview_layout.addWidget(self._preview_name) + + # Muted subline: bundle type + source (e.g. "📦 Folder · Local") + self._preview_subline = QLabel() + self._preview_subline.setStyleSheet(f"font-size: 11px; {muted_qss}") + preview_layout.addWidget(self._preview_subline) + + # Extra breathing room between the title/subline block and the description. + preview_layout.addSpacing(12) + + # Description — expandable, default expanded. + self._desc_section = ExpandableSection(expanded=True, disable_content_paddings=True) + self._desc_section.set_header_style(f"{_SECTION_LABEL_QSS} {muted_qss}") + self._preview_desc = _WrappingLabel() + self._preview_desc.setWordWrap(True) + # Plain text: bundle-derived descriptions/errors must never be interpreted + # as rich text (defense against HTML/CSS injection from crafted metadata). + self._preview_desc.setTextFormat(Qt.PlainText) + # Selectable so users can copy a description or an error message shown here. + self._preview_desc.setTextInteractionFlags( + Qt.TextSelectableByMouse | Qt.TextSelectableByKeyboard + ) + self._desc_section.set_content(self._preview_desc) + preview_layout.addWidget(self._desc_section) + + preview_layout.addSpacing(8) + # Parameters come before Steps and default to expanded — they're the most + # commonly inspected detail before submitting. + # Content paddings are disabled here; the table is indented via its own + # stylesheet margin (see below) to avoid a double indent. + self._params_section = ExpandableSection(expanded=True, disable_content_paddings=True) + self._params_section.set_header_style(f"{_SECTION_LABEL_QSS} {muted_qss}") + self._preview_params = QTableWidget() + self._preview_params.setColumnCount(3) + self._preview_params.setHorizontalHeaderLabels([tr("Name"), tr("Type"), tr("Value")]) + header = self._preview_params.horizontalHeader() + # Name/Type size to content but are capped (see _size_params_table below); + # Value takes the rest and wraps long content rather than eliding/scrolling. + header.setSectionResizeMode(0, QHeaderView.Interactive) + header.setSectionResizeMode(1, QHeaderView.ResizeToContents) + header.setSectionResizeMode(2, QHeaderView.Stretch) + header.setHighlightSections(False) + # Left-align the column headers to match the cell text below them. + header.setDefaultAlignment(Qt.AlignLeft | Qt.AlignVCenter) + self._preview_params.verticalHeader().setVisible(False) + self._preview_params.setEditTriggers(QTableWidget.NoEditTriggers) + self._preview_params.setSelectionMode(QTableWidget.NoSelection) + # #2 — subtle alternating row tint so rows are easier to scan. + self._preview_params.setAlternatingRowColors(True) + # Wrap long Name/Value text onto multiple lines instead of eliding it. + self._preview_params.setWordWrap(True) + self._preview_params.setTextElideMode(Qt.ElideNone) + self._preview_params.setShowGrid(False) + self._preview_params.setFocusPolicy(Qt.NoFocus) + # No frame on the table itself — the panel outline already contains it. + self._preview_params.setFrameShape(QFrame.NoFrame) + # #5 — header divider; header is transparent so it sits on the panel surface. + hdr_line = self._muted_hex + header.setStyleSheet( + "QHeaderView::section {" + " background: transparent;" + f" border: none; border-bottom: 1px solid {hdr_line};" + " padding: 4px 6px; font-weight: bold; }" + ) + # Table is transparent (inherits the panel surface); alternating rows provide + # the only fill, so they read as subtle stripes on the panel. The margin-left + # indents the table under its section header (QSS margin is honored even + # though programmatic setContentsMargins is not, once a stylesheet is set). + self._preview_params.setStyleSheet( + "QTableWidget { background: transparent; margin-left: 16px; }" + ) + # Let the panel's own scroll area handle overflow; the table sizes to its rows. + self._preview_params.setVerticalScrollBarPolicy(Qt.ScrollBarAlwaysOff) + self._preview_params.setHorizontalScrollBarPolicy(Qt.ScrollBarAlwaysOff) + self._preview_params.setSizeAdjustPolicy(QTableWidget.AdjustToContents) + self._params_section.set_content(self._preview_params) + preview_layout.addWidget(self._params_section) + + preview_layout.addSpacing(8) + self._steps_section = ExpandableSection(expanded=True, disable_content_paddings=True) + self._steps_section.set_header_style(f"{_SECTION_LABEL_QSS} {muted_qss}") + self._preview_steps = QLabel() + self._preview_steps.setWordWrap(True) + # Plain text: step names come from bundle metadata and must be literal. + self._preview_steps.setTextFormat(Qt.PlainText) + self._steps_section.set_content(self._preview_steps) + preview_layout.addWidget(self._steps_section) + + preview_layout.addStretch(1) + + self._clear_preview() + + # Inner padding so content doesn't crowd the panel edges. + preview_layout.setContentsMargins(14, 14, 14, 14) + + preview_scroll = QScrollArea() + preview_scroll.setWidget(preview_widget) + preview_scroll.setWidgetResizable(True) + preview_scroll.setHorizontalScrollBarPolicy(Qt.ScrollBarAlwaysOff) + preview_scroll.setFrameShape(QFrame.NoFrame) + # Make the detail page transparent so the panel's gradient shows through here + # too (a QScrollArea + its content otherwise paint an opaque Base background). + preview_scroll.setStyleSheet( + "QScrollArea, QScrollArea > QWidget > QWidget { background: transparent; }" + ) + preview_widget.setAttribute(Qt.WA_TranslucentBackground, False) + preview_widget.setStyleSheet("background: transparent;") + + # Detail page = scrollable content + an action row pinned to the bottom. + # "Download bundle" lives here (rather than the dialog's button box) so it + # is anchored to the preview and only appears while a bundle is previewed. + # It opens the bundle in the OS file explorer without closing the dialog: + # Queue bundles are downloaded/extracted to the cache first (reusing the + # Select flow); local/history bundles open in place. + self._download_button = QPushButton(tr("Download bundle")) + self._download_button.clicked.connect(self._on_download) + self._download_button.setCursor(Qt.PointingHandCursor) + # The preview panel is stylesheet-driven (transparent surface + gradient), + # which flattens a default child button so it stops reading as clickable. + # Give it an explicit, theme-aware style (fill, border, radius, hover / + # pressed feedback) derived from the palette so it stands out on the panel. + _pal = self.palette() + _btn_bg = _pal.color(QPalette.Button) + _btn_txt = _pal.color(QPalette.ButtonText) + _wt = _pal.color(QPalette.WindowText) + _border = f"rgba({_wt.red()}, {_wt.green()}, {_wt.blue()}, 120)" + _is_dark = _btn_bg.lightness() < 128 + _hover_bg = _btn_bg.lighter(118) if _is_dark else _btn_bg.darker(104) + _pressed_bg = _btn_bg.lighter(105) if _is_dark else _btn_bg.darker(112) + self._download_button.setStyleSheet( + "QPushButton {" + f" background-color: {_btn_bg.name()};" + f" color: {_btn_txt.name()};" + f" border: 1px solid {_border};" + " border-radius: 6px; padding: 5px 14px; font-weight: 600; }" + f" QPushButton:hover {{ background-color: {_hover_bg.name()}; }}" + f" QPushButton:pressed {{ background-color: {_pressed_bg.name()}; }}" + ) + download_row = QHBoxLayout() + download_row.setContentsMargins(14, 4, 14, 12) + download_row.addStretch(1) + download_row.addWidget(self._download_button) + + detail_page = QWidget() + detail_layout = QVBoxLayout(detail_page) + detail_layout.setContentsMargins(0, 0, 0, 0) + detail_layout.setSpacing(0) + detail_layout.addWidget(preview_scroll, 1) + detail_layout.addLayout(download_row) + detail_page.setStyleSheet("background: transparent;") + self._preview_stack.addWidget(detail_page) # index 1 + + # #4 + #5 — the whole panel is a raised, rounded surface distinct from the + # tree, with a subtle top-to-bottom gradient for a bit of depth. Both stops + # are derived from QPalette.Base so it adapts to the theme. objectName + # scoping keeps the styling off child widgets. + base = self.palette().color(QPalette.Base) + is_dark = base.lightness() < 128 + # Lighten the top / darken the bottom. Kept subtle: enough to add depth but + # gentle enough not to compete with the dense detail content for readability. + # (Near-black Base needs a larger % shift than a light Base to be visible.) + top = base.lighter(128) if is_dark else base.lighter(104) + bottom = base.darker(110) if is_dark else base.darker(106) + tc = self.palette().color(QPalette.WindowText) + border = f"rgba({tc.red()}, {tc.green()}, {tc.blue()}, 70)" + self._preview_stack.setObjectName("previewPanel") + self._preview_stack.setStyleSheet( + "#previewPanel {" + " background: qlineargradient(x1:0, y1:0, x2:0, y2:1," + f" stop:0 {top.name()}, stop:1 {bottom.name()});" + f" border: 1px solid {border}; border-radius: 8px; }}" + ) + splitter.addWidget(self._preview_stack) + splitter.setSizes([350, 350]) + + # Dialog buttons + self._button_box = QDialogButtonBox(QDialogButtonBox.Cancel) + self._select_button = QPushButton(tr("Select")) + self._select_button.setDefault(True) + self._select_button.setEnabled(False) + self._button_box.addButton(self._select_button, QDialogButtonBox.AcceptRole) + self._select_button.clicked.connect(self.accept) + self._button_box.rejected.connect(self.reject) + layout.addWidget(self._button_box) + + # Tab order: Source radios → Filter → Tree → Select + self.setTabOrder(self._radio_s3, self._filter_edit) + self.setTabOrder(self._filter_edit, self._tree) + self.setTabOrder(self._tree, self._select_button) + + # ── Tree Population ────────────────────────────────────────── + + def _refresh_s3_async(self): + """Refresh S3 listing in a background thread.""" + from concurrent.futures import ThreadPoolExecutor + + # Wait for any previous refresh to finish + if hasattr(self, "_s3_refresh_worker") and self._s3_refresh_worker is not None: + self._s3_refresh_worker.wait() + + assert self._s3_repo is not None + repo = self._s3_repo + + class _Worker(QThread): + # NOTE: named ``done`` rather than ``finished`` to avoid shadowing + # QThread's built-in ``finished`` signal. + done = Signal(list, set) + error = Signal(str) + + def run(self): + try: + with ThreadPoolExecutor(max_workers=2) as ex: + entries_f = ex.submit(repo.list_entries, repo.root_path()) + hidden_f = ex.submit(repo.get_hidden_set) + self.done.emit(entries_f.result(), hidden_f.result()) + except Exception as e: + # Without this, an exception (e.g. expired/invalid credentials + # raised by the S3 list call) would kill the thread silently + # and leave the tree stuck on "Loading..." forever. + self.error.emit(str(e)) + + self._s3_refresh_worker = _Worker() + self._s3_refresh_worker.done.connect(self._on_s3_refresh_done) + self._s3_refresh_worker.error.connect(self._on_s3_refresh_error) + self._s3_refresh_worker.start() + + def _on_s3_refresh_done(self, entries, hidden_set): + """Handle background S3 refresh completion.""" + if not self._radio_s3.isChecked(): + return # User switched away + self._cached_root_entries = _folders_first(entries) + self._hidden_set = hidden_set + self._populate_tree_from_cache() + self._restore_tree_state() + # Listing is now on screen (fast: one list + the local visibility file). + # Warm the preview cache in the background so opening a bundle is instant. + self._start_preview_prefetch() + + def _start_preview_prefetch(self): + """Warm the repo's preview cache off the UI thread (best-effort).""" + if self._s3_repo is None: + return + # Let any previous prefetch finish first. + prev = self._prefetch_worker + if prev is not None: + prev.wait() + + repo = self._s3_repo + + class _PrefetchWorker(QThread): + def run(self): + try: + repo.prefetch_previews() + except Exception: + # Prefetch is a pure optimization — never surface its failures; + # previews fall back to an on-demand HEAD. + logger.debug("Preview prefetch failed", exc_info=True) + + self._prefetch_worker = _PrefetchWorker() + self._prefetch_worker.start() + + def _on_s3_refresh_error(self, message: str): + """Handle a failed queue listing (e.g. expired credentials). + + Surfaces the underlying error in the inline banner and clears the + "Loading..." placeholder, instead of leaving the browser stuck. + """ + logger.warning("Failed to refresh queue bundles: %s", message) + if not self._radio_s3.isChecked(): + return # User switched away before the failure arrived + self._cached_root_entries = [] + self._model.clear() + self._model.setHorizontalHeaderLabels([tr("Name")]) + self._show_queue_warning(message) + if hasattr(self, "_tree_empty_label"): + self._tree_empty_label.setText("Could not load queue bundles") + self._tree_empty_label.setVisible(True) + + def _populate_root(self): + self._model.clear() + self._model.setHorizontalHeaderLabels([tr("Name")]) + if self._current_repo is None: + # Queue source still loading + self._path_display.setText("") + self._cached_root_entries = [] + if hasattr(self, "_tree_empty_label"): + self._tree_empty_label.setText("Loading...") + self._tree_empty_label.setVisible(True) + return + root_path = self._current_repo.root_path() + self._path_display.setText(root_path) + try: + self._cached_root_entries = self._current_repo.list_entries(root_path) + except Exception as e: + logger.warning("Failed to list bundles: %s", e, exc_info=True) + self._show_error_preview(f"Failed to list bundles:\n{e}") + self._cached_root_entries = [] + + # Fetch S3 hidden set for Queue source + self._hidden_set = set() + if isinstance(self._current_repo, S3BundleRepository): + try: + self._hidden_set = self._current_repo.get_hidden_set() + except Exception: + logger.debug("Failed to fetch hidden bundle set", exc_info=True) + + # Folders first, then bundles — each group already name-sorted by the repo. + self._cached_root_entries = _folders_first(self._cached_root_entries) + + root = self._model.invisibleRootItem() + for entry in self._cached_root_entries: + is_hidden = entry.name.startswith(".") or entry.name in self._hidden_set + self._add_entry_item(root, entry, is_hidden=is_hidden) + + self._preload_first_level() + self._update_tree_empty_state() + + def _populate_tree_from_cache(self): + """Populate tree from pre-fetched _cached_root_entries and _hidden_set.""" + self._model.clear() + self._model.setHorizontalHeaderLabels([tr("Name")]) + self._path_display.setText(self._current_repo.root_path()) + root = self._model.invisibleRootItem() + for entry in self._cached_root_entries: + is_hidden = entry.name.startswith(".") or entry.name in self._hidden_set + self._add_entry_item(root, entry, is_hidden=is_hidden) + self._preload_first_level() + self._update_tree_empty_state() + + def _apply_hidden_style(self, item: QStandardItem, hidden: bool) -> None: + """Apply (or clear) the dimmed styling that marks a hidden bundle. + + When un-hiding, the foreground override is cleared (set to ``None``) so the + item falls back to the palette's text color. Hard-coding a color such as + black would render the name nearly invisible under a dark theme. + """ + if hidden: + item.setForeground(QColor(150, 150, 150)) + else: + item.setData(None, Qt.ForegroundRole) + + def _add_entry_item( + self, parent_item: QStandardItem, entry: BrowseEntry, *, is_hidden: bool = False + ): + item = QStandardItem(self._entry_display(entry)) + item.setData(entry.path, ROLE_PATH) + item.setData(entry.is_bundle, ROLE_IS_BUNDLE) + item.setData(False, ROLE_LOADED) + item.setData(entry.is_archive, ROLE_IS_ARCHIVE) + item.setData(is_hidden, ROLE_IS_HIDDEN) + self._apply_hidden_style(item, is_hidden) + if not entry.is_bundle: + # Add a placeholder child so the expand arrow shows + placeholder = QStandardItem() + placeholder.setData(is_hidden, ROLE_IS_HIDDEN) + item.appendRow(placeholder) + parent_item.appendRow(item) + + @staticmethod + def _entry_display(entry: BrowseEntry) -> str: + icon = "\U0001f4e6" if entry.is_bundle else "\U0001f4c1" # 📦 or 📁 + suffix = ".ojd" if entry.is_archive and not entry.path.startswith("s3://") else "" + return f"{icon} {entry.name}{suffix}" + + # ── Event Handlers ─────────────────────────────────────────── + + def _source_item(self, proxy_index: QModelIndex): + """Map a proxy model index to the source model item.""" + source_index = self._proxy.mapToSource(proxy_index) + return self._model.itemFromIndex(source_index) + + def _on_expanded(self, proxy_index: QModelIndex): + self._load_children(self._source_item(proxy_index)) + + def _load_children(self, item) -> None: + """Populate a folder item's real children, replacing its placeholder. + + Idempotent: a no-op for bundles or folders already loaded (so re-expanding, + or expanding a folder whose children were preloaded, doesn't re-fetch). + """ + if not item or item.data(ROLE_IS_BUNDLE) or item.data(ROLE_LOADED): + return + # Mark as loaded and replace placeholder with real children + item.setData(True, ROLE_LOADED) + item.removeRows(0, item.rowCount()) + path = item.data(ROLE_PATH) + try: + entries = self._current_repo.list_entries(path) + except Exception as e: + logger.warning("Failed to list bundles in %s: %s", path, e, exc_info=True) + error_item = QStandardItem(f"\u26a0 Error: {e}") + error_item.setEnabled(False) + item.appendRow(error_item) + return + for entry in _folders_first(entries): + is_hidden = entry.name.startswith(".") or entry.name in self._hidden_set + # If parent is hidden, children inherit hidden state + if item.data(ROLE_IS_HIDDEN): + is_hidden = True + self._add_entry_item(item, entry, is_hidden=is_hidden) + + def _preload_first_level(self) -> None: + """Eagerly load the immediate children of each top-level folder. + + This lets the filter match one level below the root without the user first + expanding folders. Deeper levels stay lazy-loaded on expand. Kept to a + single level so the up-front cost (a directory read, or one S3 list per + top-level folder) stays bounded. + """ + root = self._model.invisibleRootItem() + for row in range(root.rowCount()): + child = root.child(row) + if child is not None and not child.data(ROLE_IS_BUNDLE): + self._load_children(child) + + def _on_clicked(self, proxy_index: QModelIndex): + self._update_selection(proxy_index) + + def _on_double_clicked(self, proxy_index: QModelIndex): + item = self._source_item(proxy_index) + if item and item.data(ROLE_IS_BUNDLE): + self._update_selection(proxy_index) + self.accept() + + def _on_selection_changed(self, current: QModelIndex, previous: QModelIndex): + self._update_selection(current) + + def _show_queue_warning(self, message: str) -> None: + """Show the inline 'Queue browsing unavailable' banner with ``message``. + + Used both for background initialization failures and for a failed queue + listing refresh (e.g. expired/invalid credentials), so the reason is + surfaced instead of leaving the tree stuck on 'Loading...'. + """ + self._s3_error = message + self._queue_warning.setText(f"\u26a0 Queue browsing unavailable: {message}") + self._queue_warning.setTextFormat(Qt.RichText) + self._queue_warning.setVisible(bool(message)) + + def _on_download(self) -> None: + """Open the selected bundle in the OS file explorer, keeping the dialog open. + + Queue (S3) bundles are downloaded — and archives extracted — to the local + cache first, reusing the same flow as Select (so a large download shows + the progress dialog and can be cancelled). Local and history bundles are + opened in place. Errors are surfaced inline in the preview panel rather + than as popups. + """ + if not self._selected_path: + return + try: + local_path = self.resolve_selection() + except Exception as e: + logger.warning("Failed to resolve bundle for download: %s", e, exc_info=True) + self._show_error_preview(f"\u26a0 Could not open bundle: {e}") + return + if not local_path: + # Cancelled, or a download error was already surfaced — nothing to open. + return + try: + self._open_in_file_explorer(local_path) + except Exception as e: + logger.warning("Failed to open %s in file explorer: %s", local_path, e, exc_info=True) + self._show_error_preview(f"\u26a0 Could not open bundle location: {e}") + + @staticmethod + def _open_in_file_explorer(path: str) -> None: + """Reveal a local path in the platform file explorer. + + Uses list-form invocation (no shell) so a path is never interpreted as a + command. ``path`` is a local directory produced by ``resolve_selection``. + """ + if sys.platform == "darwin": + subprocess.run(["open", path], check=False) + elif sys.platform.startswith("win"): + os.startfile(path) # type: ignore[attr-defined] # Windows-only + else: + subprocess.run(["xdg-open", path], check=False) + + def _on_filter_changed(self, text: str): + self._proxy.setFilterFixedString(text) + if text: + self._tree.expandAll() + elif not self._selected_path: + # Collapse back to the default root view once the filter is cleared and + # nothing is chosen (filtering had expanded everything to reveal + # matches). Deferred to the next event-loop turn so it never runs while + # the proxy is mid-refilter or inside a nested selection/clear — doing + # tree structure ops in that window can touch a stale index and crash + # (segfault). If a bundle IS selected, keep the expansion so it stays + # visible in its folder context. + QTimer.singleShot(0, self._tree.collapseAll) + self._update_tree_empty_state() + + def _update_tree_empty_state(self): + """Show/hide the 'No bundles found' overlay based on visible row count.""" + has_rows = self._proxy.rowCount() > 0 + self._tree_empty_label.setVisible(not has_rows) + if not has_rows: + self._tree_empty_label.setText("No bundles found") + self._tree_empty_label.resize(self._tree.viewport().size()) + + def eventFilter(self, obj, event): # type: ignore[override] + if obj is self._tree.viewport() and event.type() == event.Type.Resize: + self._tree_empty_label.resize(event.size()) + elif obj is self._tree and event.type() == event.Type.FocusIn: + if not self._tree.currentIndex().isValid() and self._proxy.rowCount() > 0: + self._tree.setCurrentIndex(self._proxy.index(0, 0)) + return super().eventFilter(obj, event) + + def _update_selection(self, proxy_index: QModelIndex): + item = self._source_item(proxy_index) + if not item: + self._clear_preview() + self._select_button.setEnabled(False) + self._selected_path = None + return + + path = item.data(ROLE_PATH) + is_bundle = item.data(ROLE_IS_BUNDLE) + self._path_display.setText(path) + + if is_bundle: + self._selected_path = path + self._selected_is_s3 = self._radio_s3.isChecked() + self._selected_is_archive = bool(item.data(ROLE_IS_ARCHIVE)) + self._select_button.setEnabled(True) + if path != self._last_preview_path: + self._load_preview(path, item) + else: + self._selected_path = None + self._select_button.setEnabled(False) + self._selected_is_archive = False + self._clear_preview() + # Auto-expand folders when clicked — clear filter first so children are visible + if self._filter_edit.text(): + folder_path = item.data(ROLE_PATH) + self._filter_edit.clear() + # Defer select+expand to after Qt processes the filter change + QTimer.singleShot(0, lambda p=folder_path: self._select_and_expand_path(p)) + else: + if not self._tree.isExpanded(proxy_index): + self._tree.expand(proxy_index) + + def _select_and_expand_path(self, path: str): + """Find an item by path in the proxy model, select it, and expand it.""" + proxy_index = self._find_proxy_index_by_path(path) + if proxy_index and proxy_index.isValid(): + self._tree.setCurrentIndex(proxy_index) + self._tree.expand(proxy_index) + self._tree.scrollTo(proxy_index, QTreeView.PositionAtTop) + + def _find_proxy_index_by_path(self, path: str) -> Optional[QModelIndex]: + """Walk the source model to find an item by ROLE_PATH, return its proxy index.""" + + def _search(parent_item): + for row in range(parent_item.rowCount()): + child = parent_item.child(row) + if child and child.data(ROLE_PATH) == path: + return self._proxy.mapFromSource(child.index()) + result = _search(child) + if result: + return result + return None + + return _search(self._model.invisibleRootItem()) + + def _on_source_changed(self, checked: bool): + if not self._ready: + return + # Save expanded paths for the current source before switching + self._save_tree_state() + + if self._radio_local.isChecked(): + self._current_repo = self._local_repo + elif self._radio_s3.isChecked() and self._s3_repo: + self._current_repo = self._s3_repo + elif self._radio_s3.isChecked() and self._s3_loading: + self._current_repo = None # Will be populated when set_queue_source arrives + elif self._radio_history.isChecked() and self._history_repo: + self._current_repo = self._history_repo + self._selected_path = None + self._select_button.setEnabled(False) + self._clear_preview() + + # For S3, refresh in background to avoid blocking the UI + if isinstance(self._current_repo, S3BundleRepository): + self._model.clear() + self._model.setHorizontalHeaderLabels([tr("Name")]) + self._path_display.setText(self._current_repo.root_path()) + self._tree_empty_label.setText("Loading...") + self._tree_empty_label.setVisible(True) + self._refresh_s3_async() + else: + self._populate_root() + + # Restore expanded paths for the new source + self._restore_tree_state() + + def _save_tree_state(self): + """Save expanded folder paths and selection for the current source.""" + if self._current_repo is None: + return + key = self._current_repo.root_path() + + # Save selection + if self._selected_path: + self._tree_selections[key] = self._selected_path + elif key in self._tree_selections: + del self._tree_selections[key] + + # Save expanded folders + expanded: set[str] = set() + + def _collect(parent_index): + for row in range(self._proxy.rowCount(parent_index)): + idx = self._proxy.index(row, 0, parent_index) + if self._tree.isExpanded(idx): + source_idx = self._proxy.mapToSource(idx) + item = self._model.itemFromIndex(source_idx) + if item: + path = item.data(ROLE_PATH) + if path: + expanded.add(path) + _collect(idx) + + _collect(self._tree.rootIndex()) + key = self._current_repo.root_path() + if expanded: + self._tree_states[key] = expanded + elif key in self._tree_states: + del self._tree_states[key] + + def _restore_tree_state(self): + """Restore previously expanded folders and selection for the current source.""" + if self._current_repo is None: + return + key = self._current_repo.root_path() + expanded = self._tree_states.get(key) + if expanded: + + def _expand(parent_index): + for row in range(self._proxy.rowCount(parent_index)): + idx = self._proxy.index(row, 0, parent_index) + source_idx = self._proxy.mapToSource(idx) + item = self._model.itemFromIndex(source_idx) + if item and item.data(ROLE_PATH) in expanded: + self._tree.expand(idx) + _expand(idx) + + _expand(self._tree.rootIndex()) + + # Restore selection + saved_path = self._tree_selections.get(key) + if saved_path: + proxy_index = self._find_proxy_index_by_path(saved_path) + if proxy_index and proxy_index.isValid(): + self._tree.setCurrentIndex(proxy_index) + self._tree.scrollTo(proxy_index) + + def _on_hidden_toggled(self, checked: bool): + if not self._ready: + return + self._proxy.set_show_hidden(checked) + + def _on_context_menu(self, position): + """Show hide/unhide context menu for Queue source bundles.""" + if not isinstance(self._current_repo, S3BundleRepository): + return + proxy_index = self._tree.indexAt(position) + if not proxy_index.isValid(): + return + source_index = self._proxy.mapToSource(proxy_index) + item = self._model.itemFromIndex(source_index) + if not item or not item.data(ROLE_IS_BUNDLE): + return + + path = item.data(ROLE_PATH) + name = path.rsplit("/", 1)[-1] + if name.endswith(".ojd"): + name = name[:-4] + is_hidden = bool(item.data(ROLE_IS_HIDDEN)) + + menu = QMenu(self) + if is_hidden: + action = menu.addAction(tr("Unhide bundle")) + else: + action = menu.addAction(tr("Hide bundle")) + + chosen = menu.exec_(self._tree.viewport().mapToGlobal(position)) + if chosen != action: + return + + try: + self._current_repo.set_bundle_visibility(name, hidden=not is_hidden) + item.setData(not is_hidden, ROLE_IS_HIDDEN) + self._apply_hidden_style(item, not is_hidden) + if not is_hidden: + self._hidden_set.add(name) + else: + self._hidden_set.discard(name) + self._proxy.invalidateFilter() + except Exception as e: + logger.warning("Failed to update visibility: %s", e, exc_info=True) + self._show_error_preview( + f"\u26a0 Could not {'hide' if not is_hidden else 'unhide'} bundle: {e}" + ) + + # ── Preview ────────────────────────────────────────────────── + + def _load_preview(self, path: str, item: Optional[QStandardItem] = None): + self._last_preview_path = path + try: + info = self._current_repo.get_bundle_info(path) + except Exception as e: + logger.warning("Failed to load bundle info for %s: %s", path, e, exc_info=True) + self._show_error_preview(f"Failed to load bundle info:\n{e}") + if item: + self._mark_item_error(item) + self._select_button.setEnabled(False) + return + if not info: + self._show_error_preview( + "Could not read bundle template.\nThe template may be missing or malformed." + ) + if item: + self._mark_item_error(item) + self._select_button.setEnabled(False) + return + + self._preview_stack.setCurrentIndex(1) # show detail page + # A real bundle is previewed — offer to open it. Queue bundles are fetched + # over the network, so label it "Download bundle" and show the size (from + # the preview's head_object) so the user sees how much will transfer. + # Local/History bundles open in place, so label it "Open bundle" (no size). + self._download_button.setVisible(True) + if self._radio_s3.isChecked(): + if info.size_bytes: + self._download_button.setText( + f"{tr('Download bundle')} ({human_readable_file_size(info.size_bytes)})" + ) + else: + self._download_button.setText(tr("Download bundle")) + else: + self._download_button.setText(tr("Open bundle")) + # Plain text (see label setup): the name is shown literally, so crafted + # metadata markup is inert without needing to escape it. + self._preview_name.setText(info.name) + self._preview_name.setStyleSheet("font-weight: bold; font-size: 18px;") + self._preview_name.setVisible(True) + + # Subline: bundle type + source + type_label = tr("Archive") if self._selected_is_archive else tr("Folder") + type_icon = "\U0001f4e6" # \ud83d\udce6 + if self._radio_s3.isChecked(): + source_label = tr("Queue") + elif self._radio_history.isChecked(): + source_label = tr("History") + else: + source_label = tr("Local") + self._preview_subline.setText(f"{type_icon} {type_label} \u00b7 {source_label}") + self._preview_subline.setVisible(True) + + if info.description: + self._desc_section.set_title(tr("Description")) + self._desc_section.setVisible(True) + self._preview_desc.setText(_normalize_description(info.description)) + self._preview_desc.setVisible(self._desc_section.is_expanded()) + else: + self._desc_section.setVisible(False) + + if info.step_names: + # Detect if steps were truncated in S3 metadata + steps_truncated = info.step_names and info.step_names[-1].endswith("...") + steps = info.step_names[:-1] if steps_truncated else info.step_names + # Use total_steps from metadata count if available + if info.total_steps and info.total_steps > len(steps): + count_str = str(info.total_steps) + elif steps_truncated: + count_str = f"{len(steps)}+" + else: + count_str = str(len(steps)) + self._steps_section.set_title(f"{tr('Steps')} ({count_str})") + self._steps_section.setVisible(True) + steps_text = _steps_list_text(steps) + if steps_truncated or (info.total_steps and info.total_steps > len(steps)): + hidden_count = (info.total_steps - len(steps)) if info.total_steps else None + if hidden_count: + steps_text += f"\n \u2026 {hidden_count} more" + else: + steps_text += "\n \u2026 more" + self._preview_steps.setText(steps_text) + self._preview_steps.setVisible(self._steps_section.is_expanded()) + else: + self._steps_section.setVisible(False) + self._preview_steps.setVisible(False) + + if info.parameters: + muted_color = self._preview_params.palette().color(QPalette.PlaceholderText) + # Detect if parameters were truncated in metadata + garbled = any( + p.get("name", "").endswith("...") or p.get("type", "").endswith("...") + for p in info.parameters + ) + # Drop the last entry if it's garbled from truncation + params = info.parameters[:-1] if garbled else info.parameters + # Truncation: either garbled suffix detected, or count metadata says more exist + truncated = garbled or ( + info.total_parameters is not None and info.total_parameters > len(params) + ) + # Use total_parameters from metadata count if available + if info.total_parameters and info.total_parameters > len(params): + count_str = str(info.total_parameters) + elif truncated: + count_str = f"{len(params)}+" + else: + count_str = str(len(params)) + self._params_section.set_title(f"{tr('Parameters')} ({count_str})") + self._params_section.setVisible(True) + row_count = len(params) + (1 if truncated else 0) + self._preview_params.setRowCount(row_count) + for row, p in enumerate(params): + value = p.get("_display_value", "") + # "Required" = the artist must supply it because the bundle gives no + # default/value. Only meaningful for full template info (not S3 metadata). + is_required = not value and "default" not in p and "value" not in p + + # Name is plain; the "(required)" value cell carries the signal. + self._preview_params.setItem(row, 0, QTableWidgetItem(p.get("name", "?"))) + + self._preview_params.setItem( + row, 1, QTableWidgetItem(_friendly_param_type(p.get("type", ""))) + ) + + if value: + value_item = QTableWidgetItem(str(value)) + elif p.get("_from_metadata"): + value_item = QTableWidgetItem("") + elif is_required: + value_item = QTableWidgetItem(tr("(required)")) + value_item.setForeground(REQUIRED_COLOR) # warm = needs input + else: + value_item = QTableWidgetItem(tr("(no default)")) + value_item.setForeground(muted_color) + self._preview_params.setItem(row, 2, value_item) + if truncated: + hidden_count = ( + (info.total_parameters - len(params)) if info.total_parameters else None + ) + if hidden_count: + msg = f"\u2026 {hidden_count} more" + else: + msg = "\u2026 more" + truncation_item = QTableWidgetItem(msg) + truncation_item.setForeground(muted_color) + self._preview_params.setItem(len(params), 0, truncation_item) + self._preview_params.setSpan(len(params), 0, 1, 3) + self._preview_params.setVisible(self._params_section.is_expanded()) + self._size_params_table_to_contents() + else: + self._params_section.setVisible(False) + self._preview_params.setVisible(False) + + def _size_params_table_to_contents(self) -> None: + """Fix the table's height to exactly its rows + header so it doesn't leave + a large empty body, and cap the Name column so a long name can't squeeze + the Value column. Value wraps within its remaining width.""" + # Cap the Name column at ~40% of the table width so it can't dominate. + table_width = self._preview_params.viewport().width() + if table_width > 0: + name_w = self._preview_params.columnWidth(0) + self._preview_params.setColumnWidth(0, min(name_w, int(table_width * 0.4))) + # Recompute row heights now that wrapping/column widths are settled. + self._preview_params.resizeRowsToContents() + total = self._preview_params.horizontalHeader().height() + for row in range(self._preview_params.rowCount()): + total += self._preview_params.rowHeight(row) + # +2 for the frame border + self._preview_params.setFixedHeight(total + 2) + + @staticmethod + def _make_opacity(value: float) -> QGraphicsOpacityEffect: + """Opacity effect for a widget (QLabel CSS 'opacity' has no effect).""" + eff = QGraphicsOpacityEffect() + eff.setOpacity(value) + return eff + + def _clear_preview(self): + # Switch to the centered empty-state page. (The empty prompt is its own + # widget, so it can't be polluted by the error state's red styling.) + self._last_preview_path = None + self._preview_stack.setCurrentIndex(0) + + def _mark_item_error(self, item: QStandardItem) -> None: + """Replace the bundle/folder icon with a warning icon.""" + text = item.text() + # Remove existing icon prefix + for prefix in ("\U0001f4e6 ", "\U0001f4c1 ", "\u26a0 "): + if text.startswith(prefix): + text = text[len(prefix) :] + break + item.setText(f"\u26a0 {text}") + + def _show_error_preview(self, message: str): + """Show an error message in the preview panel.""" + self._preview_stack.setCurrentIndex(1) # show detail page + # Nothing valid to inspect in an error state. + self._download_button.setVisible(False) + self._preview_name.setText("\u26a0 Error") + self._preview_name.setStyleSheet("font-weight: bold; font-size: 14px; color: red;") + self._preview_name.setVisible(True) + self._preview_subline.setVisible(False) + self._desc_section.setVisible(False) + self._preview_desc.setText(message) + self._preview_desc.setVisible(True) + self._steps_section.setVisible(False) + self._preview_steps.setVisible(False) + self._params_section.setVisible(False) + self._preview_params.setVisible(False) diff --git a/src/deadline/client/ui/dialogs/submit_job_to_deadline_dialog.py b/src/deadline/client/ui/dialogs/submit_job_to_deadline_dialog.py index b4117b014..398a79955 100644 --- a/src/deadline/client/ui/dialogs/submit_job_to_deadline_dialog.py +++ b/src/deadline/client/ui/dialogs/submit_job_to_deadline_dialog.py @@ -5,21 +5,23 @@ from __future__ import annotations +import json import logging import os -import sys -import json +import shutil from typing import Any, Dict, Optional, Protocol import yaml -from qtpy.QtCore import QSize, Qt, Signal as _Signal # pylint: disable=import-error +from qtpy.QtCore import QSize, Qt, QThread, QTimer, Signal as _Signal # pylint: disable=import-error from qtpy.QtGui import QKeyEvent # pylint: disable=import-error from qtpy.QtWidgets import ( # pylint: disable=import-error; type: ignore QApplication, QDialog, QDialogButtonBox, QFormLayout, + QLabel, QMessageBox, + QProgressBar, QPushButton, QScrollArea, QTabWidget, @@ -42,6 +44,12 @@ from ...job_bundle import create_job_history_bundle_dir from ...job_bundle.parameters import JobParameter from ...job_bundle.submission import AssetReferences +from ...job_bundle.repository import ( + S3BundleRepository, + archive_bundle_dir, + get_bundle_dir_size, + sanitize_bundle_name, +) from ..widgets.deadline_authentication_status_widget import DeadlineAuthenticationStatusWidget from ..widgets.job_attachments_tab import JobAttachmentsWidget from ..widgets.shared_job_settings_tab import SharedJobSettingsWidget @@ -49,9 +57,11 @@ from . import DeadlineConfigDialog, DeadlineLoginDialog from ._types import JobBundlePurpose from ._help_dialog import _HelpDialog +from .export_bundle_dialog import ExportBundleDialog logger = logging.getLogger(__name__) + # initialize early so once the UI opens, things are already initialized DeadlineAuthenticationStatus.getInstance() @@ -263,7 +273,7 @@ def _build_ui( self.load_bundle_button = QPushButton(tr("Load Bundle")) self.load_bundle_button.clicked.connect(self._on_load_bundle) self.button_box.addButton(self.load_bundle_button, QDialogButtonBox.AcceptRole) - self.export_bundle_button = QPushButton(tr("Export bundle")) + self.export_bundle_button = QPushButton(tr("Save bundle as")) self.export_bundle_button.clicked.connect(self.on_export_bundle) self.button_box.addButton(self.export_bundle_button, QDialogButtonBox.AcceptRole) @@ -539,86 +549,297 @@ def _on_help_button_clicked(self): def _on_load_bundle(self): """Delegates to the job_settings widget's on_load_bundle method.""" if hasattr(self.job_settings, "on_load_bundle"): - self.job_settings.on_load_bundle() + self.job_settings.on_load_bundle(s3_repo=getattr(self, "_s3_repo", None)) def on_export_bundle(self): - """ - Exports a Job Bundle, but does not submit the job. - """ - # Retrieve all the settings into the dataclass + """Export a job bundle to Queue (S3) or a local directory.""" + # Gather settings settings = self.job_settings_type() self.shared_job_settings.update_settings(settings) self.job_settings.update_settings(settings) - queue_parameters = self.shared_job_settings.get_parameters() - - asset_references = self.job_attachments.get_asset_references() + # Default export name is the bundle directory name on disk + resolved_name = ( + os.path.basename(settings.input_job_bundle_dir) + if settings.input_job_bundle_dir + else settings.name + ) - # Save the bundle + # Try to get queue repo for the dialog + queue_repo = None + queue_error = "" try: - self.job_history_bundle_dir = create_job_history_bundle_dir( - self.submitter_info.submitter_name, settings.name - ) + queue_repo = S3BundleRepository.from_config() + except Exception as e: + queue_error = str(e) - if self.show_host_requirements_tab: - host_requirements = self.host_requirements.get_requirements() - parameters_from_callback = self.on_create_job_bundle_callback( - self, - self.job_history_bundle_dir, - settings, - queue_parameters, - asset_references, - host_requirements, - purpose=JobBundlePurpose.EXPORT, - ) - else: - # Maintaining backward compatibility for submitters that do not support host_requirements yet - parameters_from_callback = self.on_create_job_bundle_callback( + # Get default local directory + local_dir = get_setting("settings.job_bundle_default_directory") + if local_dir: + local_dir = os.path.expanduser(local_dir) + else: + local_dir = os.path.expanduser("~") + + # Show export dialog + dialog = ExportBundleDialog( + default_name=resolved_name, + queue_repo=queue_repo, + queue_error=queue_error, + local_dir=local_dir, + parent=self, + ) + if dialog.exec_() != ExportBundleDialog.Accepted or not dialog.bundle_name: + return + + bundle_name = sanitize_bundle_name(dialog.bundle_name) + + # Generate the bundle with current edits applied + import tempfile + + asset_references = self.job_attachments.get_asset_references() + queue_parameters = self.shared_job_settings.get_parameters() + requirements = ( + self.host_requirements.get_requirements() if self.show_host_requirements_tab else None + ) + + if dialog.export_to_queue: + with tempfile.TemporaryDirectory() as export_dir: + if not self._generate_export_bundle( + export_dir, settings, queue_parameters, asset_references, requirements + ): + # Generation failed and the user was already notified. Abort + # rather than silently uploading the original, un-edited + # bundle (which would discard the user's in-dialog edits) or + # crashing in the upload worker with no bundle to archive. + return + self._export_to_queue(queue_repo, bundle_name, export_dir) + else: + dest_path = os.path.join(dialog.local_directory, bundle_name) + if os.path.exists(dest_path): + reply = QMessageBox.question( self, - self.job_history_bundle_dir, - settings, - queue_parameters, - asset_references, - purpose=JobBundlePurpose.EXPORT, + tr("Save bundle as"), + f"Bundle '{bundle_name}' already exists at:\n{dest_path}\n\nOverwrite?", + QMessageBox.Yes | QMessageBox.No, + QMessageBox.No, ) - if parameters_from_callback is None: - parameters_from_callback = {} + if reply != QMessageBox.Yes: + return + shutil.rmtree(dest_path) + if not self._generate_export_bundle( + dest_path, settings, queue_parameters, asset_references, requirements + ): + return + QMessageBox.information( + self, + tr("Save bundle as"), + f"Bundle saved to:\n{dest_path}", + ) - # If the callback returned job parameters, update them in the job bundle as well so that - # submission from the job history dir is equivalent. - job_parameters = parameters_from_callback.get("job_parameters", []) - if job_parameters: - self.save_job_parameters_to_job_bundle(self.job_history_bundle_dir, job_parameters) + def _generate_export_bundle( + self, + output_dir: str, + settings, + queue_parameters: list[JobParameter], + asset_references: AssetReferences, + requirements: Optional[Dict[str, Any]], + ) -> bool: + """Generate the job bundle (with the dialog's current edits) into ``output_dir``. - logger.info(f"Saved the submission as a job bundle: {self.job_history_bundle_dir}") - if sys.platform == "win32": - # Open the directory in the OS's file explorer - os.startfile(self.job_history_bundle_dir) - QMessageBox.information( + Returns ``True`` on success. On failure, shows an error dialog and + returns ``False`` so callers can abort instead of proceeding with a + stale, un-edited, or missing bundle. + """ + try: + self.on_create_job_bundle_callback( self, - tr("{submitter} job submission").format( - submitter=self.submitter_info.submitter_name - ), - tr("Saved the submission as a job bundle:\n{path}").format( - path=self.job_history_bundle_dir - ), + output_dir, + settings, + queue_parameters, + asset_references, + requirements, + purpose=JobBundlePurpose.EXPORT, ) - # Close the submitter window to signal the submission is done - self.close() + return True + except Exception as exc: + logger.warning("Failed to generate bundle for export: %s", exc) + QMessageBox.critical(self, "Export failed", f"Failed to export bundle:\n{exc}") + return False - except NonValidInputError as nvie: - QMessageBox.critical(self, tr("Non valid inputs detected"), str(nvie)) + def _export_to_queue( + self, queue_repo: Optional[S3BundleRepository], bundle_name: str, source_dir: str + ): + """Archive and upload the bundle to the queue's S3 job-bundles folder.""" + from ...job_bundle.repository import build_bundle_metadata + + if not queue_repo: + QMessageBox.critical(self, "Export failed", "Queue is not available.") + return + + bundle_metadata = build_bundle_metadata(source_dir, bundle_name=bundle_name) + + # Archive and upload + try: + # Check if bundle already exists + if queue_repo.bundle_exists(bundle_name): + reply = QMessageBox.question( + self, + "Overwrite?", + f"Bundle '{bundle_name}' already exists on the queue. Overwrite?", + QMessageBox.Yes | QMessageBox.No, + QMessageBox.No, + ) + if reply != QMessageBox.Yes: + return + + # Archive and upload on a background thread with progress + class _UploadWorker(QThread): + progress = _Signal(int, int) # (current_bytes, total_bytes) + status = _Signal(str) + # NOTE: named ``done`` rather than ``finished`` to avoid shadowing + # QThread's built-in ``finished`` signal. + done = _Signal() + error = _Signal(str) + + def __init__(self, repo, bundle_name, source_dir, metadata): + super().__init__() + self._repo = repo + self._bundle_name = bundle_name + self._source_dir = source_dir + self._metadata = metadata + + def run(self): + try: + self.status.emit("Archiving bundle...") + total_size = get_bundle_dir_size(self._source_dir) + self.progress.emit(0, max(1, total_size // 1024)) + + archived = [0] + + def _on_archived(n): + archived[0] += n + self.progress.emit(archived[0] // 1024, 0) + + buf = archive_bundle_dir(self._source_dir, progress_callback=_on_archived) + + # archive_bundle_dir() returns the buffer already rewound + # to position 0, so buf.tell() would be 0 here. Use the + # buffer's byte length for the true archive size (matches + # the CLI upload path in bundle_group.py). + total = buf.getbuffer().nbytes + self.status.emit("Uploading bundle...") + self.progress.emit(0, max(1, total // 1024)) + + _sent = [0] + + def _upload_cb(n): + _sent[0] += n + self.progress.emit(_sent[0] // 1024, 0) + + self._repo.upload_archive( + buf, + self._bundle_name, + metadata=self._metadata, + progress_callback=_upload_cb, + ) + self.done.emit() + except Exception as e: + self.error.emit(str(e)) + + progress_dialog = QDialog(self) + progress_dialog.setWindowFlags(Qt.Dialog | Qt.CustomizeWindowHint | Qt.WindowTitleHint) + progress_dialog.setWindowTitle("Save Bundle to Queue") + progress_dialog.setWindowModality(Qt.ApplicationModal) + progress_dialog.setMinimumWidth(350) + _dlg_layout = QVBoxLayout(progress_dialog) + _progress_label = QLabel("Archiving bundle...") + _progress_label.setAlignment(Qt.AlignCenter) + _progress_bar = QProgressBar() + _progress_bar.setRange(0, 0) + _cancel_btn = QPushButton("Cancel") + _cancel_btn.clicked.connect(progress_dialog.reject) + _dlg_layout.addWidget(_progress_label) + _dlg_layout.addWidget(_progress_bar) + _dlg_layout.addWidget(_cancel_btn, alignment=Qt.AlignRight) + + worker = _UploadWorker( + queue_repo, + bundle_name, + source_dir, + bundle_metadata if bundle_metadata else None, + ) + upload_error = [] + _uploaded_bytes = [0] + _total_bytes = [0] + _phase = ["Archiving"] + _finished = [False] + + def _format_size(b): + if b >= 1024 * 1024 * 1024: + return f"{b / (1024**3):.1f} GB" + elif b >= 1024 * 1024: + return f"{b / (1024**2):.1f} MB" + elif b >= 1024: + return f"{b / 1024:.1f} KB" + return f"{b} B" + + def _on_status(msg): + _progress_label.setText(msg) + if "Upload" in msg: + _phase[0] = "Uploading" + + def _on_progress(n, total): + if _finished[0]: + return + if total > 0: + _total_bytes[0] = total * 1024 + _progress_bar.setMaximum(max(1, total)) + _progress_bar.setValue(0) + _uploaded_bytes[0] = 0 + else: + _uploaded_bytes[0] = n * 1024 + _progress_bar.setValue(n) + _progress_label.setText( + f"{_phase[0]} bundle... {_format_size(_uploaded_bytes[0])} / {_format_size(_total_bytes[0])}" + ) + + def _on_finished(): + _finished[0] = True + # Defer so queued progress signals are processed first + QTimer.singleShot(0, _show_complete) + + def _show_complete(): + _progress_bar.setVisible(False) + _progress_label.setText("\u2705 Bundle saved to queue") + _cancel_btn.setText("Close") + _cancel_btn.clicked.disconnect() + _cancel_btn.clicked.connect(progress_dialog.accept) + + def _on_error(msg): + upload_error.append(msg) + progress_dialog.close() + + worker.status.connect(_on_status, Qt.QueuedConnection) + worker.progress.connect(_on_progress, Qt.QueuedConnection) + worker.done.connect(_on_finished, Qt.QueuedConnection) + worker.error.connect(_on_error, Qt.QueuedConnection) + worker.start() + + progress_dialog.exec_() + worker.wait() + + if upload_error: + raise RuntimeError(upload_error[0]) except Exception as exc: - logger.exception("Error saving bundle") - message = str(exc) - QMessageBox.critical( - self, - tr("{submitter} job submission").format( - submitter=self.submitter_info.submitter_name - ), - message, - ) # type: ignore[call-arg] + from botocore.exceptions import ClientError + + logger.error("Failed to save bundle: %s", exc, exc_info=True) + if isinstance(exc, ClientError) and exc.response["Error"]["Code"] == "AccessDenied": + msg = "You don't have permission to share bundles on this queue." + else: + msg = f"Failed to upload bundle:\n{exc}" + QMessageBox.critical(self, "Export failed", msg) def save_job_parameters_to_job_bundle( self, job_bundle_dir: str, job_parameters: list[JobParameter] diff --git a/src/deadline/client/ui/job_bundle_submitter.py b/src/deadline/client/ui/job_bundle_submitter.py index ed1a66f04..979c86f20 100644 --- a/src/deadline/client/ui/job_bundle_submitter.py +++ b/src/deadline/client/ui/job_bundle_submitter.py @@ -6,11 +6,10 @@ from logging import getLogger from typing import Any, Optional, Dict -from qtpy.QtCore import Qt # pylint: disable=import-error +from qtpy.QtCore import Qt, QThread, Signal # pylint: disable=import-error from ._utils import tr from qtpy.QtWidgets import ( # pylint: disable=import-error; type: ignore QApplication, - QFileDialog, QMainWindow, QMessageBox, QWidget, @@ -32,6 +31,7 @@ validate_directory_symlink_containment, ) from ..job_bundle.saver import save_yaml_or_json_to_file +from ..job_bundle.repository import S3BundleRepository from ..job_bundle.parameters import ( JobParameter, apply_job_parameters, @@ -48,6 +48,8 @@ from .widgets.job_bundle_settings_tab import JobBundleSettingsWidget from ..job_bundle.submission import AssetReferences from ..api._session import session_context +from ..config import get_setting +from .dialogs.job_bundle_browser_dialog import JobBundleBrowserDialog logger = getLogger(__name__) @@ -214,13 +216,74 @@ def show_job_bundle_submitter( if main_windows: parent = main_windows[0] + _s3_repo_for_reuse = None + if not input_job_bundle_dir: - input_job_bundle_dir = QFileDialog.getExistingDirectory( - parent, tr("Choose job bundle directory"), input_job_bundle_dir + # Start S3 initialization in background immediately (before any other work) + class _S3InitWorker(QThread): + # NOTE: named ``done`` rather than ``finished`` to avoid shadowing + # QThread's built-in ``finished`` signal. + done = Signal(object, str, list, set) # (repo, error, entries, hidden_set) + + def run(self): + try: + from concurrent.futures import ThreadPoolExecutor + + repo = S3BundleRepository.from_config() + with ThreadPoolExecutor(max_workers=2) as ex: + entries_f = ex.submit(repo.list_entries, repo.root_path()) + hidden_f = ex.submit(repo.get_hidden_set) + entries = entries_f.result() + hidden = hidden_f.result() + self.done.emit(repo, "", entries, hidden) + except Exception as e: + logger.debug( + "Could not retrieve queue settings for bundle browser", exc_info=True + ) + self.done.emit(None, str(e), [], set()) + + # Config + dialog setup on the main thread. + default_dir = get_setting("settings.job_bundle_default_directory") + if default_dir: + default_dir = os.path.expanduser(default_dir) + + # Get the job history directory for the current profile + job_history_dir = os.path.expanduser(get_setting("settings.job_history_dir")) + + # Show the browser immediately — Queue source will populate when ready + browser = JobBundleBrowserDialog( + queue_source=None, + queue_error="", + queue_loading=True, + local_source=default_dir, + history_source=job_history_dir, + parent=parent, ) - if not input_job_bundle_dir: + # Connect the worker's result BEFORE starting it. A fast failure (e.g. not + # logged in) can emit ``done`` almost immediately; a cross-thread queued + # signal emitted before the connection exists is dropped, which would leave + # the browser stuck on "Loading..." forever. The worker still runs + # concurrently with the dialog's event loop. + s3_worker = _S3InitWorker() + s3_worker.done.connect(browser.set_queue_source) + s3_worker.start() + if browser.exec_() != JobBundleBrowserDialog.Accepted or not browser.selected_path: + s3_worker.wait() return None + s3_worker.wait() + + _s3_repo_for_reuse = browser.s3_repo + + browser.hide() + input_job_bundle_dir = browser.resolve_selection() + while not input_job_bundle_dir: + browser.show() + if browser.exec_() != JobBundleBrowserDialog.Accepted or not browser.selected_path: + return None + browser.hide() + input_job_bundle_dir = browser.resolve_selection() + def on_create_job_bundle_callback( widget: SubmitJobToDeadlineDialog, job_bundle_dir: str, @@ -439,6 +502,9 @@ def on_create_job_bundle_callback( known_asset_paths=known_asset_paths, ) + # Store S3 repo for reuse by "Load Bundle" button (avoids re-creating from scratch) + submitter_dialog._s3_repo = _s3_repo_for_reuse + if job_parameters: # We want to validate the job parameters after the queue parameters are loaded. # Connect a parameter validation function to the queue parameter loading completion diff --git a/src/deadline/client/ui/translations/locales/de_DE.json b/src/deadline/client/ui/translations/locales/de_DE.json index a705baa32..48df59c6c 100644 --- a/src/deadline/client/ui/translations/locales/de_DE.json +++ b/src/deadline/client/ui/translations/locales/de_DE.json @@ -46,7 +46,7 @@ "Download installer": "Installer herunterladen", "Edit...": "Bearbeiten...", "Error: Timeout cannot be set to zero.": "Fehler: Timeout kann nicht auf null gesetzt werden.", - "Export bundle": "Paket exportieren", + "Save bundle as": "Bundle speichern unter", "Failed to log in to AWS Deadline Cloud:

{error}": "Anmeldung bei AWS Deadline Cloud fehlgeschlagen:

{error}", "Farm": "Farm", "Farm settings": "Farm-Einstellungen", @@ -58,6 +58,8 @@ "Hardware requirements": "Hardwareanforderungen", "Hashing progress": "Hashing-Fortschritt", "Help": "Hilfe", + "Hide bundle": "Bundle ausblenden", + "Unhide bundle": "Bundle einblenden", "Host requirements": "Host-Anforderungen", "Initial state": "Anfangszustand", "Issue With Profile Configuration": "Problem mit Profilkonfiguration", @@ -72,6 +74,8 @@ "Language": "Sprache", "Language will change next time the submitter is opened": "Die Sprache wird beim nächsten Öffnen des Submitters geändert", "Load Bundle": "Paket laden", + "Download bundle": "Paket herunterladen", + "Open bundle": "Paket öffnen", "Log in": "Anmelden", "Log in to AWS Deadline Cloud": "Bei AWS Deadline Cloud anmelden", "Logging you in...": "Sie werden angemeldet...", @@ -112,6 +116,7 @@ "Settings...": "Einstellungen...", "Shared job settings": "Gemeinsame Jobeinstellungen", "Show auto-detected": "Automatisch erkannte anzeigen", + "Show hidden": "Versteckte anzeigen", "Show submitter update notifications": "Aktualisierungsbenachrichtigungen des Submitters anzeigen", "Specify a job bundle directory or run the bundle command with the --browse flag": "Geben Sie ein Jobpaket-Verzeichnis an oder führen Sie den Bundle-Befehl mit dem Flag --browse aus", "Specify output directories": "Ausgabeverzeichnisse angeben", @@ -139,7 +144,33 @@ "Value(s)": "Wert(e)", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "Sie sind mit dem Profil '{profile}' authentifiziert, aber dieses Profil kann AWS Deadline Cloud ListFarms nicht aufrufen und keine Jobs an AWS Deadline Cloud übermitteln.\n\nSo beheben Sie dieses Problem:\n\\u2022 Überprüfen Sie, dass keine Umgebungsvariablen auf die falsche AWS-Region verweisen (z. B. AWS_DEFAULT_REGION)\n\\u2022 Wenn Sie kein Deadline Cloud Monitor-Profil verwenden, überprüfen Sie, ob das Profil Berechtigungen für diese AWS Deadline Cloud-APIs hat, die für die Übermittlung erforderlich sind:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - Sie sind abgemeldet.", + "{profile} doesn't have access permissions to submit a job.": "{profile} hat keine Zugriffsberechtigungen zum Übermitteln eines Jobs.", "{submitter} job submission": "{submitter}-Jobübermittlung", "Default maximum retries per task": "Standardm\u00e4\u00dfige maximale Wiederholungen pro Aufgabe", - "Default maximum failed tasks count": "Standardm\u00e4\u00dfige maximale Anzahl fehlgeschlagener Aufgaben" -} \ No newline at end of file + "Default maximum failed tasks count": "Standardm\u00e4\u00dfige maximale Anzahl fehlgeschlagener Aufgaben", + "Browse Job Bundles": "Job-Bundles durchsuchen", + "History": "Verlauf", + "Job bundle directory": "Job-Bundle-Verzeichnis", + "Local": "Lokal", + "Name": "Name", + "Parameters:": "Parameter:", + "Path:": "Pfad:", + "Select": "Auswählen", + "Select a job bundle to see details": "Job-Bundle auswählen, um Details anzuzeigen", + "Source:": "Quelle:", + "Steps:": "Schritte:", + "(no default)": "(kein Standard)", + "(required)": "(erforderlich)", + "Archive": "Archiv", + "Choose one from the list to preview its details": "Wählen Sie aus der Liste, um Details anzuzeigen", + "Filter bundles...": "Bundles filtern...", + "Folder": "Ordner", + "Parameters": "Parameter", + "Select a job bundle": "Job-Bundle auswählen", + "Steps": "Schritte", + "Type": "Typ", + "Value": "Wert", + "Save to": "Speichern in", + "Location": "Speicherort", + "Select directory": "Verzeichnis auswählen" +} diff --git a/src/deadline/client/ui/translations/locales/en_US.json b/src/deadline/client/ui/translations/locales/en_US.json index c1322fc04..6e1c83722 100644 --- a/src/deadline/client/ui/translations/locales/en_US.json +++ b/src/deadline/client/ui/translations/locales/en_US.json @@ -46,7 +46,7 @@ "Download installer": "Download installer", "Edit...": "Edit...", "Error: Timeout cannot be set to zero.": "Error: Timeout cannot be set to zero.", - "Export bundle": "Export bundle", + "Save bundle as": "Save bundle as", "Failed to log in to AWS Deadline Cloud:

{error}": "Failed to log in to AWS Deadline Cloud:

{error}", "Farm": "Farm", "Farm settings": "Farm settings", @@ -58,6 +58,7 @@ "Hardware requirements": "Hardware requirements", "Hashing progress": "Hashing progress", "Help": "Help", + "Hide bundle": "Hide bundle", "Host requirements": "Host requirements", "Initial state": "Initial state", "Issue With Profile Configuration": "Issue With Profile Configuration", @@ -112,6 +113,7 @@ "Settings...": "Settings...", "Shared job settings": "Shared job settings", "Show auto-detected": "Show auto-detected", + "Show hidden": "Show hidden", "Show submitter update notifications": "Show submitter update notifications", "Specify a job bundle directory or run the bundle command with the --browse flag": "Specify a job bundle directory or run the bundle command with the --browse flag", "Specify output directories": "Specify output directories", @@ -133,13 +135,42 @@ "Unknown Issue With Configured Profile": "Unknown Issue With Configured Profile", "Version {latest_version} of Deadline Cloud for {integration_name} submitter is now available.": "Version {latest_version} of Deadline Cloud for {integration_name} submitter is now available.", "View release notes": "View release notes", + "Unhide bundle": "Unhide bundle", "Unrecognized Parameters": "Unrecognized Parameters", "Upload progress": "Upload progress", "Use array parameter": "Use array parameter", "Value(s)": "Value(s)", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - You are logged out.", + "{profile} doesn't have access permissions to submit a job.": "{profile} doesn't have access permissions to submit a job.", "{submitter} job submission": "{submitter} job submission", "Default maximum retries per task": "Default maximum retries per task", - "Default maximum failed tasks count": "Default maximum failed tasks count" -} \ No newline at end of file + "Default maximum failed tasks count": "Default maximum failed tasks count", + "Browse Job Bundles": "Browse Job Bundles", + "History": "History", + "Download bundle": "Download bundle", + "Open bundle": "Open bundle", + "Job bundle directory": "Job bundle directory", + "Local": "Local", + "Name": "Name", + "Parameters:": "Parameters:", + "Path:": "Path:", + "Select": "Select", + "Select a job bundle to see details": "Select a job bundle to see details", + "Source:": "Source:", + "Steps:": "Steps:", + "(no default)": "(no default)", + "(required)": "(required)", + "Archive": "Archive", + "Choose one from the list to preview its details": "Choose one from the list to preview its details", + "Filter bundles...": "Filter bundles...", + "Folder": "Folder", + "Parameters": "Parameters", + "Select a job bundle": "Select a job bundle", + "Steps": "Steps", + "Type": "Type", + "Value": "Value", + "Save to": "Save to", + "Location": "Location", + "Select directory": "Select directory" +} diff --git a/src/deadline/client/ui/translations/locales/es_ES.json b/src/deadline/client/ui/translations/locales/es_ES.json index 6b05b2ef9..9a27546e6 100644 --- a/src/deadline/client/ui/translations/locales/es_ES.json +++ b/src/deadline/client/ui/translations/locales/es_ES.json @@ -46,7 +46,7 @@ "Download installer": "Descargar instalador", "Edit...": "Editar...", "Error: Timeout cannot be set to zero.": "Error: El tiempo de espera no puede establecerse en cero.", - "Export bundle": "Exportar paquete", + "Save bundle as": "Guardar paquete como", "Failed to log in to AWS Deadline Cloud:

{error}": "Error al iniciar sesión en AWS Deadline Cloud:

{error}", "Farm": "Granja", "Farm settings": "Configuración de granja", @@ -58,6 +58,8 @@ "Hardware requirements": "Requisitos de hardware", "Hashing progress": "Progreso de hash", "Help": "Ayuda", + "Hide bundle": "Ocultar bundle", + "Unhide bundle": "Mostrar bundle", "Host requirements": "Requisitos de host", "Initial state": "Estado inicial", "Issue With Profile Configuration": "Problema con la configuración del perfil", @@ -72,6 +74,8 @@ "Language": "Idioma", "Language will change next time the submitter is opened": "El idioma cambiará la próxima vez que se abra el submitter", "Load Bundle": "Cargar paquete", + "Download bundle": "Descargar paquete", + "Open bundle": "Abrir paquete", "Log in": "Iniciar sesión", "Log in to AWS Deadline Cloud": "Iniciar sesión en AWS Deadline Cloud", "Logging you in...": "Iniciando sesión...", @@ -112,6 +116,7 @@ "Settings...": "Configuración...", "Shared job settings": "Configuración de trabajo compartida", "Show auto-detected": "Mostrar detectados automáticamente", + "Show hidden": "Mostrar ocultos", "Show submitter update notifications": "Mostrar notificaciones de actualización del remitente", "Specify a job bundle directory or run the bundle command with the --browse flag": "Especifique un directorio de paquete de trabajos o ejecute el comando bundle con la marca --browse", "Specify output directories": "Especificar directorios de salida", @@ -139,7 +144,33 @@ "Value(s)": "Valor(es)", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "Está autenticado con el perfil '{profile}', pero este perfil no puede llamar a AWS Deadline Cloud ListFarms y no puede enviar trabajos a AWS Deadline Cloud.\n\nPara resolver este problema:\n\\u2022 Compruebe que no haya variables de entorno que apunten a la región de AWS incorrecta (por ejemplo, AWS_DEFAULT_REGION)\n\\u2022 Si no está utilizando un perfil de Deadline Cloud Monitor, compruebe que el perfil tenga permisos para estas API de AWS Deadline Cloud necesarias para el envío:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - Ha cerrado sesión.", + "{profile} doesn't have access permissions to submit a job.": "{profile} no tiene permisos de acceso para enviar un trabajo.", "{submitter} job submission": "Envío de trabajo de {submitter}", "Default maximum retries per task": "Reintentos m\u00e1ximos por tarea predeterminados", - "Default maximum failed tasks count": "Recuento m\u00e1ximo de tareas fallidas predeterminado" -} \ No newline at end of file + "Default maximum failed tasks count": "Recuento m\u00e1ximo de tareas fallidas predeterminado", + "Browse Job Bundles": "Explorar paquetes de trabajos", + "History": "Historial", + "Job bundle directory": "Directorio de paquetes de trabajos", + "Local": "Local", + "Name": "Nombre", + "Parameters:": "Parámetros:", + "Path:": "Ruta:", + "Select": "Seleccionar", + "Select a job bundle to see details": "Seleccione un paquete de trabajo para ver los detalles", + "Source:": "Origen:", + "Steps:": "Pasos:", + "(no default)": "(sin valor predeterminado)", + "(required)": "(obligatorio)", + "Archive": "Archivo", + "Choose one from the list to preview its details": "Elija uno de la lista para ver sus detalles", + "Filter bundles...": "Filtrar bundles...", + "Folder": "Carpeta", + "Parameters": "Parámetros", + "Select a job bundle": "Seleccionar un job bundle", + "Steps": "Pasos", + "Type": "Tipo", + "Value": "Valor", + "Save to": "Guardar en", + "Location": "Ubicación", + "Select directory": "Seleccionar directorio" +} diff --git a/src/deadline/client/ui/translations/locales/fr_FR.json b/src/deadline/client/ui/translations/locales/fr_FR.json index 6cd7db751..03afbadd5 100644 --- a/src/deadline/client/ui/translations/locales/fr_FR.json +++ b/src/deadline/client/ui/translations/locales/fr_FR.json @@ -46,7 +46,7 @@ "Download installer": "Télécharger l'installateur", "Edit...": "Modifier...", "Error: Timeout cannot be set to zero.": "Erreur : Le délai d'expiration ne peut pas être défini sur zéro.", - "Export bundle": "Exporter le lot", + "Save bundle as": "Enregistrer le lot sous", "Failed to log in to AWS Deadline Cloud:

{error}": "Échec de la connexion à AWS Deadline Cloud :

{error}", "Farm": "Ferme", "Farm settings": "Paramètres de ferme", @@ -58,6 +58,8 @@ "Hardware requirements": "Exigences matérielles", "Hashing progress": "Progression du hachage", "Help": "Aide", + "Hide bundle": "Masquer le bundle", + "Unhide bundle": "Afficher le bundle", "Host requirements": "Exigences d'hôte", "Initial state": "État initial", "Issue With Profile Configuration": "Problème avec la configuration du profil", @@ -72,6 +74,8 @@ "Language": "Langue", "Language will change next time the submitter is opened": "La langue changera lors de la prochaine ouverture du submitter", "Load Bundle": "Charger le lot", + "Download bundle": "Télécharger le lot", + "Open bundle": "Ouvrir le lot", "Log in": "Se connecter", "Log in to AWS Deadline Cloud": "Se connecter à AWS Deadline Cloud", "Logging you in...": "Connexion en cours...", @@ -112,6 +116,7 @@ "Settings...": "Paramètres...", "Shared job settings": "Paramètres de tâche partagés", "Show auto-detected": "Afficher les éléments détectés automatiquement", + "Show hidden": "Afficher les éléments masqués", "Show submitter update notifications": "Afficher les notifications de mise à jour du soumetteur", "Specify a job bundle directory or run the bundle command with the --browse flag": "Spécifiez un répertoire de lot de tâches ou exécutez la commande bundle avec l'indicateur --browse", "Specify output directories": "Spécifier les répertoires de sortie", @@ -139,7 +144,33 @@ "Value(s)": "Valeur(s)", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "Vous êtes authentifié avec le profil '{profile}', mais ce profil ne peut pas appeler AWS Deadline Cloud ListFarms et ne peut pas soumettre de tâches à AWS Deadline Cloud.\n\nPour résoudre ce problème :\n\\u2022 Vérifiez qu'aucune variable d'environnement ne pointe vers la mauvaise région AWS (par exemple, AWS_DEFAULT_REGION)\n\\u2022 Si vous n'utilisez pas un profil Deadline Cloud Monitor, vérifiez que le profil dispose des autorisations pour ces API AWS Deadline Cloud nécessaires à la soumission :\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - Vous êtes déconnecté.", + "{profile} doesn't have access permissions to submit a job.": "{profile} n'a pas les autorisations d'accès pour soumettre une tâche.", "{submitter} job submission": "Soumission de tâche {submitter}", "Default maximum retries per task": "Nombre maximal de tentatives par t\u00e2che par d\u00e9faut", - "Default maximum failed tasks count": "Nombre maximal de t\u00e2ches \u00e9chou\u00e9es par d\u00e9faut" -} \ No newline at end of file + "Default maximum failed tasks count": "Nombre maximal de t\u00e2ches \u00e9chou\u00e9es par d\u00e9faut", + "Browse Job Bundles": "Parcourir les lots de tâches", + "History": "Historique", + "Job bundle directory": "Répertoire des lots de tâches", + "Local": "Local", + "Name": "Nom", + "Parameters:": "Paramètres :", + "Path:": "Chemin :", + "Select": "Sélectionner", + "Select a job bundle to see details": "Sélectionnez un lot de tâches pour voir les détails", + "Source:": "Source :", + "Steps:": "Étapes :", + "(no default)": "(pas de défaut)", + "(required)": "(obligatoire)", + "Archive": "Archive", + "Choose one from the list to preview its details": "Choisissez dans la liste pour afficher les détails", + "Filter bundles...": "Filtrer les bundles...", + "Folder": "Dossier", + "Parameters": "Paramètres", + "Select a job bundle": "Sélectionner un job bundle", + "Steps": "Étapes", + "Type": "Type", + "Value": "Valeur", + "Save to": "Enregistrer dans", + "Location": "Emplacement", + "Select directory": "Sélectionner le répertoire" +} diff --git a/src/deadline/client/ui/translations/locales/id_ID.json b/src/deadline/client/ui/translations/locales/id_ID.json index f475013b9..5a856e586 100644 --- a/src/deadline/client/ui/translations/locales/id_ID.json +++ b/src/deadline/client/ui/translations/locales/id_ID.json @@ -46,7 +46,7 @@ "Download installer": "Unduh installer", "Edit...": "Edit...", "Error: Timeout cannot be set to zero.": "Kesalahan: Timeout tidak dapat diatur ke nol.", - "Export bundle": "Ekspor bundel", + "Save bundle as": "Simpan bundel sebagai", "Failed to log in to AWS Deadline Cloud:

{error}": "Gagal masuk ke AWS Deadline Cloud:

{error}", "Farm": "Peternakan", "Farm settings": "Pengaturan peternakan", @@ -58,6 +58,8 @@ "Hardware requirements": "Persyaratan perangkat keras", "Hashing progress": "Kemajuan hashing", "Help": "Bantuan", + "Hide bundle": "Sembunyikan bundle", + "Unhide bundle": "Tampilkan bundle", "Host requirements": "Persyaratan host", "Initial state": "Status awal", "Issue With Profile Configuration": "Masalah dengan konfigurasi profil", @@ -72,6 +74,8 @@ "Language": "Bahasa", "Language will change next time the submitter is opened": "Bahasa akan berubah saat submitter dibuka kembali", "Load Bundle": "Muat bundel", + "Download bundle": "Unduh bundel", + "Open bundle": "Buka bundel", "Log in": "Masuk", "Log in to AWS Deadline Cloud": "Masuk ke AWS Deadline Cloud", "Logging you in...": "Memasukkan Anda...", @@ -112,6 +116,7 @@ "Settings...": "Pengaturan...", "Shared job settings": "Pengaturan pekerjaan bersama", "Show auto-detected": "Tampilkan yang terdeteksi otomatis", + "Show hidden": "Tampilkan tersembunyi", "Show submitter update notifications": "Tampilkan notifikasi pembaruan pengirim", "Specify a job bundle directory or run the bundle command with the --browse flag": "Tentukan direktori bundel pekerjaan atau jalankan perintah bundle dengan flag --browse", "Specify output directories": "Tentukan direktori output", @@ -139,7 +144,33 @@ "Value(s)": "Nilai", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "Anda diautentikasi dengan profil '{profile}', tetapi profil ini tidak dapat memanggil AWS Deadline Cloud ListFarms dan tidak dapat mengirim pekerjaan ke AWS Deadline Cloud.\n\nUntuk mengatasi masalah ini:\n\\u2022 Periksa bahwa tidak ada variabel lingkungan yang menunjuk ke wilayah AWS yang salah (misalnya, AWS_DEFAULT_REGION)\n\\u2022 Jika Anda tidak menggunakan profil Deadline Cloud Monitor, periksa bahwa profil memiliki izin untuk API AWS Deadline Cloud ini yang diperlukan untuk pengiriman:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - Anda telah keluar.", + "{profile} doesn't have access permissions to submit a job.": "{profile} tidak memiliki izin akses untuk mengirim pekerjaan.", "{submitter} job submission": "Pengiriman pekerjaan {submitter}", "Default maximum retries per task": "Percobaan ulang maksimum default per tugas", - "Default maximum failed tasks count": "Jumlah tugas gagal maksimum default" -} \ No newline at end of file + "Default maximum failed tasks count": "Jumlah tugas gagal maksimum default", + "Browse Job Bundles": "Jelajahi bundel tugas", + "History": "Riwayat", + "Job bundle directory": "Direktori bundel tugas", + "Local": "Lokal", + "Name": "Nama", + "Parameters:": "Parameter:", + "Path:": "Jalur:", + "Select": "Pilih", + "Select a job bundle to see details": "Pilih bundel tugas untuk melihat detail", + "Source:": "Sumber:", + "Steps:": "Langkah:", + "(no default)": "(tanpa default)", + "(required)": "(wajib)", + "Archive": "Arsip", + "Choose one from the list to preview its details": "Pilih satu dari daftar untuk melihat detailnya", + "Filter bundles...": "Filter bundel...", + "Folder": "Folder", + "Parameters": "Parameter", + "Select a job bundle": "Pilih job bundle", + "Steps": "Langkah", + "Type": "Tipe", + "Value": "Nilai", + "Save to": "Simpan ke", + "Location": "Lokasi", + "Select directory": "Pilih direktori" +} diff --git a/src/deadline/client/ui/translations/locales/it_IT.json b/src/deadline/client/ui/translations/locales/it_IT.json index d3d7b2495..7a4efd667 100644 --- a/src/deadline/client/ui/translations/locales/it_IT.json +++ b/src/deadline/client/ui/translations/locales/it_IT.json @@ -46,7 +46,7 @@ "Download installer": "Scarica installer", "Edit...": "Modifica...", "Error: Timeout cannot be set to zero.": "Errore: il timeout non può essere impostato su zero.", - "Export bundle": "Esporta pacchetto", + "Save bundle as": "Salva pacchetto come", "Failed to log in to AWS Deadline Cloud:

{error}": "Accesso ad AWS Deadline Cloud non riuscito:

{error}", "Farm": "Farm", "Farm settings": "Impostazioni farm", @@ -58,6 +58,8 @@ "Hardware requirements": "Requisiti hardware", "Hashing progress": "Avanzamento hashing", "Help": "Aiuto", + "Hide bundle": "Nascondi bundle", + "Unhide bundle": "Mostra bundle", "Host requirements": "Requisiti host", "Initial state": "Stato iniziale", "Issue With Profile Configuration": "Problema con la configurazione del profilo", @@ -72,6 +74,8 @@ "Language": "Lingua", "Language will change next time the submitter is opened": "La lingua cambierà alla prossima apertura del submitter", "Load Bundle": "Carica pacchetto", + "Download bundle": "Scarica pacchetto", + "Open bundle": "Apri pacchetto", "Log in": "Accedi", "Log in to AWS Deadline Cloud": "Accedi ad AWS Deadline Cloud", "Logging you in...": "Accesso in corso...", @@ -112,6 +116,7 @@ "Settings...": "Impostazioni...", "Shared job settings": "Impostazioni lavoro condivise", "Show auto-detected": "Mostra rilevati automaticamente", + "Show hidden": "Mostra nascosti", "Show submitter update notifications": "Mostra notifiche di aggiornamento del submitter", "Specify a job bundle directory or run the bundle command with the --browse flag": "Specifica una directory pacchetto lavoro o esegui il comando bundle con il flag --browse", "Specify output directories": "Specifica directory di output", @@ -139,7 +144,33 @@ "Value(s)": "Valore/i", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "Sei autenticato con il profilo '{profile}', ma questo profilo non è in grado di chiamare AWS Deadline Cloud ListFarms e non può inviare lavori ad AWS Deadline Cloud.\n\nPer risolvere questo problema:\n\\u2022 Verifica che non ci siano variabili d'ambiente che puntano alla regione AWS errata (ad es. AWS_DEFAULT_REGION)\n\\u2022 Se non stai utilizzando un profilo Deadline Cloud Monitor, verifica che il profilo disponga delle autorizzazioni per queste API AWS Deadline Cloud necessarie per l'invio:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - Hai effettuato il logout.", + "{profile} doesn't have access permissions to submit a job.": "{profile} non dispone delle autorizzazioni di accesso per inviare un lavoro.", "{submitter} job submission": "Invio lavoro {submitter}", "Default maximum retries per task": "Numero massimo predefinito di tentativi per attivit\u00e0", - "Default maximum failed tasks count": "Numero massimo predefinito di attivit\u00e0 non riuscite" -} \ No newline at end of file + "Default maximum failed tasks count": "Numero massimo predefinito di attivit\u00e0 non riuscite", + "Browse Job Bundles": "Sfoglia bundle di processi", + "History": "Cronologia", + "Job bundle directory": "Directory bundle di processi", + "Local": "Locale", + "Name": "Nome", + "Parameters:": "Parametri:", + "Path:": "Percorso:", + "Select": "Seleziona", + "Select a job bundle to see details": "Seleziona un bundle di processi per visualizzare i dettagli", + "Source:": "Origine:", + "Steps:": "Passaggi:", + "(no default)": "(nessun valore predefinito)", + "(required)": "(obbligatorio)", + "Archive": "Archivio", + "Choose one from the list to preview its details": "Scegline uno dall'elenco per visualizzarne i dettagli", + "Filter bundles...": "Filtra bundle...", + "Folder": "Cartella", + "Parameters": "Parametri", + "Select a job bundle": "Seleziona un job bundle", + "Steps": "Passaggi", + "Type": "Tipo", + "Value": "Valore", + "Save to": "Salva in", + "Location": "Posizione", + "Select directory": "Seleziona directory" +} diff --git a/src/deadline/client/ui/translations/locales/ja_JP.json b/src/deadline/client/ui/translations/locales/ja_JP.json index 9f7a0ee41..7350292b7 100644 --- a/src/deadline/client/ui/translations/locales/ja_JP.json +++ b/src/deadline/client/ui/translations/locales/ja_JP.json @@ -46,7 +46,7 @@ "Download installer": "インストーラーをダウンロード", "Edit...": "編集...", "Error: Timeout cannot be set to zero.": "エラー: タイムアウトをゼロに設定できません。", - "Export bundle": "バンドルをエクスポート", + "Save bundle as": "バンドルを保存", "Failed to log in to AWS Deadline Cloud:

{error}": "AWS Deadline Cloud へのログインに失敗しました:

{error}", "Farm": "ファーム", "Farm settings": "ファーム設定", @@ -58,6 +58,8 @@ "Hardware requirements": "ハードウェア要件", "Hashing progress": "ハッシュ進行状況", "Help": "ヘルプ", + "Hide bundle": "バンドルを非表示", + "Unhide bundle": "バンドルを表示", "Host requirements": "ホスト要件", "Initial state": "初期状態", "Issue With Profile Configuration": "プロファイル設定の問題", @@ -72,6 +74,8 @@ "Language": "言語", "Language will change next time the submitter is opened": "言語は次回サブミッターを開いたときに変更されます", "Load Bundle": "バンドルを読み込む", + "Download bundle": "バンドルをダウンロード", + "Open bundle": "バンドルを開く", "Log in": "ログイン", "Log in to AWS Deadline Cloud": "AWS Deadline Cloud にログイン", "Logging you in...": "ログイン中...", @@ -112,6 +116,7 @@ "Settings...": "設定...", "Shared job settings": "共有ジョブ設定", "Show auto-detected": "自動検出されたものを表示", + "Show hidden": "非表示を表示", "Show submitter update notifications": "サブミッターの更新通知を表示", "Specify a job bundle directory or run the bundle command with the --browse flag": "ジョブバンドルディレクトリを指定するか、--browse フラグを使用して bundle コマンドを実行してください", "Specify output directories": "出力ディレクトリを指定", @@ -139,7 +144,33 @@ "Value(s)": "値", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "プロファイル '{profile}' で認証されていますが、このプロファイルは AWS Deadline Cloud ListFarms を呼び出すことができず、AWS Deadline Cloud にジョブを送信できません。\n\nこの問題を解決するには:\n\\u2022 誤った AWS リージョンを指す環境変数 (AWS_DEFAULT_REGION など) がないことを確認してください\n\\u2022 Deadline Cloud Monitor プロファイルを使用していない場合は、プロファイルに送信に必要な次の AWS Deadline Cloud API のアクセス許可があることを確認してください:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - ログアウトしています。", + "{profile} doesn't have access permissions to submit a job.": "{profile} にはジョブを送信するアクセス許可がありません。", "{submitter} job submission": "{submitter} ジョブ送信", "Default maximum retries per task": "\u30bf\u30b9\u30af\u3042\u305f\u308a\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u6700\u5927\u518d\u8a66\u884c\u56de\u6570", - "Default maximum failed tasks count": "\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u5931\u6557\u3057\u305f\u30bf\u30b9\u30af\u306e\u6700\u5927\u6570" -} \ No newline at end of file + "Default maximum failed tasks count": "\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u5931\u6557\u3057\u305f\u30bf\u30b9\u30af\u306e\u6700\u5927\u6570", + "Browse Job Bundles": "ジョブバンドルを参照", + "History": "履歴", + "Job bundle directory": "ジョブバンドルディレクトリ", + "Local": "ローカル", + "Name": "名前", + "Parameters:": "パラメータ:", + "Path:": "パス:", + "Select": "選択", + "Select a job bundle to see details": "ジョブバンドルを選択して詳細を表示", + "Source:": "ソース:", + "Steps:": "ステップ:", + "(no default)": "(デフォルトなし)", + "(required)": "(必須)", + "Archive": "アーカイブ", + "Choose one from the list to preview its details": "リストから選択して詳細をプレビュー", + "Filter bundles...": "バンドルを検索...", + "Folder": "フォルダー", + "Parameters": "パラメータ", + "Select a job bundle": "ジョブバンドルを選択", + "Steps": "ステップ", + "Type": "タイプ", + "Value": "値", + "Save to": "保存先", + "Location": "場所", + "Select directory": "ディレクトリを選択" +} diff --git a/src/deadline/client/ui/translations/locales/ko_KR.json b/src/deadline/client/ui/translations/locales/ko_KR.json index 7d977edc2..6330a4902 100644 --- a/src/deadline/client/ui/translations/locales/ko_KR.json +++ b/src/deadline/client/ui/translations/locales/ko_KR.json @@ -46,7 +46,7 @@ "Download installer": "설치 프로그램 다운로드", "Edit...": "편집...", "Error: Timeout cannot be set to zero.": "오류: 제한 시간을 0으로 설정할 수 없습니다.", - "Export bundle": "번들 내보내기", + "Save bundle as": "번들 다른 이름으로 저장", "Failed to log in to AWS Deadline Cloud:

{error}": "AWS Deadline Cloud에 로그인하지 못했습니다:

{error}", "Farm": "팜", "Farm settings": "팜 설정", @@ -58,6 +58,8 @@ "Hardware requirements": "하드웨어 요구 사항", "Hashing progress": "해싱 진행률", "Help": "도움말", + "Hide bundle": "번들 숨기기", + "Unhide bundle": "번들 표시", "Host requirements": "호스트 요구 사항", "Initial state": "초기 상태", "Issue With Profile Configuration": "프로필 구성 문제", @@ -72,6 +74,8 @@ "Language": "언어", "Language will change next time the submitter is opened": "언어는 다음에 제출자를 열 때 변경됩니다", "Load Bundle": "번들 로드", + "Download bundle": "번들 다운로드", + "Open bundle": "번들 열기", "Log in": "로그인", "Log in to AWS Deadline Cloud": "AWS Deadline Cloud에 로그인", "Logging you in...": "로그인 중...", @@ -112,6 +116,7 @@ "Settings...": "설정...", "Shared job settings": "공유 작업 설정", "Show auto-detected": "자동 감지된 항목 표시", + "Show hidden": "숨겨진 항목 표시", "Show submitter update notifications": "제출기 업데이트 알림 표시", "Specify a job bundle directory or run the bundle command with the --browse flag": "작업 번들 디렉터리를 지정하거나 --browse 플래그와 함께 bundle 명령을 실행하세요", "Specify output directories": "출력 디렉터리 지정", @@ -139,7 +144,33 @@ "Value(s)": "값", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "프로필 '{profile}'로 인증되었지만 이 프로필은 AWS Deadline Cloud ListFarms를 호출할 수 없으며 AWS Deadline Cloud에 작업을 제출할 수 없습니다.\n\n이 문제를 해결하려면:\n\\u2022 잘못된 AWS 리전을 가리키는 환경 변수(예: AWS_DEFAULT_REGION)가 없는지 확인하세요\n\\u2022 Deadline Cloud Monitor 프로필을 사용하지 않는 경우 프로필에 제출에 필요한 다음 AWS Deadline Cloud API에 대한 권한이 있는지 확인하세요:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - 로그아웃되었습니다.", + "{profile} doesn't have access permissions to submit a job.": "{profile}에 작업을 제출할 액세스 권한이 없습니다.", "{submitter} job submission": "{submitter} 작업 제출", "Default maximum retries per task": "\uc791\uc5c5\ub2f9 \uae30\ubcf8 \ucd5c\ub300 \uc7ac\uc2dc\ub3c4 \ud69f\uc218", - "Default maximum failed tasks count": "\uae30\ubcf8 \ucd5c\ub300 \uc2e4\ud328 \uc791\uc5c5 \uc218" -} \ No newline at end of file + "Default maximum failed tasks count": "\uae30\ubcf8 \ucd5c\ub300 \uc2e4\ud328 \uc791\uc5c5 \uc218", + "Browse Job Bundles": "작업 번들 찾아보기", + "History": "기록", + "Job bundle directory": "작업 번들 디렉터리", + "Local": "로컬", + "Name": "이름", + "Parameters:": "파라미터:", + "Path:": "경로:", + "Select": "선택", + "Select a job bundle to see details": "작업 번들을 선택하여 세부 정보 보기", + "Source:": "소스:", + "Steps:": "단계:", + "(no default)": "(기본값 없음)", + "(required)": "(필수)", + "Archive": "아카이브", + "Choose one from the list to preview its details": "목록에서 선택하여 세부 정보를 미리 봅니다", + "Filter bundles...": "번들 필터...", + "Folder": "폴더", + "Parameters": "파라미터", + "Select a job bundle": "작업 번들 선택", + "Steps": "단계", + "Type": "유형", + "Value": "값", + "Save to": "저장 위치", + "Location": "위치", + "Select directory": "디렉터리 선택" +} diff --git a/src/deadline/client/ui/translations/locales/pt_BR.json b/src/deadline/client/ui/translations/locales/pt_BR.json index fb07e3127..d429a983e 100644 --- a/src/deadline/client/ui/translations/locales/pt_BR.json +++ b/src/deadline/client/ui/translations/locales/pt_BR.json @@ -46,7 +46,7 @@ "Download installer": "Baixar instalador", "Edit...": "Editar...", "Error: Timeout cannot be set to zero.": "Erro: o tempo limite não pode ser definido como zero.", - "Export bundle": "Exportar pacote", + "Save bundle as": "Salvar pacote como", "Failed to log in to AWS Deadline Cloud:

{error}": "Falha ao fazer login no AWS Deadline Cloud:

{error}", "Farm": "Fazenda", "Farm settings": "Configurações da fazenda", @@ -58,6 +58,8 @@ "Hardware requirements": "Requisitos de hardware", "Hashing progress": "Progresso de hash", "Help": "Ajuda", + "Hide bundle": "Ocultar bundle", + "Unhide bundle": "Exibir bundle", "Host requirements": "Requisitos de host", "Initial state": "Estado inicial", "Issue With Profile Configuration": "Problema com a configuração do perfil", @@ -72,6 +74,8 @@ "Language": "Idioma", "Language will change next time the submitter is opened": "O idioma será alterado na próxima vez que o submitter for aberto", "Load Bundle": "Carregar pacote", + "Download bundle": "Baixar pacote", + "Open bundle": "Abrir pacote", "Log in": "Fazer login", "Log in to AWS Deadline Cloud": "Fazer login no AWS Deadline Cloud", "Logging you in...": "Fazendo login...", @@ -112,6 +116,7 @@ "Settings...": "Configurações...", "Shared job settings": "Configurações de trabalho compartilhadas", "Show auto-detected": "Mostrar detectados automaticamente", + "Show hidden": "Mostrar ocultos", "Show submitter update notifications": "Mostrar notificações de atualização do submissor", "Specify a job bundle directory or run the bundle command with the --browse flag": "Especifique um diretório de pacote de tarefas ou execute o comando bundle com a flag --browse", "Specify output directories": "Especificar diretórios de saída", @@ -139,7 +144,33 @@ "Value(s)": "Valor(es)", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "Você está autenticado com o perfil '{profile}', mas este perfil não consegue chamar o AWS Deadline Cloud ListFarms e não consegue enviar trabalhos para o AWS Deadline Cloud.\n\nPara resolver este problema:\n\\u2022 Verifique se não há variáveis de ambiente apontando para a região da AWS errada (por exemplo, AWS_DEFAULT_REGION)\n\\u2022 Se você não estiver usando um perfil do Deadline Cloud Monitor, verifique se o perfil tem permissões para essas APIs do AWS Deadline Cloud necessárias para envio:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - Você está desconectado.", + "{profile} doesn't have access permissions to submit a job.": "{profile} não tem permissões de acesso para enviar um trabalho.", "{submitter} job submission": "Envio de trabalho {submitter}", "Default maximum retries per task": "M\u00e1ximo padr\u00e3o de tentativas por tarefa", - "Default maximum failed tasks count": "Contagem m\u00e1xima padr\u00e3o de tarefas com falha" -} \ No newline at end of file + "Default maximum failed tasks count": "Contagem m\u00e1xima padr\u00e3o de tarefas com falha", + "Browse Job Bundles": "Procurar pacotes de trabalho", + "History": "Histórico", + "Job bundle directory": "Diretório de pacotes de trabalho", + "Local": "Local", + "Name": "Nome", + "Parameters:": "Parâmetros:", + "Path:": "Caminho:", + "Select": "Selecionar", + "Select a job bundle to see details": "Selecione um pacote de trabalho para ver os detalhes", + "Source:": "Origem:", + "Steps:": "Etapas:", + "(no default)": "(sem padrão)", + "(required)": "(obrigatório)", + "Archive": "Arquivo", + "Choose one from the list to preview its details": "Escolha um da lista para visualizar seus detalhes", + "Filter bundles...": "Filtrar bundles...", + "Folder": "Pasta", + "Parameters": "Parâmetros", + "Select a job bundle": "Selecionar um job bundle", + "Steps": "Etapas", + "Type": "Tipo", + "Value": "Valor", + "Save to": "Salvar em", + "Location": "Local", + "Select directory": "Selecionar diretório" +} diff --git a/src/deadline/client/ui/translations/locales/tr_TR.json b/src/deadline/client/ui/translations/locales/tr_TR.json index cfcae03b9..d20be25a7 100644 --- a/src/deadline/client/ui/translations/locales/tr_TR.json +++ b/src/deadline/client/ui/translations/locales/tr_TR.json @@ -46,7 +46,7 @@ "Download installer": "Installer'ı indir", "Edit...": "Düzenle...", "Error: Timeout cannot be set to zero.": "Hata: Zaman aşımı sıfıra ayarlanamaz.", - "Export bundle": "Paketi dışa aktar", + "Save bundle as": "Paketi farklı kaydet", "Failed to log in to AWS Deadline Cloud:

{error}": "AWS Deadline Cloud'da oturum açılamadı:

{error}", "Farm": "Farm", "Farm settings": "Farm ayarları", @@ -58,6 +58,8 @@ "Hardware requirements": "Donanım gereksinimleri", "Hashing progress": "Karma oluşturma ilerlemesi", "Help": "Yardım", + "Hide bundle": "Bundle'ı gizle", + "Unhide bundle": "Bundle'ı göster", "Host requirements": "Ana bilgisayar gereksinimleri", "Initial state": "Başlangıç durumu", "Issue With Profile Configuration": "Profil yapılandırmasıyla ilgili sorun", @@ -72,6 +74,8 @@ "Language": "Dil", "Language will change next time the submitter is opened": "Dil, submitter bir sonraki açılışında değişecektir", "Load Bundle": "Paket yükle", + "Download bundle": "Paketi indir", + "Open bundle": "Paketi aç", "Log in": "Oturum aç", "Log in to AWS Deadline Cloud": "AWS Deadline Cloud'da oturum aç", "Logging you in...": "Oturum açılıyor...", @@ -112,6 +116,7 @@ "Settings...": "Ayarlar...", "Shared job settings": "Paylaşılan iş ayarları", "Show auto-detected": "Otomatik algılanmışları göster", + "Show hidden": "Gizlileri göster", "Show submitter update notifications": "Gönderi aracı güncelleme bildirimlerini göster", "Specify a job bundle directory or run the bundle command with the --browse flag": "Bir iş paketi dizini belirtin veya bundle komutunu --browse bayrağıyla çalıştırın", "Specify output directories": "Çıkış dizinlerini belirtin", @@ -139,7 +144,33 @@ "Value(s)": "Değer(ler)", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "'{profile}' profiliyle kimlik doğrulandınız, ancak bu profil AWS Deadline Cloud ListFarms'ı çağıramıyor ve AWS Deadline Cloud'a iş gönderemiyor.\n\nBu sorunu çözmek için:\n\\u2022 Yanlış AWS bölgesine işaret eden ortam değişkenleri olmadığını kontrol edin (örn. AWS_DEFAULT_REGION)\n\\u2022 Deadline Cloud Monitor profili kullanmıyorsanız, profilin gönderim için gereken şu AWS Deadline Cloud API'leri için izinlere sahip olduğunu kontrol edin:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - Oturumunuz kapatıldı.", + "{profile} doesn't have access permissions to submit a job.": "{profile} bir iş göndermek için erişim izinlerine sahip değil.", "{submitter} job submission": "{submitter} iş gönderimi", "Default maximum retries per task": "G\u00f6rev ba\u015f\u0131na varsay\u0131lan maksimum yeniden deneme", - "Default maximum failed tasks count": "Varsay\u0131lan maksimum ba\u015far\u0131s\u0131z g\u00f6rev say\u0131s\u0131" -} \ No newline at end of file + "Default maximum failed tasks count": "Varsay\u0131lan maksimum ba\u015far\u0131s\u0131z g\u00f6rev say\u0131s\u0131", + "Browse Job Bundles": "İş paketlerine göz at", + "History": "Geçmiş", + "Job bundle directory": "İş paketi dizini", + "Local": "Yerel", + "Name": "Ad", + "Parameters:": "Parametreler:", + "Path:": "Yol:", + "Select": "Seç", + "Select a job bundle to see details": "Ayrıntıları görmek için bir iş paketi seçin", + "Source:": "Kaynak:", + "Steps:": "Adımlar:", + "(no default)": "(varsayılan yok)", + "(required)": "(zorunlu)", + "Archive": "Arşiv", + "Choose one from the list to preview its details": "Ayrıntılarını önizlemek için listeden birini seçin", + "Filter bundles...": "Paketleri filtrele...", + "Folder": "Klasör", + "Parameters": "Parametreler", + "Select a job bundle": "İş paketi seçin", + "Steps": "Adımlar", + "Type": "Tür", + "Value": "Değer", + "Save to": "Kaydet", + "Location": "Konum", + "Select directory": "Dizin seçin" +} diff --git a/src/deadline/client/ui/translations/locales/zh_CN.json b/src/deadline/client/ui/translations/locales/zh_CN.json index 471187d4a..0798e42db 100644 --- a/src/deadline/client/ui/translations/locales/zh_CN.json +++ b/src/deadline/client/ui/translations/locales/zh_CN.json @@ -46,7 +46,7 @@ "Download installer": "下载安装程序", "Edit...": "编辑...", "Error: Timeout cannot be set to zero.": "错误: 超时不能设置为零。", - "Export bundle": "导出捆绑包", + "Save bundle as": "另存包为", "Failed to log in to AWS Deadline Cloud:

{error}": "登录 AWS Deadline Cloud 失败:

{error}", "Farm": "服务器农场", "Farm settings": "服务器农场设置", @@ -58,6 +58,8 @@ "Hardware requirements": "硬件要求", "Hashing progress": "哈希进度", "Help": "帮助", + "Hide bundle": "隐藏 Bundle", + "Unhide bundle": "显示 Bundle", "Host requirements": "主机要求", "Initial state": "初始状态", "Issue With Profile Configuration": "配置文件配置问题", @@ -72,6 +74,8 @@ "Language": "语言", "Language will change next time the submitter is opened": "语言将在下次打开提交器时更改", "Load Bundle": "加载捆绑包", + "Download bundle": "下载捆绑包", + "Open bundle": "打开捆绑包", "Log in": "登录", "Log in to AWS Deadline Cloud": "登录 AWS Deadline Cloud", "Logging you in...": "正在登录...", @@ -112,6 +116,7 @@ "Settings...": "设置...", "Shared job settings": "共享作业设置", "Show auto-detected": "显示自动检测的", + "Show hidden": "显示隐藏项", "Show submitter update notifications": "显示提交器更新通知", "Specify a job bundle directory or run the bundle command with the --browse flag": "指定作业捆绑包目录或使用 --browse 标志运行 bundle 命令", "Specify output directories": "指定输出目录", @@ -139,7 +144,33 @@ "Value(s)": "值", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "您已使用配置文件 '{profile}' 进行身份验证，但此配置文件无法调用 AWS Deadline Cloud ListFarms 并且无法向 AWS Deadline Cloud 提交作业。\n\n要解决此问题:\n\\u2022 检查是否没有指向错误 AWS 区域的环境变量 (例如 AWS_DEFAULT_REGION)\n\\u2022 如果您未使用 Deadline Cloud Monitor 配置文件，请检查配置文件是否具有提交所需的这些 AWS Deadline Cloud API 的权限:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - 您已登出。", + "{profile} doesn't have access permissions to submit a job.": "{profile} 没有提交作业的访问权限。", "{submitter} job submission": "{submitter} 作业提交", "Default maximum retries per task": "\u6bcf\u4e2a\u4efb\u52a1\u7684\u9ed8\u8ba4\u6700\u5927\u91cd\u8bd5\u6b21\u6570", - "Default maximum failed tasks count": "\u9ed8\u8ba4\u6700\u5927\u5931\u8d25\u4efb\u52a1\u6570" -} \ No newline at end of file + "Default maximum failed tasks count": "\u9ed8\u8ba4\u6700\u5927\u5931\u8d25\u4efb\u52a1\u6570", + "Browse Job Bundles": "浏览作业包", + "History": "历史记录", + "Job bundle directory": "作业包目录", + "Local": "本地", + "Name": "名称", + "Parameters:": "参数:", + "Path:": "路径:", + "Select": "选择", + "Select a job bundle to see details": "选择作业包以查看详细信息", + "Source:": "来源:", + "Steps:": "步骤:", + "(no default)": "(无默认值)", + "(required)": "(必填)", + "Archive": "归档", + "Choose one from the list to preview its details": "从列表中选择以预览其详细信息", + "Filter bundles...": "筛选包...", + "Folder": "文件夹", + "Parameters": "参数", + "Select a job bundle": "选择作业包", + "Steps": "步骤", + "Type": "类型", + "Value": "值", + "Save to": "保存到", + "Location": "位置", + "Select directory": "选择目录" +} diff --git a/src/deadline/client/ui/translations/locales/zh_TW.json b/src/deadline/client/ui/translations/locales/zh_TW.json index 2d3877dfe..3e3f37dcd 100644 --- a/src/deadline/client/ui/translations/locales/zh_TW.json +++ b/src/deadline/client/ui/translations/locales/zh_TW.json @@ -46,7 +46,7 @@ "Download installer": "下載安裝程式", "Edit...": "編輯...", "Error: Timeout cannot be set to zero.": "錯誤: 逾時不能設定為零。", - "Export bundle": "匯出套件", + "Save bundle as": "另存套件為", "Failed to log in to AWS Deadline Cloud:

{error}": "登入 AWS Deadline Cloud 失敗:

{error}", "Farm": "伺服器陣列", "Farm settings": "伺服器陣列設定", @@ -58,6 +58,8 @@ "Hardware requirements": "硬體需求", "Hashing progress": "雜湊進度", "Help": "說明", + "Hide bundle": "隱藏 Bundle", + "Unhide bundle": "顯示 Bundle", "Host requirements": "主機需求", "Initial state": "初始狀態", "Issue With Profile Configuration": "設定檔組態問題", @@ -72,6 +74,8 @@ "Language": "語言", "Language will change next time the submitter is opened": "語言將在下次開啟提交器時變更", "Load Bundle": "載入套件", + "Download bundle": "下載套件", + "Open bundle": "開啟套件", "Log in": "登入", "Log in to AWS Deadline Cloud": "登入 AWS Deadline Cloud", "Logging you in...": "正在登入...", @@ -112,6 +116,7 @@ "Settings...": "設定...", "Shared job settings": "共用任務設定", "Show auto-detected": "顯示自動偵測的", + "Show hidden": "顯示隱藏項目", "Show submitter update notifications": "顯示提交器更新通知", "Specify a job bundle directory or run the bundle command with the --browse flag": "指定任務套件目錄或使用 --browse 旗標執行 bundle 命令", "Specify output directories": "指定輸出目錄", @@ -139,7 +144,33 @@ "Value(s)": "值", "You are authenticated with the profile '{profile}', but this profile is unable to call AWS Deadline Cloud ListFarms and unable to submit jobs to AWS Deadline Cloud.\n\nTo resolve this issue:\n\u2022 Check that there aren't any environment variables pointing to the wrong AWS region (e.g., AWS_DEFAULT_REGION)\n\u2022 If you are not using a Deadline Cloud Monitor profile, check that the profile has permissions for these AWS Deadline Cloud APIs needed for submitting:\n \u2022 deadline:AssumeQueueRoleForUser\n \u2022 deadline:CreateJob\n \u2022 deadline:GetJob\n \u2022 deadline:GetQueue\n \u2022 deadline:GetQueueEnvironment\n \u2022 deadline:GetStorageProfileForQueue\n \u2022 deadline:GetStorageProfile\n \u2022 deadline:ListFarms\n \u2022 deadline:ListQueues\n \u2022 deadline:ListQueueEnvironments\n \u2022 deadline:ListStorageProfilesForQueue": "您已使用設定檔 '{profile}' 進行身分驗證，但此設定檔無法呼叫 AWS Deadline Cloud ListFarms 並且無法向 AWS Deadline Cloud 提交任務。\n\n若要解決此問題:\n\\u2022 檢查是否沒有指向錯誤 AWS 區域的環境變數 (例如 AWS_DEFAULT_REGION)\n\\u2022 如果您未使用 Deadline Cloud Monitor 設定檔，請檢查設定檔是否具有提交所需的這些 AWS Deadline Cloud API 的許可:\n \\u2022 deadline:AssumeQueueRoleForUser\n \\u2022 deadline:CreateJob\n \\u2022 deadline:GetJob\n \\u2022 deadline:GetQueue\n \\u2022 deadline:GetQueueEnvironment\n \\u2022 deadline:GetStorageProfileForQueue\n \\u2022 deadline:GetStorageProfile\n \\u2022 deadline:ListFarms\n \\u2022 deadline:ListQueues\n \\u2022 deadline:ListQueueEnvironments\n \\u2022 deadline:ListStorageProfilesForQueue", "{profile} - You are logged out.": "{profile} - 您已登出。", + "{profile} doesn't have access permissions to submit a job.": "{profile} 沒有提交任務的存取許可。", "{submitter} job submission": "{submitter} 任務提交", "Default maximum retries per task": "\u6bcf\u500b\u4efb\u52d9\u7684\u9810\u8a2d\u91cd\u8a66\u6b21\u6578\u4e0a\u9650", - "Default maximum failed tasks count": "\u9810\u8a2d\u5931\u6557\u4efb\u52d9\u6578\u4e0a\u9650" -} \ No newline at end of file + "Default maximum failed tasks count": "\u9810\u8a2d\u5931\u6557\u4efb\u52d9\u6578\u4e0a\u9650", + "Browse Job Bundles": "瀏覽工作套件", + "History": "歷史記錄", + "Job bundle directory": "工作套件目錄", + "Local": "本機", + "Name": "名稱", + "Parameters:": "參數:", + "Path:": "路徑:", + "Select": "選取", + "Select a job bundle to see details": "選取工作套件以查看詳細資訊", + "Source:": "來源:", + "Steps:": "步驟:", + "(no default)": "(無預設值)", + "(required)": "(必填)", + "Archive": "封存", + "Choose one from the list to preview its details": "從清單中選擇以預覽其詳細資訊", + "Filter bundles...": "篩選套件...", + "Folder": "資料夾", + "Parameters": "參數", + "Select a job bundle": "選擇工作套件", + "Steps": "步驟", + "Type": "類型", + "Value": "值", + "Save to": "儲存到", + "Location": "位置", + "Select directory": "選擇目錄" +} diff --git a/src/deadline/client/ui/widgets/__init__.py b/src/deadline/client/ui/widgets/__init__.py index 9955001c2..1c229d429 100644 --- a/src/deadline/client/ui/widgets/__init__.py +++ b/src/deadline/client/ui/widgets/__init__.py @@ -25,9 +25,12 @@ "DeadlineFarmListComboBoxController", "DeadlineQueueListComboBoxController", "DeadlineStorageProfileListComboBoxController", + # Expandable section + "ExpandableSection", ] from .deadline_authentication_status_widget import DeadlineAuthenticationStatusWidget +from .expandable_section import ExpandableSection from .host_requirements_tab import ( CustomAmountWidget, CustomAttributeValueWidget, diff --git a/src/deadline/client/ui/widgets/expandable_section.py b/src/deadline/client/ui/widgets/expandable_section.py new file mode 100644 index 000000000..a21713152 --- /dev/null +++ b/src/deadline/client/ui/widgets/expandable_section.py @@ -0,0 +1,85 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +""" +Expandable section widget — a clickable header that toggles content visibility. +Similar to Cloudscape's ExpandableSection component. +""" + +from __future__ import annotations + +from qtpy.QtCore import Qt, QSize, Signal # type: ignore +from qtpy.QtWidgets import QToolButton, QVBoxLayout, QWidget # type: ignore + + +class ExpandableSection(QWidget): + """A collapsible section with a toggle arrow and title. + + The arrow and title act as a single clickable header. Clicking toggles + the visibility of the content widget. + + Example:: + + section = ExpandableSection("Parameters (5)") + section.set_content(my_table_widget) + layout.addWidget(section) + """ + + toggled = Signal(bool) + + def __init__( + self, + title: str = "", + expanded: bool = False, + disable_content_paddings: bool = False, + parent=None, + ): + super().__init__(parent) + self._layout = QVBoxLayout(self) + self._layout.setContentsMargins(0, 0, 0, 0) + self._layout.setSpacing(0) + self._disable_content_paddings = disable_content_paddings + + self._header = QToolButton() + self._header.setStyleSheet("QToolButton { border: none; padding: 0px; }") + self._header.setToolButtonStyle(Qt.ToolButtonTextBesideIcon) + self._header.setIconSize(QSize(7, 7)) + self._header.setCheckable(True) + self._header.setChecked(expanded) + self._header.setArrowType(Qt.DownArrow if expanded else Qt.RightArrow) + if title: + self._header.setText(f" {title}") + self._header.toggled.connect(self._on_toggled) + self._layout.addWidget(self._header) + + self._content: QWidget | None = None + + def set_title(self, title: str): + """Update the section header text.""" + self._header.setText(f" {title}") + + def set_header_style(self, qss: str): + """Set custom stylesheet on the header button.""" + self._header.setStyleSheet(f"QToolButton {{ border: none; padding: 0px; {qss} }}") + + def set_content(self, widget: QWidget): + """Set the content widget that will be shown/hidden.""" + old = self._content + if old is not None: + self._layout.removeWidget(old) + self._content = widget + if not self._disable_content_paddings: + widget.setContentsMargins(16, 4, 0, 0) + widget.setVisible(self._header.isChecked()) + self._layout.addWidget(widget) + + def is_expanded(self) -> bool: + return self._header.isChecked() + + def set_expanded(self, expanded: bool): + self._header.setChecked(expanded) + + def _on_toggled(self, checked: bool): + self._header.setArrowType(Qt.DownArrow if checked else Qt.RightArrow) + if self._content is not None: + self._content.setVisible(checked) + self.toggled.emit(checked) diff --git a/src/deadline/client/ui/widgets/job_bundle_settings_tab.py b/src/deadline/client/ui/widgets/job_bundle_settings_tab.py index 9c5edf759..a157f4fbe 100644 --- a/src/deadline/client/ui/widgets/job_bundle_settings_tab.py +++ b/src/deadline/client/ui/widgets/job_bundle_settings_tab.py @@ -10,20 +10,20 @@ from logging import getLogger from typing import Any, Optional -from qtpy.QtCore import Signal # type: ignore +from qtpy.QtCore import QThread, Signal # type: ignore from qtpy.QtWidgets import ( # type: ignore QVBoxLayout, QWidget, - QFileDialog, QMessageBox, ) from ..dataclasses import JobBundleSettings +from ...config import get_setting +from ...job_bundle.repository import S3BundleRepository from .openjd_parameters_widget import OpenJDParametersWidget from ...job_bundle.submission import AssetReferences from ...job_bundle.loader import read_yaml_or_json_object, validate_directory_symlink_containment from ...job_bundle.parameters import read_job_bundle_parameters -from ...config import config_file logger = getLogger(__name__) @@ -76,18 +76,83 @@ def refresh_ui(self, settings: JobBundleSettings): lambda message: self.parameter_changed.emit(message) ) - def on_load_bundle(self): + def on_load_bundle(self, s3_repo=None): """ Browse and load the selected submission bundle """ - # Open the file picker dialog - bundle_path = os.path.expanduser(config_file.get_setting("settings.job_history_dir")) - input_job_bundle_dir = QFileDialog.getExistingDirectory( - self, "Choose job bundle directory", bundle_path - ) - if not input_job_bundle_dir: + from ..dialogs.job_bundle_browser_dialog import JobBundleBrowserDialog + + # Determine the default local browse directory + default_dir = get_setting("settings.job_bundle_default_directory") + if default_dir: + default_dir = os.path.expanduser(default_dir) + + # Get the job history directory for the current profile + job_history_dir = os.path.expanduser(get_setting("settings.job_history_dir")) + + s3_worker = None + if s3_repo: + # Reuse existing repo — no background init needed + browser = JobBundleBrowserDialog( + queue_source=s3_repo, + queue_error="", + local_source=default_dir, + history_source=job_history_dir, + parent=self, + ) + else: + # Start S3 initialization in background + class _S3InitWorker(QThread): + # NOTE: named ``done`` rather than ``finished`` to avoid shadowing + # QThread's built-in ``finished`` signal. + done = Signal(object, str, list, set) + + def run(self): + try: + from concurrent.futures import ThreadPoolExecutor + + repo = S3BundleRepository.from_config() + with ThreadPoolExecutor(max_workers=2) as ex: + entries_f = ex.submit(repo.list_entries, repo.root_path()) + hidden_f = ex.submit(repo.get_hidden_set) + entries = entries_f.result() + hidden = hidden_f.result() + self.done.emit(repo, "", entries, hidden) + except Exception as e: + self.done.emit(None, str(e), [], set()) + + browser = JobBundleBrowserDialog( + queue_source=None, + queue_error="", + queue_loading=True, + local_source=default_dir, + history_source=job_history_dir, + parent=self, + ) + # Connect the worker's result BEFORE starting it. A fast failure (e.g. + # not logged in) can emit ``done`` almost immediately; a cross-thread + # queued signal emitted before the connection exists is dropped, which + # would leave the browser stuck on "Loading..." forever. + s3_worker = _S3InitWorker() + s3_worker.done.connect(browser.set_queue_source) + s3_worker.start() + + if browser.exec_() != JobBundleBrowserDialog.Accepted or not browser.selected_path: + if s3_worker: + s3_worker.wait() return + browser.hide() + input_job_bundle_dir = browser.resolve_selection() + while not input_job_bundle_dir: + browser.show() + if browser.exec_() != JobBundleBrowserDialog.Accepted or not browser.selected_path: + if s3_worker: + s3_worker.wait() + return + browser.hide() + input_job_bundle_dir = browser.resolve_selection() + # Update job bundle directory path self.input_job_bundle_dir = input_job_bundle_dir diff --git a/test/fixtures/bundles/metadata-limit-test/template.yaml b/test/fixtures/bundles/metadata-limit-test/template.yaml new file mode 100644 index 000000000..13337def1 --- /dev/null +++ b/test/fixtures/bundles/metadata-limit-test/template.yaml @@ -0,0 +1,158 @@ +description: 'This is a very long description. This is a very long description. This + is a very long description. This is a very long description. This is a very long + description. This is a very long description. This is a very long description. This + is a very long description. This is a very long description. This is a very long + description. This is a very long description. This is a very long description. This + is a very long description. This is a very long description. This is a very long + description. This is a very long description. This is a very long description. This + is a very long description. This is a very long description. This is a very long + description. ' +name: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +parameterDefinitions: +- default: value_0 + name: Parameter_000_LongName + type: STRING +- default: value_1 + name: Parameter_001_LongName + type: STRING +- default: value_2 + name: Parameter_002_LongName + type: STRING +- default: value_3 + name: Parameter_003_LongName + type: STRING +- default: value_4 + name: Parameter_004_LongName + type: STRING +- default: value_5 + name: Parameter_005_LongName + type: STRING +- default: value_6 + name: Parameter_006_LongName + type: STRING +- default: value_7 + name: Parameter_007_LongName + type: STRING +- default: value_8 + name: Parameter_008_LongName + type: STRING +- default: value_9 + name: Parameter_009_LongName + type: STRING +- default: value_10 + name: Parameter_010_LongName + type: STRING +- default: value_11 + name: Parameter_011_LongName + type: STRING +- default: value_12 + name: Parameter_012_LongName + type: STRING +- default: value_13 + name: Parameter_013_LongName + type: STRING +- default: value_14 + name: Parameter_014_LongName + type: STRING +- default: value_15 + name: Parameter_015_LongName + type: STRING +- default: value_16 + name: Parameter_016_LongName + type: STRING +- default: value_17 + name: Parameter_017_LongName + type: STRING +- default: value_18 + name: Parameter_018_LongName + type: STRING +- default: value_19 + name: Parameter_019_LongName + type: STRING +- default: value_20 + name: Parameter_020_LongName + type: STRING +- default: value_21 + name: Parameter_021_LongName + type: STRING +- default: value_22 + name: Parameter_022_LongName + type: STRING +- default: value_23 + name: Parameter_023_LongName + type: STRING +- default: value_24 + name: Parameter_024_LongName + type: STRING +- default: value_25 + name: Parameter_025_LongName + type: STRING +- default: value_26 + name: Parameter_026_LongName + type: STRING +- default: value_27 + name: Parameter_027_LongName + type: STRING +- default: value_28 + name: Parameter_028_LongName + type: STRING +- default: value_29 + name: Parameter_029_LongName + type: STRING +- default: value_30 + name: Parameter_030_LongName + type: STRING +- default: value_31 + name: Parameter_031_LongName + type: STRING +- default: value_32 + name: Parameter_032_LongName + type: STRING +- default: value_33 + name: Parameter_033_LongName + type: STRING +- default: value_34 + name: Parameter_034_LongName + type: STRING +- default: value_35 + name: Parameter_035_LongName + type: STRING +- default: value_36 + name: Parameter_036_LongName + type: STRING +- default: value_37 + name: Parameter_037_LongName + type: STRING +- default: value_38 + name: Parameter_038_LongName + type: STRING +- default: value_39 + name: Parameter_039_LongName + type: STRING +specificationVersion: jobtemplate-2023-09 +steps: +- name: Step_000_RenderingProcess +- name: Step_001_RenderingProcess +- name: Step_002_RenderingProcess +- name: Step_003_RenderingProcess +- name: Step_004_RenderingProcess +- name: Step_005_RenderingProcess +- name: Step_006_RenderingProcess +- name: Step_007_RenderingProcess +- name: Step_008_RenderingProcess +- name: Step_009_RenderingProcess +- name: Step_010_RenderingProcess +- name: Step_011_RenderingProcess +- name: Step_012_RenderingProcess +- name: Step_013_RenderingProcess +- name: Step_014_RenderingProcess +- name: Step_015_RenderingProcess +- name: Step_016_RenderingProcess +- name: Step_017_RenderingProcess +- name: Step_018_RenderingProcess +- name: Step_019_RenderingProcess +- name: Step_020_RenderingProcess +- name: Step_021_RenderingProcess +- name: Step_022_RenderingProcess +- name: Step_023_RenderingProcess +- name: Step_024_RenderingProcess diff --git a/test/unit/deadline_client/cli/test_cli_bundle_repository.py b/test/unit/deadline_client/cli/test_cli_bundle_repository.py new file mode 100644 index 000000000..bb813a8ed --- /dev/null +++ b/test/unit/deadline_client/cli/test_cli_bundle_repository.py @@ -0,0 +1,835 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +"""Tests for the bundle CLI commands (list, upload, download, cache).""" + +import io +import json +import os +import time +import zipfile + +import yaml +from botocore.exceptions import ClientError +from click.testing import CliRunner +from unittest.mock import MagicMock, patch + +from deadline.client.cli import main +from deadline.client.job_bundle.repository import ( + BrowseEntry, + LocalBundleRepository, + METADATA_LIMIT_NAME, + S3_METADATA_TOTAL_BUDGET, + S3BundleRepository, +) + +BUNDLE_GROUP = "deadline.client.cli._groups.bundle_group" + + +class TestBundleList: + def test_list_local_path(self, tmp_path): + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: Test\nsteps:\n- name: S1\n") + (tmp_path / "not-a-bundle").mkdir() + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "list", str(tmp_path)]) + assert result.exit_code == 0 + assert "my-bundle" in result.output + assert "not-a-bundle" not in result.output + + def test_list_local_json(self, tmp_path): + bundle = tmp_path / "render-job" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: Render\nsteps: []\n") + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "list", str(tmp_path), "--output", "json"]) + assert result.exit_code == 0 + data = json.loads(result.output) + assert len(data) == 1 + assert data[0]["name"] == "render-job" + assert data[0]["format"] == "folder" + + def test_list_local_no_archives(self, tmp_path): + bundle = tmp_path / "dir-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: Dir\nsteps: []\n") + + ojd_path = tmp_path / "archive-bundle.ojd" + with zipfile.ZipFile(str(ojd_path), "w") as zf: + zf.writestr("template.yaml", "name: Zipped\nsteps: []\n") + + runner = CliRunner() + + result = runner.invoke(main, ["bundle", "list", str(tmp_path)]) + assert result.exit_code == 0 + assert "dir-bundle" in result.output + assert "archive-bundle" in result.output + + result = runner.invoke(main, ["bundle", "list", str(tmp_path), "--no-archives"]) + assert result.exit_code == 0 + assert "dir-bundle" in result.output + assert "archive-bundle" not in result.output + + def test_list_empty_dir(self, tmp_path): + runner = CliRunner() + result = runner.invoke(main, ["bundle", "list", str(tmp_path)]) + assert result.exit_code == 0 + assert result.output.strip() == "" + + +class TestBundleUpload: + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + @patch("boto3.client") + def test_upload_creates_zip(self, mock_boto3_client, mock_s3_settings, mock_config, tmp_path): + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "Test Bundle", + "steps": [{"name": "Run"}], + } + ) + ) + + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DeadlineCloud"), + mock_session, + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "upload", str(bundle)]) + + assert result.exit_code == 0, result.output + assert "Uploaded bundle to" in result.output + mock_s3.upload_fileobj.assert_called_once() + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_not_a_bundle(self, mock_s3_settings, mock_config, tmp_path): + not_bundle = tmp_path / "empty" + not_bundle.mkdir() + + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DeadlineCloud"), + MagicMock(), + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "upload", str(not_bundle)]) + assert result.exit_code == 1 + assert "not appear to be a job bundle" in result.output + + def _bundle_dir(self, tmp_path): + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump({"specificationVersion": "jobtemplate-2023-09", "name": "T", "steps": []}) + ) + return bundle + + def _mock_queue(self, mock_s3_settings): + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DeadlineCloud"), + mock_session, + ) + return mock_s3 + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_name_with_slash_is_sanitized(self, mock_s3_settings, mock_config, tmp_path): + """An embedded '/' in --name is flattened so the object stays in job-bundles/.""" + bundle = self._bundle_dir(tmp_path) + mock_s3 = self._mock_queue(mock_s3_settings) + + result = CliRunner().invoke(main, ["bundle", "upload", str(bundle), "--name", "a/b"]) + + assert result.exit_code == 0, result.output + s3_key = mock_s3.upload_fileobj.call_args.args[2] + assert s3_key == "DeadlineCloud/job-bundles/a_b.ojd" + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_name_traversal_stays_within_prefix( + self, mock_s3_settings, mock_config, tmp_path + ): + """A '../..'-style --name cannot escape the job-bundles/ prefix. + + Previously the name was interpolated into the S3 key verbatim, so + "../../evil" could write to an arbitrary sub-prefix. Routing through + sanitize_bundle_name flattens the separators, keeping the object inside + job-bundles/ with no extra path segments. + """ + bundle = self._bundle_dir(tmp_path) + mock_s3 = self._mock_queue(mock_s3_settings) + + result = CliRunner().invoke(main, ["bundle", "upload", str(bundle), "--name", "../../evil"]) + + assert result.exit_code == 0, result.output + s3_key = mock_s3.upload_fileobj.call_args.args[2] + assert s3_key.startswith("DeadlineCloud/job-bundles/") + # No sub-prefix beyond the fixed job-bundles path (exactly two slashes). + assert s3_key.count("/") == 2 + assert s3_key == "DeadlineCloud/job-bundles/.._.._evil.ojd" + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_rejects_unsafe_name(self, mock_s3_settings, mock_config, tmp_path): + """A name that is empty/only-separators after sanitization is rejected.""" + bundle = self._bundle_dir(tmp_path) + mock_s3 = self._mock_queue(mock_s3_settings) + + for bad_name in ("..", "///"): + result = CliRunner().invoke(main, ["bundle", "upload", str(bundle), "--name", bad_name]) + assert result.exit_code == 1, (bad_name, result.output) + assert "empty or invalid" in result.output + mock_s3.upload_fileobj.assert_not_called() + + +class TestMetadataTruncation: + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_truncates_metadata_with_warning( + self, mock_s3_settings, mock_config, tmp_path, capsys + ): + bundle = tmp_path / "big-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "A" * 300, + "description": "D" * 1200, + "steps": [{"name": f"Step_{i:03d}_Long"} for i in range(40)], + "parameterDefinitions": [ + {"name": f"Param_{i:03d}_Long", "type": "STRING"} for i in range(50) + ], + } + ) + ) + + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DC"), + mock_session, + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "upload", str(bundle)]) + assert result.exit_code == 0, result.output + + # Verify metadata values respect limits + call_args = mock_s3.upload_fileobj.call_args + metadata = call_args[1]["ExtraArgs"]["Metadata"] + # The object is typed as a zip (courtesy hint; not trusted on download). + assert call_args[1]["ExtraArgs"]["ContentType"] == "application/zip" + assert len(metadata["ojd-name"].encode("utf-8")) <= METADATA_LIMIT_NAME + # Total metadata must stay within S3's 2KB budget + total = sum( + 12 + len(k.encode("utf-8")) + len(v.encode("utf-8")) for k, v in metadata.items() + ) + assert total <= S3_METADATA_TOTAL_BUDGET + + # Verify truncated values end with "..." + assert metadata["ojd-name"].endswith("...") + assert metadata["ojd-desc"].endswith("...") + assert metadata["ojd-steps"].endswith("...") + assert metadata["ojd-params"].endswith("...") + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_no_truncation_when_within_limits(self, mock_s3_settings, mock_config, tmp_path): + bundle = tmp_path / "small-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "Short Name", + "description": "Brief", + "steps": [{"name": "Render"}], + "parameterDefinitions": [{"name": "Frames", "type": "STRING"}], + } + ) + ) + + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DC"), + mock_session, + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "upload", str(bundle)]) + assert result.exit_code == 0, result.output + + # No warnings + assert "truncated" not in result.output.lower() + + # Values stored as-is without "..." + call_args = mock_s3.upload_fileobj.call_args + metadata = call_args[1]["ExtraArgs"]["Metadata"] + assert metadata["ojd-name"] == "Short Name" + assert not metadata["ojd-name"].endswith("...") + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_sanitizes_control_characters_in_name( + self, mock_s3_settings, mock_config, tmp_path + ): + """Control characters in --name are sanitized (replaced with '_'), + consistent with the GUI export path and sanitize_bundle_name.""" + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: Test\nsteps:\n- name: S1\n") + + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="bucket", rootPrefix="DC"), + mock_session, + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "upload", str(bundle), "--name", "bad\x01name"]) + assert result.exit_code == 0, result.output + s3_key = mock_s3.upload_fileobj.call_args.args[2] + assert s3_key == "DC/job-bundles/bad_name.ojd" + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_archive_produces_same_metadata_as_directory( + self, mock_s3_settings, mock_config, tmp_path + ): + """Uploading a .ojd archive should produce the same S3 metadata as uploading + the equivalent directory bundle.""" + template_content = yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "Metadata Test", + "description": "A bundle for testing metadata parity", + "steps": [{"name": "StepOne"}, {"name": "StepTwo"}], + "parameterDefinitions": [ + {"name": "Frames", "type": "STRING", "default": "1-10"}, + {"name": "OutputDir", "type": "PATH"}, + ], + } + ) + + # Create directory bundle + bundle_dir = tmp_path / "my-bundle" + bundle_dir.mkdir() + (bundle_dir / "template.yaml").write_text(template_content) + + # Create .ojd archive with same content + archive_path = tmp_path / "my-bundle.ojd" + with zipfile.ZipFile(archive_path, "w") as zf: + zf.writestr("template.yaml", template_content) + + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DC"), + mock_session, + ) + + runner = CliRunner() + + # Upload directory + result_dir = runner.invoke(main, ["bundle", "upload", str(bundle_dir)]) + assert result_dir.exit_code == 0, result_dir.output + dir_metadata = mock_s3.upload_fileobj.call_args[1]["ExtraArgs"]["Metadata"] + + mock_s3.reset_mock() + mock_s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + + # Upload archive + result_ojd = runner.invoke(main, ["bundle", "upload", str(archive_path)]) + assert result_ojd.exit_code == 0, result_ojd.output + ojd_metadata = mock_s3.upload_fileobj.call_args[1]["ExtraArgs"]["Metadata"] + + # Metadata should be identical + assert dir_metadata == ojd_metadata + assert dir_metadata["ojd-name"] == "Metadata Test" + assert "ojd-step-count" in dir_metadata + assert dir_metadata["ojd-step-count"] == "2" + assert "ojd-param-count" in dir_metadata + assert dir_metadata["ojd-param-count"] == "2" + + +class TestBundleUploadOverwrite: + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_prompts_when_bundle_exists_and_user_confirms( + self, mock_s3_settings, mock_config, tmp_path + ): + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: Test\nsteps:\n- name: S1\n") + + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.return_value = {} # exists + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DC"), + mock_session, + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "upload", str(bundle)], input="y\n") + assert result.exit_code == 0, result.output + assert "already exists" in result.output + mock_s3.upload_fileobj.assert_called_once() + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}._get_queue_s3_settings") + def test_upload_aborts_when_bundle_exists_and_user_declines( + self, mock_s3_settings, mock_config, tmp_path + ): + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: Test\nsteps:\n- name: S1\n") + + mock_session = MagicMock() + mock_s3 = MagicMock() + mock_s3.head_object.return_value = {} # exists + mock_session.client.return_value = mock_s3 + mock_s3_settings.return_value = ( + MagicMock(s3BucketName="test-bucket", rootPrefix="DC"), + mock_session, + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "upload", str(bundle)], input="n\n") + assert result.exit_code == 0, result.output + assert "canceled" in result.output.lower() + mock_s3.upload_fileobj.assert_not_called() + + +class TestBundleHide: + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_hide_bundle(self, mock_from_config, mock_config): + mock_repo = MagicMock() + mock_repo.get_hidden_set.return_value = set() + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "hide", "blender-render"]) + + assert result.exit_code == 0, result.output + assert "Hidden bundle: blender-render" in result.output + mock_repo.set_bundle_visibility.assert_called_once_with("blender-render", hidden=True) + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_hide_already_hidden(self, mock_from_config, mock_config): + mock_repo = MagicMock() + mock_repo.get_hidden_set.return_value = {"blender-render"} + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "hide", "blender-render"]) + + assert result.exit_code == 0, result.output + assert "already hidden" in result.output + mock_repo.set_bundle_visibility.assert_not_called() + + +class TestBundleUnhide: + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_unhide_bundle(self, mock_from_config, mock_config): + mock_repo = MagicMock() + mock_repo.get_hidden_set.return_value = {"blender-render"} + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "unhide", "blender-render"]) + + assert result.exit_code == 0, result.output + assert "Unhidden bundle: blender-render" in result.output + mock_repo.set_bundle_visibility.assert_called_once_with("blender-render", hidden=False) + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_unhide_not_hidden(self, mock_from_config, mock_config): + mock_repo = MagicMock() + mock_repo.get_hidden_set.return_value = set() + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "unhide", "blender-render"]) + + assert result.exit_code == 0, result.output + assert "not hidden" in result.output + mock_repo.set_bundle_visibility.assert_not_called() + + +class TestBundleListShowHidden: + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_list_queue_hides_hidden_by_default(self, mock_from_config, mock_config): + mock_repo = MagicMock() + mock_repo.get_hidden_set.return_value = {"old-job"} + mock_repo.root_path.return_value = "s3://bucket/prefix/job-bundles/" + mock_repo.list_entries.return_value = [ + BrowseEntry( + name="blender-render", + path="s3://b/p/blender-render.ojd", + is_bundle=True, + is_archive=True, + ), + BrowseEntry( + name="old-job", path="s3://b/p/old-job.ojd", is_bundle=True, is_archive=True + ), + ] + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "list", "--queue"]) + + assert result.exit_code == 0, result.output + assert "blender-render" in result.output + assert "old-job" not in result.output + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_list_queue_show_hidden(self, mock_from_config, mock_config): + mock_repo = MagicMock() + mock_repo.get_hidden_set.return_value = {"old-job"} + mock_repo.root_path.return_value = "s3://bucket/prefix/job-bundles/" + mock_repo.list_entries.return_value = [ + BrowseEntry( + name="blender-render", + path="s3://b/p/blender-render.ojd", + is_bundle=True, + is_archive=True, + ), + BrowseEntry( + name="old-job", path="s3://b/p/old-job.ojd", is_bundle=True, is_archive=True + ), + ] + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "list", "--queue", "--show-hidden"]) + + assert result.exit_code == 0, result.output + assert "blender-render" in result.output + assert "old-job (hidden)" in result.output + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_list_queue_show_hidden_json(self, mock_from_config, mock_config): + mock_repo = MagicMock() + mock_repo.get_hidden_set.return_value = {"old-job"} + mock_repo.root_path.return_value = "s3://bucket/prefix/job-bundles/" + mock_repo.list_entries.return_value = [ + BrowseEntry( + name="blender-render", + path="s3://b/p/blender-render.ojd", + is_bundle=True, + is_archive=True, + ), + BrowseEntry( + name="old-job", path="s3://b/p/old-job.ojd", is_bundle=True, is_archive=True + ), + ] + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke( + main, ["bundle", "list", "--queue", "--show-hidden", "--output", "json"] + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert len(data) == 2 + hidden_entry = next(e for e in data if e["name"] == "old-job") + visible_entry = next(e for e in data if e["name"] == "blender-render") + assert hidden_entry["hidden"] is True + assert "hidden" not in visible_entry + + +class TestLocalArchiveCache: + """Tests for LocalBundleRepository.extract_bundle mtime-based caching.""" + + def test_extract_caches_and_reuses(self, fresh_deadline_config, tmp_path): + """Second extraction of unchanged archive uses the cache.""" + # Create an .ojd archive + archive = tmp_path / "my-bundle.ojd" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("template.yaml", "name: Test\nsteps:\n- name: S1\n") + zf.writestr("script.sh", "echo hello") + + repo = LocalBundleRepository(root=str(tmp_path)) + + # First extraction + result1 = repo.extract_bundle(str(archive), str(tmp_path / "out")) + assert os.path.isfile(os.path.join(result1, "template.yaml")) + assert os.path.isfile(os.path.join(result1, "script.sh")) + + # Record the extraction time by checking a file's mtime in cache + template_mtime1 = os.path.getmtime(os.path.join(result1, "template.yaml")) + + # Second extraction — should reuse cache (same path returned) + result2 = repo.extract_bundle(str(archive), str(tmp_path / "out")) + assert result2 == result1 + + # File mtime should be unchanged (no re-extraction happened) + template_mtime2 = os.path.getmtime(os.path.join(result2, "template.yaml")) + assert template_mtime1 == template_mtime2 + + def test_extract_invalidates_on_mtime_change(self, fresh_deadline_config, tmp_path): + """Modified archive triggers re-extraction.""" + # Create initial archive + archive = tmp_path / "my-bundle.ojd" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("template.yaml", "name: V1\nsteps:\n- name: S1\n") + + repo = LocalBundleRepository(root=str(tmp_path)) + + # First extraction + result1 = repo.extract_bundle(str(archive), str(tmp_path / "out")) + with open(os.path.join(result1, "template.yaml")) as f: + assert "V1" in f.read() + + # Modify the archive (ensure different mtime) + time.sleep(0.05) + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("template.yaml", "name: V2\nsteps:\n- name: S2\n") + + # Second extraction — should detect mtime change and re-extract + result2 = repo.extract_bundle(str(archive), str(tmp_path / "out")) + with open(os.path.join(result2, "template.yaml")) as f: + assert "V2" in f.read() + + def test_extract_handles_wrapper_directory(self, fresh_deadline_config, tmp_path): + """Archive with a single wrapper dir returns the inner path.""" + archive = tmp_path / "wrapped.ojd" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("inner/template.yaml", "name: Wrapped\nsteps:\n- name: S1\n") + + repo = LocalBundleRepository(root=str(tmp_path)) + result = repo.extract_bundle(str(archive), str(tmp_path / "out")) + + # Should return the inner directory, not the extraction root + assert os.path.basename(result) == "inner" + assert os.path.isfile(os.path.join(result, "template.yaml")) + + +class TestBundleDownload: + """Tests for `deadline bundle download` — validates the full download + extract + copy flow.""" + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_download_to_output_dir( + self, mock_from_config, mock_config, tmp_path, fresh_deadline_config + ): + """Download extracts bundle to output dir.""" + # Create a real .ojd archive to serve as the download + bundle_content = {"template.yaml": "name: DownloadTest\nsteps:\n- name: S1\n"} + archive_buf = io.BytesIO() + with zipfile.ZipFile(archive_buf, "w") as zf: + for name, content in bundle_content.items(): + zf.writestr(name, content) + archive_bytes = archive_buf.getvalue() + + mock_repo = MagicMock() + mock_repo.root_path.return_value = "s3://bucket/DC/job-bundles/" + mock_repo.list_entries.return_value = [ + BrowseEntry( + name="test-bundle", + path="s3://bucket/DC/job-bundles/test-bundle.ojd", + is_bundle=True, + is_archive=True, + ), + ] + mock_repo.get_bundle_size.return_value = len(archive_bytes) + + # Make download_full_bundle extract to a real temp dir + extract_dir = tmp_path / "cache" / "test-bundle" + extract_dir.mkdir(parents=True) + with zipfile.ZipFile(io.BytesIO(archive_bytes), "r") as zf: + zf.extractall(str(extract_dir)) + + mock_repo.download_full_bundle.return_value = str(extract_dir) + mock_from_config.return_value = mock_repo + + output_dir = tmp_path / "output" + runner = CliRunner() + result = runner.invoke(main, ["bundle", "download", "test-bundle", "-o", str(output_dir)]) + + assert result.exit_code == 0, result.output + assert "Downloaded bundle to:" in result.output + assert os.path.isfile(str(output_dir / "test-bundle" / "template.yaml")) + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_download_no_output_dir_prints_cache_path( + self, mock_from_config, mock_config, tmp_path, fresh_deadline_config + ): + """Without -o, prints the cache path.""" + mock_repo = MagicMock() + mock_repo.root_path.return_value = "s3://bucket/DC/job-bundles/" + mock_repo.list_entries.return_value = [ + BrowseEntry( + name="my-bundle", + path="s3://bucket/DC/job-bundles/my-bundle.ojd", + is_bundle=True, + is_archive=True, + ), + ] + mock_repo.get_bundle_size.return_value = 1024 + cache_path = str(tmp_path / "cached-bundle") + mock_repo.download_full_bundle.return_value = cache_path + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "download", "my-bundle"]) + + assert result.exit_code == 0, result.output + assert cache_path in result.output + + @patch(f"{BUNDLE_GROUP}._apply_cli_options_to_config") + @patch(f"{BUNDLE_GROUP}.S3BundleRepository.from_config") + def test_download_not_found(self, mock_from_config, mock_config, fresh_deadline_config): + """Bundle not found shows error with available bundles.""" + mock_repo = MagicMock() + mock_repo.root_path.return_value = "s3://bucket/DC/job-bundles/" + mock_repo.list_entries.return_value = [ + BrowseEntry( + name="other-bundle", + path="s3://b/k/other-bundle.ojd", + is_bundle=True, + is_archive=True, + ), + ] + mock_from_config.return_value = mock_repo + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "download", "nonexistent"]) + + assert result.exit_code == 1 + assert "not found" in result.output + assert "other-bundle" in result.output + + +class TestBundleInfo: + """Tests for `deadline bundle info` — validates local and queue info output.""" + + def test_info_local_bundle(self, tmp_path, fresh_deadline_config): + """Info on a local bundle directory prints template details.""" + bundle = tmp_path / "my-render" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "Render Job", + "description": "Renders frames", + "steps": [{"name": "RenderStep"}], + "parameterDefinitions": [ + {"name": "Frames", "type": "STRING", "default": "1-10"} + ], + } + ) + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "info", str(bundle)]) + + assert result.exit_code == 0, result.output + assert "Render Job" in result.output + assert "RenderStep" in result.output + assert "Frames" in result.output + + def test_info_local_json_output(self, tmp_path, fresh_deadline_config): + """Info with --output json returns structured data.""" + bundle = tmp_path / "json-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "JSON Test", + "steps": [{"name": "S1"}], + } + ) + ) + + runner = CliRunner() + result = runner.invoke(main, ["bundle", "info", str(bundle), "--output", "json"]) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["name"] == "JSON Test" + assert data["steps"] == ["S1"] + + def test_info_not_found(self, tmp_path, fresh_deadline_config): + """Info on nonexistent path shows error.""" + runner = CliRunner() + result = runner.invoke(main, ["bundle", "info", str(tmp_path / "nope")]) + + assert result.exit_code == 1 + assert "not found" in result.output.lower() + + +class TestDownloadProgressHeadObjectReuse: + """Verify that get_bundle_size + download_full_bundle reuses the head_object call.""" + + def test_get_bundle_size_caches_head_for_download(self, fresh_deadline_config): + mock_s3 = MagicMock() + mock_s3.head_object.return_value = { + "ETag": '"abc123"', + "ContentLength": 4096, + "LastModified": "2026-01-01T00:00:00Z", + } + # download_fileobj writes bytes into the buffer + # Write fake zip-like bytes into the buffer + mock_s3.download_fileobj.side_effect = lambda Fileobj, **kwargs: Fileobj.write( + b"PK\x03\x04" + b"\x00" * 100 + ) + + repo = S3BundleRepository(bucket_name="bucket", root_prefix="DC", session=MagicMock()) + repo._s3 = mock_s3 + + # get_bundle_size does head_object + size = repo.get_bundle_size("s3://bucket/DC/job-bundles/test.ojd") + assert size == 4096 + assert mock_s3.head_object.call_count == 1 + + # download_full_bundle should reuse the cached head — no additional head_object + # (It will fail on extraction since our fake data isn't a real zip, but + # we only care about the head_object count) + try: + repo.download_full_bundle("s3://bucket/DC/job-bundles/test.ojd", "/tmp") + except Exception: + pass # Expected — fake zip data + + # Still only 1 head_object call total + assert mock_s3.head_object.call_count == 1 diff --git a/test/unit/deadline_client/cli/test_cli_config.py b/test/unit/deadline_client/cli/test_cli_config.py index d454367da..2fe01d3a4 100644 --- a/test/unit/deadline_client/cli/test_cli_config.py +++ b/test/unit/deadline_client/cli/test_cli_config.py @@ -36,7 +36,7 @@ def test_cli_config_show_defaults(fresh_deadline_config): assert fresh_deadline_config in result.output # Assert the expected number of settings - assert len(settings.keys()) == 25 + assert len(settings.keys()) == 26 for setting_name in settings.keys(): assert setting_name in result.output @@ -113,6 +113,7 @@ def test_cli_config_show_modified_config(fresh_deadline_config): config.set_setting("settings.max_failed_tasks_count", "50") config.set_setting("settings.deadline_regions", "us-west-2,us-east-1") config.set_setting("defaults.farm_region", "us-east-1") + config.set_setting("settings.job_bundle_default_directory", "/my/bundles") runner = CliRunner() result = runner.invoke(main, ["config", "show"]) @@ -143,6 +144,7 @@ def test_cli_config_show_modified_config(fresh_deadline_config): assert "\\known\\asset\\path" in result.output else: assert "/known/asset/path" in result.output + assert "/my/bundles" in result.output # It shouldn't say anywhere that there is a default setting assert "(default)" not in result.output diff --git a/test/unit/deadline_client/config/test_config_file.py b/test/unit/deadline_client/config/test_config_file.py index 4945ed638..a7f59dade 100644 --- a/test/unit/deadline_client/config/test_config_file.py +++ b/test/unit/deadline_client/config/test_config_file.py @@ -29,6 +29,7 @@ ("settings.locale", "", "ja_JP"), ("settings.force_s3_check", "false", "true"), ("settings.deadline_regions", "", "us-east-1,eu-west-1"), + ("settings.job_bundle_default_directory", "", "/my/bundles"), ] diff --git a/test/unit/deadline_client/job_bundle/test_repository.py b/test/unit/deadline_client/job_bundle/test_repository.py new file mode 100644 index 000000000..c37a070df --- /dev/null +++ b/test/unit/deadline_client/job_bundle/test_repository.py @@ -0,0 +1,1812 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +"""Tests for the job bundle repository module.""" + +from __future__ import annotations + +import io +import json +import math +import os +import sys +import zipfile +from pathlib import Path + +import pytest +import yaml + +from unittest.mock import MagicMock, patch +from botocore.exceptions import ClientError + +from deadline.client.exceptions import DeadlineOperationError +from deadline.client.job_bundle.repository import ( + LocalBundleRepository, + MAX_ARCHIVE_ENTRIES, + METADATA_LIMIT_NAME, + PREVIEW_MAX_DESC_LEN, + PREVIEW_MAX_NAME_LEN, + PREVIEW_MAX_PARAM_VALUE_LEN, + PREVIEW_MAX_PARAMS, + PREVIEW_MAX_STEPS, + S3BundleRepository, + S3_METADATA_TOTAL_BUDGET, + PREVIEW_PREFETCH_MAX_WORKERS, + _make_s3_client, + _bundle_info_from_s3_metadata, + _check_archive_extraction_safety, + _decode_s3_value, + _DOWNLOAD_SPOOL_THRESHOLD, + _encode_s3_value, + _extract_archive, + _extract_archive_from_fileobj, + _is_archive, + _open_download_sink, + _parse_template, + _safe_zip_extract, + _strip_archive_ext, + _truncate_s3_value, + archive_bundle_dir, + build_bundle_metadata, + extract_bundle_info, + get_bundle_cache_dir, + get_bundle_dir_size, + read_template_from_archive, + sanitize_bundle_name, +) + + +class TestParseTemplate: + def test_parse_yaml(self): + raw = "name: Test\nsteps:\n- name: Step1\n" + result = _parse_template(raw, "template.yaml") + assert result == {"name": "Test", "steps": [{"name": "Step1"}]} + + def test_parse_json(self): + raw = json.dumps({"name": "Test", "steps": [{"name": "Step1"}]}) + result = _parse_template(raw, "template.json") + assert result == {"name": "Test", "steps": [{"name": "Step1"}]} + + def test_parse_invalid_yaml(self): + result = _parse_template("{{invalid", "template.yaml") + assert result is None + + def test_parse_invalid_json(self): + result = _parse_template("{invalid", "template.json") + assert result is None + + +class TestExtractBundleInfoCaps: + """extract_bundle_info must cap oversized fields and tolerate malformed + templates so a maliciously constructed bundle cannot DoS or crash the + preview. Caps are at (never above) the OpenJD spec maxima; steps have no + spec bound so we cap them ourselves.""" + + def test_oversized_name_and_description_are_capped(self): + template = { + "name": "A" * 10_000, + "description": "D" * 10_000, + "steps": [{"name": "S"}], + } + info = extract_bundle_info(template, "/path") + assert len(info.name) == PREVIEW_MAX_NAME_LEN + assert len(info.description) == PREVIEW_MAX_DESC_LEN + + def test_huge_step_count_is_capped_with_true_total(self): + template = {"name": "J", "steps": [{"name": f"S{i}"} for i in range(50_000)]} + info = extract_bundle_info(template, "/path") + assert len(info.step_names) == PREVIEW_MAX_STEPS + # The real total is preserved so the preview can show "… N more". + assert info.total_steps == 50_000 + + def test_huge_parameter_count_is_capped_with_true_total(self): + template = { + "name": "J", + "steps": [], + "parameterDefinitions": [{"name": f"P{i}", "type": "STRING"} for i in range(10_000)], + } + info = extract_bundle_info(template, "/path") + assert len(info.parameters) == PREVIEW_MAX_PARAMS + assert info.total_parameters == 10_000 + + def test_oversized_parameter_value_is_capped(self): + template = { + "name": "J", + "steps": [], + "parameterDefinitions": [ + {"name": "P", "type": "STRING", "default": "x" * 10_000}, + ], + } + info = extract_bundle_info(template, "/path") + assert len(info.parameters[0]["_display_value"]) == PREVIEW_MAX_PARAM_VALUE_LEN + + def test_within_limits_reports_no_truncation(self): + template = { + "name": "J", + "steps": [{"name": "S1"}, {"name": "S2"}], + "parameterDefinitions": [{"name": "P", "type": "STRING"}], + } + info = extract_bundle_info(template, "/path") + assert info.total_steps is None + assert info.total_parameters is None + + def test_malformed_template_does_not_crash(self): + # steps/parameterDefinitions not lists, non-dict entries, non-string name. + template = { + "name": 12345, + "description": ["not", "a", "string"], + "steps": "not-a-list", + "parameterDefinitions": [{"name": "ok", "type": "STRING"}, "garbage", 42], + } + info = extract_bundle_info(template, "/path/to/bundle") + assert info.name == "bundle" # falls back to basename + assert info.description == "" + assert info.step_names == [] + # Only the well-formed parameter dict survives. + assert [p["name"] for p in info.parameters] == ["ok"] + + def test_does_not_mutate_caller_template(self): + template = { + "name": "J", + "steps": [], + "parameterDefinitions": [{"name": "P", "type": "STRING", "default": "1"}], + } + extract_bundle_info(template, "/path") + # The caller's parameter dict must not gain a _display_value key. + assert "_display_value" not in template["parameterDefinitions"][0] + + +class TestExtractBundleInfo: + def test_full_template(self): + template = { + "name": "My Job", + "description": "A test job", + "steps": [{"name": "Step1"}, {"name": "Step2"}], + "parameterDefinitions": [ + {"name": "Param1", "type": "STRING"}, + {"name": "Param2", "type": "PATH"}, + ], + } + info = extract_bundle_info(template, "/path/to/bundle") + assert info.name == "My Job" + assert info.description == "A test job" + assert info.step_names == ["Step1", "Step2"] + assert len(info.parameters) == 2 + + def test_minimal_template(self): + template = {"steps": [{"name": "OnlyStep"}]} + info = extract_bundle_info(template, "/path/to/bundle") + assert info.name == "bundle" + assert info.description == "" + assert info.step_names == ["OnlyStep"] + assert info.parameters == [] + + def test_parameter_values_from_file(self): + template = { + "name": "Job", + "steps": [], + "parameterDefinitions": [ + {"name": "Frames", "type": "STRING", "default": "1-10"}, + {"name": "Output", "type": "PATH"}, + ], + } + pv = {"parameterValues": [{"name": "Frames", "value": "1-50"}]} + info = extract_bundle_info(template, "/path", pv) + frames = next(p for p in info.parameters if p["name"] == "Frames") + output = next(p for p in info.parameters if p["name"] == "Output") + assert frames["_display_value"] == "1-50" # from parameter_values + assert "_display_value" not in output # no value or default + + def test_parameter_default_used_when_no_value(self): + template = { + "name": "Job", + "steps": [], + "parameterDefinitions": [ + {"name": "Frames", "type": "STRING", "default": "1-10"}, + ], + } + info = extract_bundle_info(template, "/path") + frames = info.parameters[0] + assert frames["_display_value"] == "1-10" + + def test_name_with_param_reference(self): + template = { + "name": "Render {{Param.SceneName}}", + "steps": [], + "parameterDefinitions": [ + {"name": "SceneName", "type": "STRING", "default": "my_scene"}, + ], + } + info = extract_bundle_info(template, "/path") + assert info.name == "Render {{Param.SceneName}}" + + def test_name_not_resolved_with_parameter_values(self): + template = { + "name": "{{Param.JobName}}", + "steps": [], + "parameterDefinitions": [ + {"name": "JobName", "type": "STRING", "default": "Default Name"}, + ], + } + pv = {"parameterValues": [{"name": "JobName", "value": "Custom Name"}]} + info = extract_bundle_info(template, "/path", pv) + assert info.name == "{{Param.JobName}}" + + def test_name_unresolved_param(self): + template = { + "name": "{{Param.Missing}}", + "steps": [], + "parameterDefinitions": [], + } + info = extract_bundle_info(template, "/path") + assert info.name == "{{Param.Missing}}" + + def test_name_not_resolved_from_pv(self): + """Parameter values don't affect the displayed name.""" + template = { + "name": "{{Param.JobName}}", + "steps": [], + "parameterDefinitions": [], + } + pv = {"parameterValues": [{"name": "JobName", "value": "From PV"}]} + info = extract_bundle_info(template, "/path", pv) + assert info.name == "{{Param.JobName}}" + + +class TestBundleInfoFromS3Metadata: + def test_full_metadata(self): + metadata = { + "ojd-name": "My Bundle", + "ojd-desc": "A description", + "ojd-steps": "Step1,Step2", + "ojd-params": "Frames:STRING,Output:PATH", + } + info = _bundle_info_from_s3_metadata(metadata, "s3://bucket/key") + assert info is not None + assert info.name == "My Bundle" + assert info.description == "A description" + assert info.step_names == ["Step1", "Step2"] + assert len(info.parameters) == 2 + assert info.parameters[0] == {"name": "Frames", "type": "STRING", "_from_metadata": True} + assert info.parameters[1] == {"name": "Output", "type": "PATH", "_from_metadata": True} + + def test_missing_name_returns_none(self): + info = _bundle_info_from_s3_metadata({}, "s3://bucket/key") + assert info is None + + def test_name_only(self): + info = _bundle_info_from_s3_metadata({"ojd-name": "Simple"}, "s3://bucket/key") + assert info is not None + assert info.name == "Simple" + assert info.step_names == [] + assert info.parameters == [] + + def test_oversized_metadata_values_are_capped(self): + """Attacker-influenced S3 metadata must be capped to the PREVIEW_MAX_* limits, + matching the template preview path, so a crafted object can't bloat the preview.""" + metadata = { + "ojd-name": "N" * (PREVIEW_MAX_NAME_LEN + 500), + "ojd-desc": "D" * (PREVIEW_MAX_DESC_LEN + 500), + "ojd-steps": ",".join(f"step{i}" for i in range(PREVIEW_MAX_STEPS + 50)), + "ojd-params": ",".join(f"p{i}:STRING" for i in range(PREVIEW_MAX_PARAMS + 50)), + } + info = _bundle_info_from_s3_metadata(metadata, "s3://bucket/key") + assert info is not None + assert len(info.name) == PREVIEW_MAX_NAME_LEN + assert len(info.description) == PREVIEW_MAX_DESC_LEN + assert len(info.step_names) == PREVIEW_MAX_STEPS + assert len(info.parameters) == PREVIEW_MAX_PARAMS + + def test_oversized_param_name_and_type_are_capped(self): + metadata = { + "ojd-name": "Bundle", + "ojd-params": f"{'x' * 5000}:{'y' * 5000}", + } + info = _bundle_info_from_s3_metadata(metadata, "s3://bucket/key") + assert info is not None + assert len(info.parameters) == 1 + assert len(info.parameters[0]["name"]) <= PREVIEW_MAX_NAME_LEN + assert len(info.parameters[0]["type"]) <= PREVIEW_MAX_NAME_LEN + + +class TestArchiveHelpers: + def test_is_archive(self): + assert _is_archive("bundle.ojd") + assert not _is_archive("bundle.zip") + assert not _is_archive("bundle.tar.gz") + assert not _is_archive("bundle.tgz") + assert not _is_archive("bundle.tar.bz2") + assert not _is_archive("bundle.tar.xz") + assert not _is_archive("bundle.tar") + assert not _is_archive("bundle") + assert not _is_archive("template.yaml") + + def test_strip_archive_ext(self): + assert _strip_archive_ext("bundle.ojd") == "bundle" + assert _strip_archive_ext("my-job.ojd") == "my-job" + assert _strip_archive_ext("noext") == "noext" + + +class TestReadTemplateFromArchive: + def _make_ojd(self, tmp_path, contents: dict[str, str]) -> str: + ojd_path = str(tmp_path / "bundle.ojd") + with zipfile.ZipFile(ojd_path, "w") as zf: + for name, data in contents.items(): + zf.writestr(name, data) + return ojd_path + + def test_ojd_root_template(self, tmp_path): + path = self._make_ojd(tmp_path, {"template.yaml": "name: OjdBundle\nsteps: []\n"}) + result = read_template_from_archive(path) + assert result is not None + raw, fname = result + assert "OjdBundle" in raw + assert fname == "template.yaml" + + def test_ojd_wrapped_template(self, tmp_path): + path = self._make_ojd(tmp_path, {"my-bundle/template.yaml": "name: Wrapped\nsteps: []\n"}) + result = read_template_from_archive(path) + assert result is not None + raw, fname = result + assert "Wrapped" in raw + + def test_ojd_json_template(self, tmp_path): + path = self._make_ojd( + tmp_path, + {"template.json": json.dumps({"name": "JSONBundle", "steps": []})}, + ) + result = read_template_from_archive(path) + assert result is not None + raw, fname = result + assert fname == "template.json" + + def test_ojd_no_template(self, tmp_path): + path = self._make_ojd(tmp_path, {"readme.txt": "no template here"}) + result = read_template_from_archive(path) + assert result is None + + def test_non_zip_ojd_returns_none(self, tmp_path): + """A .ojd that isn't actually a zip (corrupt/renamed/malicious) must be + tolerated on the read/preview path, not raise.""" + path = str(tmp_path / "bogus.ojd") + with open(path, "wb") as f: + f.write(b"this is definitely not a zip archive") + assert read_template_from_archive(path) is None + + def test_truncated_zip_ojd_returns_none(self, tmp_path): + """A file that starts with the zip magic but is truncated/garbage must also + be handled gracefully.""" + path = str(tmp_path / "truncated.ojd") + with open(path, "wb") as f: + f.write(b"PK\x03\x04" + b"\x00" * 8) # zip local-file magic, then garbage + assert read_template_from_archive(path) is None + + +class TestExtractNonZipArchive: + """Extraction must convert a raw zipfile.BadZipFile into a clear ValueError so + a corrupt/renamed/non-zip .ojd surfaces a friendly, catchable error.""" + + def test_extract_archive_rejects_non_zip(self, tmp_path): + path = str(tmp_path / "bogus.ojd") + with open(path, "wb") as f: + f.write(b"not a zip file at all") + dest = str(tmp_path / "out") + with pytest.raises(ValueError, match="not a valid .ojd archive"): + _extract_archive(path, dest) + + def test_extract_archive_from_fileobj_rejects_non_zip(self, tmp_path): + buf = io.BytesIO(b"still not a zip") + dest = str(tmp_path / "out") + with pytest.raises(ValueError, match="not a valid .ojd archive"): + _extract_archive_from_fileobj(buf, dest) + + +class TestLocalBundleRepository: + def test_root_path_default(self): + repo = LocalBundleRepository() + assert repo.root_path() == os.path.expanduser("~") + + def test_root_path_custom(self, tmp_path): + repo = LocalBundleRepository(root=str(tmp_path)) + assert repo.root_path() == str(tmp_path) + + def test_list_entries_empty(self, tmp_path): + repo = LocalBundleRepository(root=str(tmp_path)) + entries = repo.list_entries(str(tmp_path)) + assert entries == [] + + def test_list_entries_with_bundles_and_dirs(self, tmp_path): + bundle_dir = tmp_path / "my-bundle" + bundle_dir.mkdir() + (bundle_dir / "template.yaml").write_text("name: Test Bundle\nsteps:\n- name: Step1\n") + + regular_dir = tmp_path / "regular-dir" + regular_dir.mkdir() + + (tmp_path / "some-file.txt").write_text("not a dir") + + repo = LocalBundleRepository(root=str(tmp_path)) + entries = repo.list_entries(str(tmp_path)) + + assert len(entries) == 2 + names = {e.name for e in entries} + assert "my-bundle" in names + assert "regular-dir" in names + + bundle_entry = next(e for e in entries if e.name == "my-bundle") + assert bundle_entry.is_bundle is True + assert bundle_entry.is_archive is False + + dir_entry = next(e for e in entries if e.name == "regular-dir") + assert dir_entry.is_bundle is False + + def test_list_entries_with_valid_archive(self, tmp_path): + ojd_path = tmp_path / "render-job.ojd" + with zipfile.ZipFile(str(ojd_path), "w") as zf: + zf.writestr("template.yaml", "name: Render\nsteps:\n- name: S1\n") + + repo = LocalBundleRepository(root=str(tmp_path)) + entries = repo.list_entries(str(tmp_path)) + + archive_entries = [e for e in entries if e.is_archive] + assert len(archive_entries) == 1 + assert archive_entries[0].name == "render-job" + assert archive_entries[0].is_bundle is True + + def test_list_entries_invalid_archive_excluded(self, tmp_path): + """An .ojd without a template should not appear as a bundle.""" + ojd_path = tmp_path / "random.ojd" + with zipfile.ZipFile(str(ojd_path), "w") as zf: + zf.writestr("readme.txt", "not a bundle") + + repo = LocalBundleRepository(root=str(tmp_path)) + entries = repo.list_entries(str(tmp_path)) + assert len(entries) == 0 + + def test_list_entries_include_archives_false(self, tmp_path): + """With include_archives=False, archives are skipped entirely.""" + ojd_path = tmp_path / "bundle.ojd" + with zipfile.ZipFile(str(ojd_path), "w") as zf: + zf.writestr("template.yaml", "name: Zipped\nsteps: []\n") + + bundle_dir = tmp_path / "dir-bundle" + bundle_dir.mkdir() + (bundle_dir / "template.yaml").write_text("name: Dir\nsteps: []\n") + + repo = LocalBundleRepository(root=str(tmp_path), include_archives=False) + entries = repo.list_entries(str(tmp_path)) + + assert len(entries) == 1 + assert entries[0].name == "dir-bundle" + + def test_list_entries_nonexistent_path(self): + repo = LocalBundleRepository() + entries = repo.list_entries("/nonexistent/path/that/does/not/exist") + assert entries == [] + + def test_get_bundle_info_yaml(self, tmp_path): + bundle_dir = tmp_path / "test-bundle" + bundle_dir.mkdir() + (bundle_dir / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "Test Bundle", + "description": "A test", + "steps": [{"name": "Render"}], + "parameterDefinitions": [{"name": "Frames", "type": "STRING"}], + } + ) + ) + + repo = LocalBundleRepository(root=str(tmp_path)) + info = repo.get_bundle_info(str(bundle_dir)) + + assert info is not None + assert info.name == "Test Bundle" + assert info.description == "A test" + assert info.step_names == ["Render"] + assert len(info.parameters) == 1 + + def test_get_bundle_info_with_parameter_values(self, tmp_path): + bundle_dir = tmp_path / "pv-bundle" + bundle_dir.mkdir() + (bundle_dir / "template.yaml").write_text( + yaml.dump( + { + "name": "{{Param.JobName}}", + "steps": [{"name": "Run"}], + "parameterDefinitions": [ + {"name": "JobName", "type": "STRING", "default": "Default"}, + {"name": "Frames", "type": "STRING"}, + ], + } + ) + ) + (bundle_dir / "parameter_values.yaml").write_text( + yaml.dump( + { + "parameterValues": [ + {"name": "JobName", "value": "My Custom Job"}, + {"name": "Frames", "value": "1-100"}, + ] + } + ) + ) + + repo = LocalBundleRepository(root=str(tmp_path)) + info = repo.get_bundle_info(str(bundle_dir)) + + assert info is not None + assert info.name == "{{Param.JobName}}" + frames = next(p for p in info.parameters if p["name"] == "Frames") + assert frames["_display_value"] == "1-100" + + def test_get_bundle_info_archive(self, tmp_path): + ojd_path = tmp_path / "my-job.ojd" + with zipfile.ZipFile(str(ojd_path), "w") as zf: + zf.writestr( + "template.yaml", + yaml.dump( + { + "name": "Archive Job", + "description": "From an ojd", + "steps": [{"name": "Run"}], + "parameterDefinitions": [{"name": "Input", "type": "PATH"}], + } + ), + ) + + repo = LocalBundleRepository(root=str(tmp_path)) + info = repo.get_bundle_info(str(ojd_path)) + + assert info is not None + assert info.name == "Archive Job" + assert info.description == "From an ojd" + assert info.step_names == ["Run"] + assert len(info.parameters) == 1 + + def test_get_bundle_info_not_a_bundle(self, tmp_path): + regular_dir = tmp_path / "not-a-bundle" + regular_dir.mkdir() + + repo = LocalBundleRepository(root=str(tmp_path)) + info = repo.get_bundle_info(str(regular_dir)) + assert info is None + + def test_extract_bundle_flat(self, tmp_path): + ojd_path = tmp_path / "flat.ojd" + with zipfile.ZipFile(str(ojd_path), "w") as zf: + zf.writestr("template.yaml", "name: Flat\nsteps: []\n") + zf.writestr("scripts/run.sh", "#!/bin/bash\necho hello\n") + + dest = tmp_path / "extracted" + dest.mkdir() + repo = LocalBundleRepository() + result = repo.extract_bundle(str(ojd_path), str(dest)) + + assert os.path.isfile(os.path.join(result, "template.yaml")) + assert os.path.isfile(os.path.join(result, "scripts", "run.sh")) + + def test_extract_bundle_wrapped(self, tmp_path): + ojd_path = tmp_path / "wrapped.ojd" + with zipfile.ZipFile(str(ojd_path), "w") as zf: + zf.writestr("my-bundle/template.yaml", "name: Wrapped\nsteps: []\n") + zf.writestr("my-bundle/scripts/run.sh", "#!/bin/bash\n") + + dest = tmp_path / "extracted" + dest.mkdir() + repo = LocalBundleRepository() + result = repo.extract_bundle(str(ojd_path), str(dest)) + + assert os.path.isfile(os.path.join(result, "template.yaml")) + + def test_nested_bundles(self, tmp_path): + parent = tmp_path / "projects" + parent.mkdir() + + nested_bundle = parent / "my-job" + nested_bundle.mkdir() + (nested_bundle / "template.yaml").write_text("name: Nested\nsteps:\n- name: S1\n") + + repo = LocalBundleRepository(root=str(tmp_path)) + entries = repo.list_entries(str(parent)) + assert len(entries) == 1 + assert entries[0].is_bundle is True + assert entries[0].name == "my-job" + + def testread_parameter_values_yaml(self, tmp_path): + bundle_dir = tmp_path / "bundle" + bundle_dir.mkdir() + (bundle_dir / "parameter_values.yaml").write_text( + yaml.dump({"parameterValues": [{"name": "X", "value": "1"}]}) + ) + result = LocalBundleRepository.read_parameter_values(str(bundle_dir)) + assert result is not None + assert result["parameterValues"][0]["value"] == "1" + + def testread_parameter_values_json(self, tmp_path): + bundle_dir = tmp_path / "bundle" + bundle_dir.mkdir() + (bundle_dir / "parameter_values.json").write_text( + json.dumps({"parameterValues": [{"name": "Y", "value": "2"}]}) + ) + result = LocalBundleRepository.read_parameter_values(str(bundle_dir)) + assert result is not None + assert result["parameterValues"][0]["value"] == "2" + + def testread_parameter_values_none(self, tmp_path): + bundle_dir = tmp_path / "bundle" + bundle_dir.mkdir() + result = LocalBundleRepository.read_parameter_values(str(bundle_dir)) + assert result is None + + +class TestSafeZipExtract: + def test_rejects_absolute_path(self, tmp_path): + archive = tmp_path / "bad.zip" + with zipfile.ZipFile(str(archive), "w") as zf: + zf.writestr("/etc/passwd", "malicious") + + dest = tmp_path / "out" + dest.mkdir() + with zipfile.ZipFile(str(archive), "r") as zf: + with pytest.raises(ValueError, match="absolute path"): + _safe_zip_extract(zf, str(dest)) + + def test_rejects_parent_directory_traversal(self, tmp_path): + archive = tmp_path / "bad.zip" + with zipfile.ZipFile(str(archive), "w") as zf: + zf.writestr("../../etc/passwd", "malicious") + + dest = tmp_path / "out" + dest.mkdir() + with zipfile.ZipFile(str(archive), "r") as zf: + with pytest.raises(ValueError, match="outside target directory"): + _safe_zip_extract(zf, str(dest)) + + def test_allows_normal_archive(self, tmp_path): + archive = tmp_path / "good.zip" + with zipfile.ZipFile(str(archive), "w") as zf: + zf.writestr("template.yaml", "name: Test\n") + zf.writestr("subdir/file.txt", "hello") + + dest = tmp_path / "out" + dest.mkdir() + with zipfile.ZipFile(str(archive), "r") as zf: + _safe_zip_extract(zf, str(dest)) + + assert (dest / "template.yaml").exists() + assert (dest / "subdir" / "file.txt").exists() + + +class TestSanitizeBundleName: + def test_slashes_replaced(self): + assert sanitize_bundle_name("path/to/bundle") == "path_to_bundle" + + def test_backslashes_replaced_on_windows(self): + if sys.platform == "win32": + assert sanitize_bundle_name("path\\to\\bundle") == "path_to_bundle" + + def test_backslashes_preserved_on_posix(self): + if sys.platform != "win32": + assert sanitize_bundle_name("path\\to\\bundle") == "path\\to\\bundle" + + def test_windows_illegal_chars_replaced_on_windows(self): + if sys.platform == "win32": + assert sanitize_bundle_name("file:name*with?bad") == "file_name_with_bad_chars_" + + def test_colons_preserved_on_posix(self): + if sys.platform != "win32": + assert sanitize_bundle_name("my:bundle") == "my:bundle" + + def test_empty_after_sanitization_raises(self): + with pytest.raises(ValueError, match="empty or unsafe"): + sanitize_bundle_name("///") + + def test_long_name_preserved(self): + long_name = "a" * 1000 + assert sanitize_bundle_name(long_name) == long_name + + def test_normal_name_unchanged(self): + assert sanitize_bundle_name("blender-render_v2.1") == "blender-render_v2.1" + + +class TestS3BundleVisibility: + """Bundle visibility is a local, per-user view stored in a file — hide/unhide + changes only this user's listing and never touches S3.""" + + def _make_repo(self, tmp_path, monkeypatch, bucket="test-bucket"): + # Point the bundle cache at a temp location so tests don't touch ~/.deadline. + monkeypatch.setattr( + "deadline.client.job_bundle.repository.get_bundle_cache_dir", + lambda: str(tmp_path / "cache"), + ) + with patch("boto3.Session"): + repo = S3BundleRepository( + bucket_name=bucket, + root_prefix="DeadlineCloud", + session=MagicMock(), + ) + repo._s3 = MagicMock() + return repo + + def test_empty_when_nothing_hidden(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + assert repo.get_hidden_set() == set() + # Reading visibility must not touch S3. + repo._s3.head_object.assert_not_called() + repo._s3.get_paginator.assert_not_called() + + def test_hide_then_read_roundtrip(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + repo.set_bundle_visibility("bundle-a", hidden=True) + repo.set_bundle_visibility("rendering/bundle-c", hidden=True) + assert repo.get_hidden_set() == {"bundle-a", "rendering/bundle-c"} + # Hide is purely local — no S3 calls. + repo._s3.head_object.assert_not_called() + repo._s3.copy_object.assert_not_called() + + def test_unhide_removes_from_set(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + repo.set_bundle_visibility("bundle-a", hidden=True) + repo.set_bundle_visibility("bundle-b", hidden=True) + repo.set_bundle_visibility("bundle-a", hidden=False) + assert repo.get_hidden_set() == {"bundle-b"} + + def test_view_persists_across_instances(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + repo.set_bundle_visibility("bundle-a", hidden=True) + # A fresh repo for the same queue sees the persisted view. + repo2 = self._make_repo(tmp_path, monkeypatch) + assert repo2.get_hidden_set() == {"bundle-a"} + + def test_view_file_is_versioned_and_sorted(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + for name in ("z-bundle", "a-bundle", "m-bundle"): + repo.set_bundle_visibility(name, hidden=True) + view_files = list((tmp_path / "cache").glob("*/.visibility.json")) + assert len(view_files) == 1 + data = json.loads(view_files[0].read_text()) + assert data["hidden"] == ["a-bundle", "m-bundle", "z-bundle"] + assert data["version"] == 1 + + def test_hide_noop_when_already_hidden(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + repo.set_bundle_visibility("bundle-a", hidden=True) + repo.set_bundle_visibility("bundle-a", hidden=True) + assert repo.get_hidden_set() == {"bundle-a"} + + def test_unhide_noop_when_not_hidden(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + repo.set_bundle_visibility("bundle-a", hidden=False) + assert repo.get_hidden_set() == set() + + def test_view_is_per_queue(self, tmp_path, monkeypatch): + repo_a = self._make_repo(tmp_path, monkeypatch, bucket="bucket-a") + repo_b = self._make_repo(tmp_path, monkeypatch, bucket="bucket-b") + repo_a.set_bundle_visibility("x", hidden=True) + repo_b.set_bundle_visibility("y", hidden=True) + assert repo_a.get_hidden_set() == {"x"} + # A different queue (bucket) has an independent hidden set and its own file. + assert repo_b.get_hidden_set() == {"y"} + view_files = list((tmp_path / "cache").glob("*/.visibility.json")) + assert len(view_files) == 2 # one per queue + + def test_malformed_view_file_is_ignored(self, tmp_path, monkeypatch): + repo = self._make_repo(tmp_path, monkeypatch) + repo.set_bundle_visibility("bundle-a", hidden=True) # creates the file + view_file = next((tmp_path / "cache").glob("*/.visibility.json")) + view_file.write_text("{ not valid json") + assert repo.get_hidden_set() == set() + + +class TestMakeS3Client: + """The S3 client's connection pool is sized to cover the background preview + prefetch's parallel HEADs (and managed transfers), so urllib3 doesn't warn.""" + + def _pool(self, mock_default_config): + return mock_default_config.call_args[1]["max_pool_connections"] + + @patch("deadline.client.api._session.get_default_client_config") + @patch("deadline.client.job_bundle.repository.config_file") + def test_pool_uses_larger_configured_setting(self, mock_config_file, mock_default_config): + mock_config_file.get_setting.return_value = "50" + session = MagicMock() + _make_s3_client(session) + assert self._pool(mock_default_config) == 50 + session.client.assert_called_once() + + @patch("deadline.client.api._session.get_default_client_config") + @patch("deadline.client.job_bundle.repository.config_file") + def test_pool_covers_prefetch_workers_when_setting_is_small( + self, mock_config_file, mock_default_config + ): + mock_config_file.get_setting.return_value = "4" + _make_s3_client(MagicMock()) + assert self._pool(mock_default_config) == PREVIEW_PREFETCH_MAX_WORKERS + + @patch("deadline.client.api._session.get_default_client_config") + @patch("deadline.client.job_bundle.repository.config_file") + def test_pool_falls_back_when_setting_unparseable(self, mock_config_file, mock_default_config): + mock_config_file.get_setting.return_value = "not-a-number" + _make_s3_client(MagicMock()) + assert self._pool(mock_default_config) == PREVIEW_PREFETCH_MAX_WORKERS + + +class TestPrefetchPreviews: + """prefetch_previews warms _head_cache with parallel HEADs so preview/size/download + reuse them; it's a background optimization decoupled from visibility.""" + + def _make_repo(self): + with patch("boto3.Session"): + repo = S3BundleRepository("test-bucket", "DeadlineCloud", session=MagicMock()) + repo._s3 = MagicMock() + return repo + + def _set_listing(self, repo, keys): + paginator = MagicMock() + paginator.paginate.return_value = [{"Contents": [{"Key": k} for k in keys]}] + repo._s3.get_paginator.return_value = paginator + + def test_prefetch_all_warms_cache(self): + repo = self._make_repo() + prefix = repo._prefix + self._set_listing( + repo, + [f"{prefix}a.ojd", f"{prefix}rendering/b.ojd", f"{prefix}notes.txt"], + ) + repo._s3.head_object.side_effect = lambda Bucket, Key: {"ETag": f'"{Key}"', "Metadata": {}} + + repo.prefetch_previews() + + # Only .ojd objects are prefetched. + assert set(repo._head_cache) == {f"{prefix}a.ojd", f"{prefix}rendering/b.ojd"} + assert repo._s3.head_object.call_count == 2 + + def test_prefetch_rebuilds_and_drops_stale(self): + repo = self._make_repo() + prefix = repo._prefix + repo._head_cache[f"{prefix}deleted.ojd"] = {"ETag": '"old"'} + self._set_listing(repo, [f"{prefix}still.ojd"]) + repo._s3.head_object.return_value = {"ETag": '"new"', "Metadata": {}} + + repo.prefetch_previews() + + assert f"{prefix}deleted.ojd" not in repo._head_cache + assert f"{prefix}still.ojd" in repo._head_cache + + def test_prefetch_tolerates_head_failures(self): + repo = self._make_repo() + prefix = repo._prefix + self._set_listing(repo, [f"{prefix}good.ojd", f"{prefix}bad.ojd"]) + + def _head(Bucket, Key): + if Key.endswith("bad.ojd"): + raise ClientError({"Error": {"Code": "AccessDenied"}}, "HeadObject") + return {"ETag": '"e"', "Metadata": {}} + + repo._s3.head_object.side_effect = _head + repo.prefetch_previews() # must not raise + assert f"{prefix}good.ojd" in repo._head_cache + assert f"{prefix}bad.ojd" not in repo._head_cache + + def test_preview_reuses_prefetched_head(self): + repo = self._make_repo() + key = f"{repo._prefix}blender.ojd" + repo._head_cache[key] = { + "ETag": '"e1"', + "ContentLength": 4096, + "Metadata": {"ojd-name": "Blender", "ojd-steps": "Render"}, + } + + info = repo.get_bundle_info(f"s3://test-bucket/{key}") + + repo._s3.head_object.assert_not_called() + repo._s3.get_object.assert_not_called() + assert info is not None + assert info.name == "Blender" + assert info.size_bytes == 4096 + + def test_get_bundle_size_reuses_prefetched_head(self): + repo = self._make_repo() + key = f"{repo._prefix}blender.ojd" + repo._head_cache[key] = {"ETag": '"e1"', "ContentLength": 9999, "Metadata": {}} + assert repo.get_bundle_size(f"s3://test-bucket/{key}") == 9999 + repo._s3.head_object.assert_not_called() + + +class TestArchiveBundleDir: + """Tests for archive_bundle_dir — validates archiving produces correct zip content + and progress callbacks report accurate byte counts.""" + + def test_archives_all_files(self, tmp_path): + """All files in the bundle directory end up in the archive.""" + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: Test\nsteps: []\n") + (bundle / "script.sh").write_text("#!/bin/bash\necho hello\n") + (bundle / "subdir").mkdir() + (bundle / "subdir" / "data.json").write_text('{"key": "value"}') + + buf = archive_bundle_dir(str(bundle)) + + with zipfile.ZipFile(buf, "r") as zf: + names = sorted(zf.namelist()) + assert "template.yaml" in names + assert "script.sh" in names + assert "subdir/data.json" in names + + def test_progress_reports_total_bytes(self, tmp_path): + """Progress callback receives total bytes equal to the source directory size.""" + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: T\nsteps: []\n") + (bundle / "big.bin").write_bytes(b"x" * 1000) + + reported = [] + archive_bundle_dir(str(bundle), progress_callback=lambda n: reported.append(n)) + + assert sum(reported) == get_bundle_dir_size(str(bundle)) + + def test_skips_symlinks(self, tmp_path): + """Symlinked files are not included in the archive.""" + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text("name: T\nsteps: []\n") + target = tmp_path / "outside.txt" + target.write_text("secret") + (bundle / "link.txt").symlink_to(target) + + buf = archive_bundle_dir(str(bundle)) + + with zipfile.ZipFile(buf, "r") as zf: + assert "link.txt" not in zf.namelist() + + +class TestGetBundleDirSize: + def test_returns_total_file_size(self, tmp_path): + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "a.txt").write_bytes(b"x" * 100) + (bundle / "b.txt").write_bytes(b"y" * 200) + + assert get_bundle_dir_size(str(bundle)) == 300 + + +class TestSafeZipExtractWithProgress: + """Tests that _safe_zip_extract progress callback reports correct sizes.""" + + def test_progress_reports_uncompressed_sizes(self, tmp_path): + """Each callback receives the uncompressed file size.""" + archive = tmp_path / "test.zip" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("small.txt", "hello") # 5 bytes + zf.writestr("bigger.txt", "x" * 100) # 100 bytes + + reported = [] + with zipfile.ZipFile(archive, "r") as zf: + _safe_zip_extract( + zf, str(tmp_path / "out"), progress_callback=lambda n: reported.append(n) + ) + + assert sorted(reported) == [5, 100] + + def test_size_callback_reports_total(self, tmp_path): + """size_callback is called once with total uncompressed size before extraction.""" + archive = tmp_path / "test.zip" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("a.txt", "aaa") + zf.writestr("b.txt", "bbbbb") + + size_reports = [] + with zipfile.ZipFile(archive, "r") as zf: + _safe_zip_extract( + zf, + str(tmp_path / "out"), + progress_callback=lambda n: None, + size_callback=lambda t: size_reports.append(t), + ) + + assert size_reports == [8] # 3 + 5 + + +class TestFromConfig: + """Tests for S3BundleRepository.from_config — validates the parallel initialization path.""" + + @patch("deadline.client.api.get_queue_user_boto3_session") + @patch("deadline.client.api.get_boto3_client") + @patch("deadline.client.api.get_boto3_session") + @patch("deadline.client.job_bundle.repository.config_file") + def test_creates_repo_with_correct_bucket_and_prefix( + self, mock_config_file, mock_get_session, mock_get_client, mock_get_queue_session + ): + """from_config returns a repo with bucket/prefix from GetQueue response.""" + mock_config_file.get_setting.side_effect = lambda key, config=None: { + "defaults.farm_id": "farm-123", + "defaults.queue_id": "queue-456", + }.get(key, "") + + mock_deadline_client = MagicMock() + mock_deadline_client.get_queue.return_value = { + "jobAttachmentSettings": { + "s3BucketName": "my-bucket", + "rootPrefix": "DeadlineCloud", + }, + } + mock_get_client.return_value = mock_deadline_client + + mock_s3_session = MagicMock() + mock_s3_client = MagicMock() + mock_s3_session.client.return_value = mock_s3_client + mock_get_queue_session.return_value = mock_s3_session + + repo = S3BundleRepository.from_config() + + assert repo._bucket == "my-bucket" + assert "job-bundles" in repo._prefix + assert repo._s3 is mock_s3_client + mock_deadline_client.get_queue.assert_called_once_with( + farmId="farm-123", queueId="queue-456" + ) + + @patch("deadline.client.job_bundle.repository.config_file") + def test_raises_without_farm_or_queue(self, mock_config_file): + """from_config raises when farm/queue IDs are not configured.""" + mock_config_file.get_setting.return_value = "" + with pytest.raises(DeadlineOperationError, match="farm and queue"): + S3BundleRepository.from_config() + + @patch("deadline.client.api.get_queue_user_boto3_session") + @patch("deadline.client.api.get_boto3_client") + @patch("deadline.client.api.get_boto3_session") + @patch("deadline.client.job_bundle.repository.config_file") + def test_raises_without_attachment_settings( + self, mock_config_file, mock_get_session, mock_get_client, mock_get_queue_session + ): + """from_config raises when queue has no job attachment settings.""" + mock_config_file.get_setting.side_effect = lambda key, config=None: { + "defaults.farm_id": "farm-123", + "defaults.queue_id": "queue-456", + }.get(key, "") + + mock_deadline_client = MagicMock() + mock_deadline_client.get_queue.return_value = {} + mock_get_client.return_value = mock_deadline_client + mock_get_queue_session.return_value = MagicMock() + + with pytest.raises(DeadlineOperationError, match="attachment settings"): + S3BundleRepository.from_config() + + +class TestS3ListEntries: + """Tests for S3BundleRepository.list_entries — validates S3 listing logic.""" + + def _make_repo(self): + with patch("boto3.Session"): + repo = S3BundleRepository( + bucket_name="test-bucket", root_prefix="DC", session=MagicMock() + ) + repo._s3 = MagicMock() + return repo + + def test_lists_ojd_files_as_bundles(self): + repo = self._make_repo() + paginator = MagicMock() + paginator.paginate.return_value = [ + { + "CommonPrefixes": [], + "Contents": [ + {"Key": "DC/job-bundles/render.ojd"}, + {"Key": "DC/job-bundles/sim.ojd"}, + ], + } + ] + repo._s3.get_paginator.return_value = paginator + + entries = repo.list_entries(repo.root_path()) + + bundle_names = [e.name for e in entries if e.is_bundle] + assert sorted(bundle_names) == ["render", "sim"] + assert all(e.is_archive for e in entries if e.is_bundle) + + def test_lists_subfolders_as_non_bundles(self): + repo = self._make_repo() + paginator = MagicMock() + paginator.paginate.return_value = [ + { + "CommonPrefixes": [{"Prefix": "DC/job-bundles/rendering/"}], + "Contents": [], + } + ] + repo._s3.get_paginator.return_value = paginator + + entries = repo.list_entries(repo.root_path()) + + assert len(entries) == 1 + assert entries[0].name == "rendering" + assert entries[0].is_bundle is False + + def test_ignores_non_ojd_files(self): + repo = self._make_repo() + paginator = MagicMock() + paginator.paginate.return_value = [ + { + "CommonPrefixes": [], + "Contents": [ + {"Key": "DC/job-bundles/readme.txt"}, + {"Key": "DC/job-bundles/valid.ojd"}, + ], + } + ] + repo._s3.get_paginator.return_value = paginator + + entries = repo.list_entries(repo.root_path()) + + assert len(entries) == 1 + assert entries[0].name == "valid" + + +class TestResolveArchiveBundle: + """Tests for _resolve_archive_bundle — validates download, cache, and extract flow.""" + + def _make_repo(self, fresh_deadline_config): + with patch("boto3.Session"): + repo = S3BundleRepository( + bucket_name="test-bucket", root_prefix="DC", session=MagicMock() + ) + repo._s3 = MagicMock() + return repo + + def _make_ojd_bytes(self): + """Create a minimal .ojd archive in memory.""" + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + zf.writestr("template.yaml", "name: TestBundle\nsteps:\n- name: Step1\n") + return buf.getvalue() + + def test_downloads_and_extracts_to_cache(self, fresh_deadline_config, tmp_path): + repo = self._make_repo(fresh_deadline_config) + ojd_data = self._make_ojd_bytes() + + repo._s3.head_object.return_value = {"ETag": '"abc"', "ContentLength": len(ojd_data)} + repo._s3.download_fileobj.side_effect = lambda Fileobj, **kw: Fileobj.write(ojd_data) + + path = "s3://test-bucket/DC/job-bundles/test.ojd" + result = repo._resolve_archive_bundle(path) + + assert os.path.isfile(os.path.join(result, "template.yaml")) + + def test_uses_cache_on_matching_etag(self, fresh_deadline_config, tmp_path): + repo = self._make_repo(fresh_deadline_config) + ojd_data = self._make_ojd_bytes() + + repo._s3.head_object.return_value = {"ETag": '"abc"', "ContentLength": len(ojd_data)} + repo._s3.download_fileobj.side_effect = lambda Fileobj, **kw: Fileobj.write(ojd_data) + + path = "s3://test-bucket/DC/job-bundles/cached.ojd" + + # First call downloads + result1 = repo._resolve_archive_bundle(path) + assert repo._s3.download_fileobj.call_count == 1 + + # Second call uses cache (same ETag) + result2 = repo._resolve_archive_bundle(path) + assert repo._s3.download_fileobj.call_count == 1 # No additional download + assert result1 == result2 + + def test_calls_progress_callbacks(self, fresh_deadline_config): + repo = self._make_repo(fresh_deadline_config) + ojd_data = self._make_ojd_bytes() + + repo._s3.head_object.return_value = {"ETag": '"new"', "ContentLength": len(ojd_data)} + repo._s3.download_fileobj.side_effect = lambda Fileobj, **kw: Fileobj.write(ojd_data) + + dl_progress = [] + ex_progress = [] + size_reports = [] + + path = "s3://test-bucket/DC/job-bundles/progress.ojd" + repo._resolve_archive_bundle( + path, + progress_callback=lambda n: dl_progress.append(n), + extract_callback=lambda n: ex_progress.append(n), + extract_size_callback=lambda t: size_reports.append(t), + ) + + # Download callback is called (by download_fileobj via Callback kwarg) + # Extract callback receives file sizes + assert len(ex_progress) > 0 + assert len(size_reports) == 1 + assert size_reports[0] > 0 + + +class TestArchiveExtractionSafety: + """_check_archive_extraction_safety guards against zip bombs, excessive entry + counts, and extractions that wouldn't fit on disk — always enforced.""" + + def _fake_zip(self, entries): + """entries: list of (file_size, compress_size) tuples.""" + zf = MagicMock() + infos = [] + for file_size, compress_size in entries: + zi = MagicMock() + zi.file_size = file_size + zi.compress_size = compress_size + infos.append(zi) + zf.infolist.return_value = infos + return zf + + def _ample_disk(self): + return patch( + "deadline.client.job_bundle.repository.shutil.disk_usage", + return_value=MagicMock(free=100 * 1024**3), + ) + + def test_rejects_zip_bomb_high_ratio(self, tmp_path): + # 1 GB uncompressed from 1 KB compressed — ratio far above the limit. + zf = self._fake_zip([(1024 * 1024 * 1024, 1024)]) + with pytest.raises(ValueError, match="zip bomb"): + _check_archive_extraction_safety(zf, str(tmp_path)) + + def test_rejects_too_many_entries(self, tmp_path): + zf = self._fake_zip([(10, 10)] * (MAX_ARCHIVE_ENTRIES + 1)) + with pytest.raises(ValueError, match="too many entries"): + _check_archive_extraction_safety(zf, str(tmp_path)) + + def test_allows_small_high_ratio_within_floor(self, tmp_path): + # 100 MB (< 256 MB floor) from tiny compressed — allowed despite high ratio. + zf = self._fake_zip([(100 * 1024 * 1024, 100)]) + with self._ample_disk(): + _check_archive_extraction_safety(zf, str(tmp_path)) # must not raise + + def test_allows_large_low_ratio_bundle(self, tmp_path): + # 1 GB uncompressed from 900 MB compressed (real data) — passes the ratio. + zf = self._fake_zip([(1024**3, 900 * 1024 * 1024)]) + with self._ample_disk(): + _check_archive_extraction_safety(zf, str(tmp_path)) # must not raise + + def test_blocks_when_insufficient_disk(self, tmp_path): + # Low ratio (passes bomb check) but larger than the free space available. + zf = self._fake_zip([(100 * 1024 * 1024, 60 * 1024 * 1024)]) + with patch( + "deadline.client.job_bundle.repository.shutil.disk_usage", + return_value=MagicMock(free=1024), # only 1 KB free + ): + with pytest.raises(ValueError, match="disk space"): + _check_archive_extraction_safety(zf, str(tmp_path)) + + def test_safe_zip_extract_rejects_bomb_end_to_end(self, tmp_path): + # A real, highly compressible archive; lower the floor so it trips the + # ratio check, proving _safe_zip_extract enforces the guard on real zips. + archive = tmp_path / "bomb.ojd" + with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf: + zf.writestr("payload.bin", b"\x00" * (2 * 1024 * 1024)) # 2 MB of zeros + dest = tmp_path / "out" + dest.mkdir() + with patch("deadline.client.job_bundle.repository.MAX_ARCHIVE_UNCOMPRESSED_FLOOR", 1024): + with zipfile.ZipFile(archive, "r") as zf: + with pytest.raises(ValueError, match="zip bomb"): + _safe_zip_extract(zf, str(dest)) + + +class TestTemplateReadCap: + """_read_template_from_zip refuses to read an implausibly large template.""" + + def test_rejects_oversized_template(self, tmp_path): + archive = tmp_path / "b.ojd" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("template.yaml", "name: T\nsteps: []\n") + with patch("deadline.client.job_bundle.repository.MAX_TEMPLATE_BYTES", 5): + # read_template_from_archive swallows the error and returns None. + assert read_template_from_archive(str(archive)) is None + + def test_reads_normal_template(self, tmp_path): + archive = tmp_path / "b.ojd" + with zipfile.ZipFile(archive, "w") as zf: + zf.writestr("template.yaml", "name: T\nsteps: []\n") + result = read_template_from_archive(str(archive)) + assert result is not None + assert result[1] == "template.yaml" + + +class TestDownloadSink: + """_open_download_sink chooses memory vs. a temp file based on size, and spills + large downloads to disk in the cache dir (not shared /tmp).""" + + def test_small_download_stays_in_memory(self, tmp_path): + with patch( + "deadline.client.job_bundle.repository.get_bundle_cache_dir", + return_value=str(tmp_path), + ): + sink = _open_download_sink(1024) + try: + assert isinstance(sink, io.BytesIO) + finally: + sink.close() + + def test_large_download_spills_to_temp_file_in_cache_dir(self, tmp_path): + cache = tmp_path / "cache" + with patch( + "deadline.client.job_bundle.repository.get_bundle_cache_dir", + return_value=str(cache), + ): + sink = _open_download_sink(_DOWNLOAD_SPOOL_THRESHOLD + 1) + try: + # A real on-disk temp file (not an in-memory buffer), created under the + # cache dir so it shares the extraction filesystem and stays off /tmp. + assert not isinstance(sink, io.BytesIO) + assert sink.fileno() >= 0 # backed by a real file descriptor + assert cache.is_dir() # cache dir was created for the spill + finally: + sink.close() + + def test_zero_size_hint_stays_in_memory(self, tmp_path): + # Unknown size (no ContentLength) must not force a temp file. + with patch( + "deadline.client.job_bundle.repository.get_bundle_cache_dir", + return_value=str(tmp_path), + ): + sink = _open_download_sink(0) + try: + assert isinstance(sink, io.BytesIO) + finally: + sink.close() + + +class TestS3GetBundleInfo: + """Tests for S3BundleRepository.get_bundle_info — validates metadata preview path.""" + + def _make_repo(self): + with patch("boto3.Session"): + repo = S3BundleRepository( + bucket_name="test-bucket", root_prefix="DC", session=MagicMock() + ) + repo._s3 = MagicMock() + return repo + + def test_returns_info_from_s3_metadata(self): + """When head_object returns bundle metadata, uses it without downloading.""" + repo = self._make_repo() + repo._s3.head_object.return_value = { + "ETag": '"abc"', + "Metadata": { + "ojd-name": "Preview Bundle", + "ojd-desc": "A description", + "ojd-steps": "Step1", + "ojd-params": "Frames:STRING", + }, + } + + info = repo.get_bundle_info("s3://test-bucket/DC/job-bundles/preview.ojd") + + assert info is not None + assert info.name == "Preview Bundle" + assert info.description == "A description" + assert info.step_names == ["Step1"] + # No download needed + repo._s3.get_object.assert_not_called() + + def test_size_bytes_populated_from_head_content_length(self): + """The archive size from head_object is stamped onto BundleInfo so the UI + can show how much a Queue download will transfer, without an extra call.""" + repo = self._make_repo() + repo._s3.head_object.return_value = { + "ETag": '"abc"', + "ContentLength": 12_345_678, + "Metadata": {"ojd-name": "Sized Bundle"}, + } + + info = repo.get_bundle_info("s3://test-bucket/DC/job-bundles/sized.ojd") + + assert info is not None + assert info.size_bytes == 12_345_678 + # Size came from the (already-performed) HEAD — no extra download. + repo._s3.get_object.assert_not_called() + + def test_falls_back_to_download_when_no_metadata(self, fresh_deadline_config): + """When head_object has no bundle metadata, downloads and parses the archive.""" + repo = self._make_repo() + repo._s3.head_object.return_value = {"ETag": '"xyz"', "Metadata": {}} + + # Provide a real archive via get_object + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + zf.writestr("template.yaml", "name: Downloaded\nsteps:\n- name: S1\n") + + repo._s3.get_object.return_value = { + # A BytesIO is a faithful stand-in for a botocore StreamingBody: + # read(n) returns chunks and b"" at EOF, so streaming (copyfileobj) + # terminates. A MagicMock with a fixed read() return value would loop + # forever. + "Body": io.BytesIO(buf.getvalue()), + "ETag": '"xyz"', + "LastModified": "2026-01-01", + } + + info = repo.get_bundle_info("s3://test-bucket/DC/job-bundles/fallback.ojd") + + assert info is not None + assert info.name == "Downloaded" + assert info.step_names == ["S1"] + + def test_large_archive_download_streams_via_temp_file(self, fresh_deadline_config): + """A large ContentLength routes the download through a temp-file sink and + still extracts/parses correctly (exercises the spill-to-disk branch).""" + repo = self._make_repo() + repo._s3.head_object.return_value = { + "ETag": '"xyz"', + "Metadata": {}, + "ContentLength": _DOWNLOAD_SPOOL_THRESHOLD + 1, + } + + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + zf.writestr("template.yaml", "name: BigBundle\nsteps:\n- name: S1\n") + + repo._s3.get_object.return_value = { + "Body": io.BytesIO(buf.getvalue()), + "ETag": '"xyz"', + "LastModified": "2026-01-01", + } + + info = repo.get_bundle_info("s3://test-bucket/DC/job-bundles/big.ojd") + + assert info is not None + assert info.name == "BigBundle" + assert info.step_names == ["S1"] + + +class TestBuildBundleMetadata: + """Tests for build_bundle_metadata — validates S3 metadata extraction from bundle dirs.""" + + def test_extracts_name_and_description(self, tmp_path): + + bundle = tmp_path / "my-bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "Test Bundle", + "description": "A test description", + "steps": [{"name": "Step1"}], + } + ) + ) + + metadata = build_bundle_metadata(str(bundle)) + + assert metadata["ojd-name"] == "Test Bundle" + assert metadata["ojd-desc"] == "A test description" + assert metadata["ojd-steps"] == "Step1" + + def test_overrides_name(self, tmp_path): + + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text(yaml.dump({"name": "Original", "steps": []})) + + metadata = build_bundle_metadata(str(bundle), bundle_name="Override") + + assert metadata["ojd-name"] == "Override" + + def test_returns_empty_for_missing_template(self, tmp_path): + + empty = tmp_path / "empty" + empty.mkdir() + + assert build_bundle_metadata(str(empty)) == {} + + def test_truncates_long_values(self, tmp_path): + + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text(yaml.dump({"name": "A" * 300, "steps": []})) + + metadata = build_bundle_metadata(str(bundle)) + + assert len(metadata["ojd-name"].encode("utf-8")) <= METADATA_LIMIT_NAME + assert metadata["ojd-name"].endswith("...") + + def test_metadata_values_are_ascii_safe_for_non_ascii_template(self, tmp_path): + """Non-ASCII template fields must round-trip through ASCII-safe metadata. + + S3 user metadata is sent as x-amz-meta-* HTTP headers, and botocore + rejects any non-ASCII value with a ParamValidationError, so a bundle + whose name/description/steps/params contain Japanese, accented Latin, or + emoji characters would previously fail to upload entirely. Every stored + metadata value must be ASCII-safe, and must decode back to the original + Unicode so previews are accurate. + """ + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "レンダリング", # Japanese + "description": "Café rendering job 🎬", # accent + emoji + "steps": [{"name": "描画ステップ"}], + "parameterDefinitions": [ + {"name": "出力先", "type": "PATH"}, + ], + }, + allow_unicode=True, + ) + ) + + metadata = build_bundle_metadata(str(bundle)) + + # Every stored value must be ASCII (header-safe) so the upload succeeds. + for key, value in metadata.items(): + value.encode("ascii") # must not raise + + # ...and it must decode back to the original Unicode for previews. + info = _bundle_info_from_s3_metadata(metadata, "s3://bucket/key.ojd") + assert info is not None + assert info.name == "レンダリング" + assert info.description == "Café rendering job 🎬" + assert info.step_names == ["描画ステップ"] + assert info.parameters[0]["name"] == "出力先" + assert info.parameters[0]["type"] == "PATH" + + def test_large_non_ascii_bundle_stays_within_total_budget(self, tmp_path): + """base64 encoding expands non-ASCII ~4/3 (+12B wrapper per field), so verify + the per-field limits and 2 KB total budget still hold for a large all-Japanese + bundle. The budget is enforced on the encoded (on-wire) values, so no numeric + limit change was needed — this proves that. + """ + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump( + { + "specificationVersion": "jobtemplate-2023-09", + "name": "あ" * 300, + "description": "い" * 1000, + "steps": [{"name": f"ステップ{i:03d}描画"} for i in range(40)], + "parameterDefinitions": [ + {"name": f"パラメータ{i:03d}", "type": "STRING"} for i in range(50) + ], + }, + allow_unicode=True, + ) + ) + + metadata = build_bundle_metadata(str(bundle)) + + # Every stored value is header-safe. + for value in metadata.values(): + value.encode("ascii") # must not raise + # Per-field encoded limit respected. + assert len(metadata["ojd-name"].encode("utf-8")) <= METADATA_LIMIT_NAME + # Total encoded metadata (what actually goes on the wire) within S3 budget. + total = sum( + 12 + len(k.encode("utf-8")) + len(v.encode("utf-8")) for k, v in metadata.items() + ) + assert total <= S3_METADATA_TOTAL_BUDGET + + +class TestEncodeDecodeS3Value: + """Tests for the S3-metadata encode/decode round-trip (_encode/_decode/_truncate).""" + + def test_ascii_value_unchanged(self): + # ASCII values are stored verbatim (backward compatible with old uploads). + assert _encode_s3_value("hello world") == "hello world" + assert _decode_s3_value("hello world") == "hello world" + + def test_non_ascii_is_encoded_to_ascii(self): + for value in ("日本", "café", "job🎬", "Café job 🎬", "出力先:PATH"): + encoded = _encode_s3_value(value) + encoded.encode("ascii") # must not raise — header-safe + assert encoded != value + + def test_round_trips_non_ascii(self): + for value in ("レンダリング", "café", "job🎬", "Step1,描画ステップ", "出力先:PATH"): + assert _decode_s3_value(_encode_s3_value(value)) == value + + def test_decode_of_plain_ascii_is_noop(self): + # Older uploads / bundles stored via other means have plain ASCII values. + assert _decode_s3_value("Blender Render") == "Blender Render" + + def test_decode_handles_foreign_encoded_word_variants(self): + """Decoding must handle any valid RFC 2047 encoded-word, not just this + module's exact output — bundles uploaded by other clients or earlier + builds may use uppercase charset, folded multi-words, or Q-encoding. + """ + # Uppercase charset + Base64 (as produced by email.header). + assert _decode_s3_value("=?UTF-8?B?44Os44Oz44OA44Oq44Oz44Kw?=") == "レンダリング" + # Folded into multiple space-separated encoded-words. + folded = ( + "=?UTF-8?B?5pel5pys6Kqe44Gu44OG44K544OI55So44K444On?= " + "=?UTF-8?B?44OW44OQ44Oz44OJ44Or44Gn44GZ44CC?=" + ) + assert _decode_s3_value(folded) == "日本語のテスト用ジョブバンドルです。" + # Quoted-printable (Q) encoding. + assert _decode_s3_value("=?UTF-8?Q?Caf=C3=A9?=") == "Café" + + def test_truncate_keeps_encoded_value_within_byte_limit(self): + value = "あ" * 200 # long, all non-ASCII + result = _truncate_s3_value(value, METADATA_LIMIT_NAME, field="ojd-name") + result.encode("ascii") # header-safe + assert len(result.encode("utf-8")) <= METADATA_LIMIT_NAME + assert result.endswith("...") + + def test_truncated_non_ascii_value_still_round_trips_prefix(self): + value = "描画" * 100 + result = _truncate_s3_value(value, METADATA_LIMIT_NAME, field="ojd-steps") + decoded = _decode_s3_value(result) + # Decodes to a prefix of the original followed by the truncation marker. + assert decoded.endswith("...") + prefix = decoded[:-3] + assert prefix and value.startswith(prefix) + + def test_encoded_length_matches_exact_formula(self): + """Encoded length is exactly 12 + 4*ceil(N/3) — the guarantee that lets + truncation size the cut in one shot without a retry loop.""" + for value in ("あ", "café", "レンダリング", "job🎬", "描画ステップ" * 5): + encoded = _encode_s3_value(value) + n = len(value.encode("utf-8")) + assert len(encoded) == 12 + 4 * math.ceil(n / 3) + + def test_one_shot_truncation_never_exceeds_limit(self): + """The computed truncation must fit the limit for every limit, with no + retry — proves the exact-math approach is correct across the range.""" + non_ascii = "描画ステップ" * 50 + ascii_value = "A" * 400 + mixed = ("hello " + "日本") * 40 + for value in (non_ascii, ascii_value, mixed): + for limit in range(16, 400): + result = _truncate_s3_value(value, limit) + assert len(result.encode("utf-8")) <= limit, (value[:5], limit) + result.encode("ascii") # header-safe + _decode_s3_value(result) # must not raise + + +class TestS3BundleExists: + def _make_repo(self): + with patch("boto3.Session"): + repo = S3BundleRepository( + bucket_name="test-bucket", root_prefix="DC", session=MagicMock() + ) + repo._s3 = MagicMock() + return repo + + def test_returns_true_when_exists(self): + repo = self._make_repo() + repo._s3.head_object.return_value = {"ETag": '"abc"'} + + assert repo.bundle_exists("my-bundle") is True + repo._s3.head_object.assert_called_once_with( + Bucket="test-bucket", Key="DC/job-bundles/my-bundle.ojd" + ) + + def test_returns_false_when_not_found(self): + repo = self._make_repo() + repo._s3.head_object.side_effect = ClientError({"Error": {"Code": "404"}}, "HeadObject") + + assert repo.bundle_exists("missing") is False + + +class TestS3UploadArchive: + def _make_repo(self): + with patch("boto3.Session"): + repo = S3BundleRepository( + bucket_name="test-bucket", root_prefix="DC", session=MagicMock() + ) + repo._s3 = MagicMock() + return repo + + def test_uploads_with_metadata(self): + repo = self._make_repo() + buf = io.BytesIO(b"fake archive data") + + result = repo.upload_archive(buf, "test-bundle", metadata={"ojd-name": "Test"}) + + assert result == "s3://test-bucket/DC/job-bundles/test-bundle.ojd" + repo._s3.upload_fileobj.assert_called_once() + call_kwargs = repo._s3.upload_fileobj.call_args[1] + assert call_kwargs["ExtraArgs"]["Metadata"]["ojd-name"] == "Test" + # ContentType is advertised as a courtesy hint (not trusted on download). + assert call_kwargs["ExtraArgs"]["ContentType"] == "application/zip" + + def test_uploads_without_metadata(self): + repo = self._make_repo() + buf = io.BytesIO(b"data") + + repo.upload_archive(buf, "simple") + + repo._s3.upload_fileobj.assert_called_once() + # ContentType is set even when no user metadata is provided. + call_kwargs = repo._s3.upload_fileobj.call_args[1] + assert call_kwargs["ExtraArgs"]["ContentType"] == "application/zip" + assert "Metadata" not in call_kwargs["ExtraArgs"] + + def test_calls_progress_callback(self): + repo = self._make_repo() + buf = io.BytesIO(b"data") + cb = MagicMock() + + repo.upload_archive(buf, "prog", progress_callback=cb) + + call_kwargs = repo._s3.upload_fileobj.call_args[1] + assert call_kwargs["Callback"] is cb + + +class TestS3ClearCacheFor: + def _make_repo(self, fresh_deadline_config): + with patch("boto3.Session"): + repo = S3BundleRepository( + bucket_name="test-bucket", root_prefix="DC", session=MagicMock() + ) + repo._s3 = MagicMock() + return repo + + def test_removes_cache_directory(self, fresh_deadline_config, tmp_path): + + repo = self._make_repo(fresh_deadline_config) + + # Create a fake cache entry + cache_dir = get_bundle_cache_dir() + os.makedirs(cache_dir, exist_ok=True) + + path = "s3://test-bucket/DC/job-bundles/cached.ojd" + # Pre-populate cache so clear has something to remove + repo.clear_cache_for(path) + # No error raised — passes even if cache didn't exist + + +class TestGetBundleCacheDir: + def test_returns_path_under_deadline_cache(self, fresh_deadline_config): + + result = get_bundle_cache_dir() + + assert ".deadline" in result + assert "cache" in result + assert "job-bundles" in result + + def test_truncates_multibyte_utf8_correctly(self, tmp_path): + """Non-ASCII names are encoded to an ASCII-safe form, then byte-truncated. + + S3 user metadata must be US-ASCII, so multi-byte characters are RFC + 2047-encoded (see _encode_s3_value) before the byte limit is applied. The + stored value stays within the byte limit and remains header-safe. + """ + # CJK characters are 3 bytes each in UTF-8 and must be encoded for S3. + bundle = tmp_path / "bundle" + bundle.mkdir() + (bundle / "template.yaml").write_text( + yaml.dump({"name": "日本語テスト" * 50, "steps": []}, allow_unicode=True) + ) + + metadata = build_bundle_metadata(str(bundle)) + + # Value must be ASCII-safe (header-safe) and within the byte limit. + metadata["ojd-name"].encode("ascii") # must not raise + encoded = metadata["ojd-name"].encode("utf-8") + assert len(encoded) <= METADATA_LIMIT_NAME + assert metadata["ojd-name"].endswith("...") + # It decodes back to a (truncated) prefix of the original CJK name. + info = _bundle_info_from_s3_metadata(metadata, "s3://bucket/key.ojd") + assert info is not None + assert info.name.endswith("...") + assert ("日本語テスト" * 50).startswith(info.name[:-3]) + + def test_metadata_limit_fixture_stays_within_budget(self): + """Verify the static metadata-limit-test fixture produces valid truncated metadata.""" + + # Fixture lives at test/fixtures/bundles/metadata-limit-test relative to repo root + fixture_path = Path(__file__).parents[3] / "fixtures" / "bundles" / "metadata-limit-test" + if not fixture_path.is_dir(): + pytest.skip("metadata-limit-test fixture not found") + + metadata = build_bundle_metadata(str(fixture_path)) + + assert metadata # Should produce metadata + # Total must stay within S3 budget + total = sum( + 12 + len(k.encode("utf-8")) + len(v.encode("utf-8")) for k, v in metadata.items() + ) + assert total <= S3_METADATA_TOTAL_BUDGET + # Name should be truncated (fixture has 300-char name) + assert metadata["ojd-name"].endswith("...") diff --git a/test/unit/deadline_client/ui/controllers/test_async_task.py b/test/unit/deadline_client/ui/controllers/test_async_task.py index 6f7ad434c..9af72016f 100644 --- a/test/unit/deadline_client/ui/controllers/test_async_task.py +++ b/test/unit/deadline_client/ui/controllers/test_async_task.py @@ -214,6 +214,60 @@ def test_auto_delete_is_enabled(self): assert task.autoDelete() is True + def test_run_swallows_deleted_signal_source_on_result(self, qtbot): + """ + If the signals object is deleted while the task is running, emitting the + result must not raise - it should be swallowed (no listeners remain). + + This reproduces the "RuntimeError: Signal source has been deleted" + cascade that occurs when the owning runner/widget is torn down before a + slow task returns. + """ + fn = Mock(return_value="result") + task = AsyncTask(fn) + + # Simulate the WorkerSignals C++ object having been deleted: any access to + # a signal raises RuntimeError, mirroring PySide behavior. + deleted_signals = Mock() + deleted_signals.result.emit.side_effect = RuntimeError("Signal source has been deleted") + deleted_signals.error.emit.side_effect = RuntimeError("Signal source has been deleted") + deleted_signals.finished.emit.side_effect = RuntimeError("Signal source has been deleted") + task.signals = deleted_signals + + # Must not raise despite every emit failing. + task.run() + + deleted_signals.result.emit.assert_called_once_with("result") + # finished is always attempted in the finally block. + deleted_signals.finished.emit.assert_called_once_with() + + def test_run_swallows_deleted_signal_source_on_error(self, qtbot): + """A deleted signal source during error emission must not raise.""" + fn = Mock(side_effect=ValueError("boom")) + task = AsyncTask(fn) + + deleted_signals = Mock() + deleted_signals.error.emit.side_effect = RuntimeError("Signal source has been deleted") + deleted_signals.finished.emit.side_effect = RuntimeError("Signal source has been deleted") + task.signals = deleted_signals + + # Must not raise despite the emits failing. + task.run() + + deleted_signals.error.emit.assert_called_once() + deleted_signals.finished.emit.assert_called_once_with() + + def test_safe_emit_does_nothing_when_canceled(self): + """_safe_emit is a no-op for canceled tasks and never touches the signal.""" + task = AsyncTask(Mock()) + signals = Mock() + task.signals = signals + + task.cancel() + task._safe_emit("result", "value") + + signals.result.emit.assert_not_called() + class TestStreamingAsyncTask: """Tests for StreamingAsyncTask, the progressive-result variant of AsyncTask.""" @@ -321,3 +375,23 @@ def gen(): assert progress == ["a"] assert errors == [test_error] + + def test_run_swallows_deleted_signal_source(self, qtbot): + """ + If the signals object is deleted while the streaming task is running, + emitting progress/result/finished must not raise. + """ + task = StreamingAsyncTask(lambda: iter(["a", "b"])) + + deleted_signals = Mock() + deleted_signals.progress.emit.side_effect = RuntimeError("Signal source has been deleted") + deleted_signals.result.emit.side_effect = RuntimeError("Signal source has been deleted") + deleted_signals.finished.emit.side_effect = RuntimeError("Signal source has been deleted") + task.signals = deleted_signals + + # Must not raise despite every emit failing. + task.run() + + assert deleted_signals.progress.emit.call_count == 2 + deleted_signals.result.emit.assert_called_once_with(None) + deleted_signals.finished.emit.assert_called_once_with() diff --git a/test/unit/deadline_client/ui/gui/conftest.py b/test/unit/deadline_client/ui/gui/conftest.py index aa4a638a6..332c8b262 100644 --- a/test/unit/deadline_client/ui/gui/conftest.py +++ b/test/unit/deadline_client/ui/gui/conftest.py @@ -12,6 +12,8 @@ _has_pyside6 = importlib.util.find_spec("PySide6") is not None _QT_TEST_FILES = [ + "test_gui_browser_dialog.py", + "test_gui_browser_dialog_threading.py", "test_gui_host_requirements.py", "test_gui_job_attachments.py", "test_gui_job_bundle_submitter.py", diff --git a/test/unit/deadline_client/ui/gui/test_gui_browser_dialog.py b/test/unit/deadline_client/ui/gui/test_gui_browser_dialog.py new file mode 100644 index 000000000..ac407cd7b --- /dev/null +++ b/test/unit/deadline_client/ui/gui/test_gui_browser_dialog.py @@ -0,0 +1,530 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +"""GUI tests for the job bundle browser dialog styling (pytest-qt).""" + +import time +from unittest.mock import MagicMock, patch + +from qtpy.QtCore import Qt, QTimer +from qtpy.QtGui import QColor, QStandardItem +from qtpy.QtWidgets import QApplication, QDialog, QLabel, QProgressBar + +from deadline.client.job_bundle.repository import BundleInfo, S3BundleRepository +from deadline.client.ui.dialogs.job_bundle_browser_dialog import ( + JobBundleBrowserDialog, + ROLE_LOADED, + ROLE_PATH, + _DownloadCancelled, +) + + +class TestHiddenBundleStyling: + def test_hide_dims_and_unhide_clears_foreground_override(self, qtbot, tmp_path): + """Unhiding must clear the foreground override, not hard-code a color. + + Previously unhide set the foreground to QColor(0, 0, 0) (black), which is + nearly invisible under a dark theme. It should instead clear the override + so the item falls back to the palette's text color. + """ + dialog = JobBundleBrowserDialog(local_source=str(tmp_path)) + qtbot.addWidget(dialog) + item = QStandardItem("my-bundle") + + # Hiding dims the item with a theme-neutral gray. + dialog._apply_hidden_style(item, hidden=True) + assert item.foreground().color() == QColor(150, 150, 150) + assert item.data(Qt.ItemDataRole.ForegroundRole) is not None + + # Unhiding clears the override entirely so the palette text color applies. + dialog._apply_hidden_style(item, hidden=False) + assert item.data(Qt.ItemDataRole.ForegroundRole) is None + # Explicitly: it must NOT be reset to hard-coded black. + assert item.data(Qt.ItemDataRole.ForegroundRole) != QColor(0, 0, 0) + + +class TestDownloadCancellation: + def test_cancel_aborts_download_cooperatively_and_clears_cache(self, qtbot, tmp_path): + """Cancelling a download must abort cooperatively, not via terminate(). + + The download callback checks a cancel flag and raises to unwind the + transfer cleanly (leaving the boto3 client/socket consistent), after + which the partial cache is cleared. This test cancels mid-download and + asserts the transfer stopped early and the cache was cleaned up. + """ + repo = MagicMock() + repo.get_bundle_size.return_value = 1024 * 1000 + observed = {"count": 0, "cancelled_via_exception": False} + + def _download(path, dest, progress_callback=None): + # Simulate a chunked transfer. Cooperative cancel raises + # _DownloadCancelled *into* this call (so it unwinds in Python); + # QThread.terminate() would instead kill the thread abruptly and this + # except block would never run. + try: + for _ in range(1000): + progress_callback(1024) # may raise _DownloadCancelled + observed["count"] += 1 + time.sleep(0.005) + return "/tmp/full-download" + except _DownloadCancelled: + observed["cancelled_via_exception"] = True + raise + + repo.download_full_bundle.side_effect = _download + + dialog = JobBundleBrowserDialog(local_source=str(tmp_path)) + qtbot.addWidget(dialog) + dialog._s3_repo = repo + dialog._selected_is_s3 = True + dialog._selected_path = "s3://bucket/prefix/bundle.ojd" + + # Cancel the modal download dialog shortly after it appears. + def _cancel(): + for widget in QApplication.topLevelWidgets(): + if isinstance(widget, QDialog) and widget.windowTitle() == "Downloading Bundle": + widget.reject() + return + QTimer.singleShot(20, _cancel) + + QTimer.singleShot(50, _cancel) + + result = dialog.resolve_selection() + + assert result is None, "cancelled download should resolve to None" + # The transfer was aborted by raising into the callback (cooperative), + # not by killing the thread. + assert observed["cancelled_via_exception"] is True + # Partial cache must be cleaned up after the worker unwinds. + repo.clear_cache_for.assert_called_once_with("s3://bucket/prefix/bundle.ojd") + # The transfer aborted before completing all chunks. + assert 0 < observed["count"] < 1000 + + +class TestDownloadSizeReporting: + def test_large_bundle_size_reports_human_readable_total(self, qtbot, tmp_path): + """A >2 GiB archive must report its true total, not overflow to a bogus "1 KB". + + ``size_ready`` carries the raw byte count. A plain ``Signal(int)`` maps to + a 32-bit C++ int, so a size above INT_MAX (2,147,483,647) overflowed and + the dialog showed "1 KB" as the total. The signal is now ``qlonglong`` and + the label is formatted with ``human_readable_file_size``. + """ + repo = MagicMock() + # ~2 GiB, larger than INT_MAX — this is the value that previously overflowed. + big_size = 2148139290 + repo.get_bundle_size.return_value = big_size + + def _download(path, dest, progress_callback=None): + # Emit a handful of 100 MiB chunks so the dialog stays open long + # enough for the inspector timer to read the label/range. + for _ in range(30): + progress_callback(100 * 1024 * 1024) + time.sleep(0.01) + return "/tmp/full-download" + + repo.download_full_bundle.side_effect = _download + + dialog = JobBundleBrowserDialog(local_source=str(tmp_path)) + qtbot.addWidget(dialog) + dialog._s3_repo = repo + dialog._selected_is_s3 = True + dialog._selected_path = "s3://bucket/prefix/big.ojd" + + captured_max = [0] + captured_label = [""] + + def _inspect(): + for widget in QApplication.topLevelWidgets(): + if isinstance(widget, QDialog) and widget.windowTitle() == "Downloading Bundle": + bar = widget.findChild(QProgressBar) + label = widget.findChild(QLabel) + # Wait until both the size (range) and at least one progress + # update (value + label) have been processed. + if ( + bar is not None + and bar.maximum() > 1 + and bar.value() > 0 + and label is not None + and "GB" in label.text() + ): + captured_max[0] = bar.maximum() + captured_label[0] = label.text() + widget.reject() + return + QTimer.singleShot(10, _inspect) + + QTimer.singleShot(20, _inspect) + + dialog.resolve_selection() + + # The full 2 GiB size survived the signal round-trip without overflowing: + # the bar's maximum is the size in KiB (~2 million, safely within int32). + assert captured_max[0] == big_size // 1024 + # The label reports a human-readable GB total, never a bogus "1 KB". + assert "GB" in captured_label[0] + assert "1 KB" not in captured_label[0] + + +class TestPreviewInjectionHardening: + """Bundle-derived preview values (name/description/steps) come from a template + or S3 metadata that a queue-writer can control. The preview renders them + inertly: the name, description, and steps labels are all forced to plain text + so no value is ever interpreted as markup. (Qt QLabels have no script engine, + so this is about HTML/CSS injection, not code execution.)""" + + def _dialog_with_info(self, qtbot, tmp_path, info): + dialog = JobBundleBrowserDialog(local_source=str(tmp_path)) + qtbot.addWidget(dialog) + dialog._selected_is_archive = False + repo = MagicMock() + repo.get_bundle_info.return_value = info + dialog._current_repo = repo + dialog._load_preview("some/path") + return dialog + + def test_malicious_name_is_plain_text(self, qtbot, tmp_path): + info = BundleInfo( + path="s3://bucket/evil.ojd", + name='pwn', + ) + dialog = self._dialog_with_info(qtbot, tmp_path, info) + + # Plain text: the name is shown literally and never parsed as markup, + # so the payload is inert (no rich-text/HTML rendering path at all). + assert dialog._preview_name.textFormat() == Qt.PlainText # type: ignore[attr-defined] + assert dialog._preview_name.text() == info.name + + def test_description_label_is_plain_text(self, qtbot, tmp_path): + info = BundleInfo( + path="s3://bucket/evil.ojd", + name="Bundle", + description="", + ) + dialog = self._dialog_with_info(qtbot, tmp_path, info) + + # Forced plain text so markup is shown literally and never interpreted. + assert dialog._preview_desc.textFormat() == Qt.PlainText # type: ignore[attr-defined] + assert "