fix(gcp): improve ListImage lookup to include non-default project images - #1797
Merged
cb-spider merged 5 commits intoAug 13, 2026
Merged
Conversation
added 3 commits
August 10, 2026 01:42
deeplearning-platform-release alone returned 26k+ deprecated image versions, making ListImage() take ~95s. Activating the filter that was already written but never applied drops it to ~20s (47546 -> 246 images), matching the risk flagged in issue cloud-barista#1184.
Splits arrImageProjectList into arrStandardImageProjectList (general OS) and arrExtendedImageProjectList (GPU/HPC/ML, the group responsible for most of ListImage()'s latency), and extracts the per-project fetch loop into listImagesByProject() so each group can be called independently. Pure structural change: verified the returned image set and order are byte-for-byte identical to pre-refactor output (246/246 images, same set, same order) against a live GCP account. Sets up cb-spider#1184 Stage 2, which will swap the extended group to family-based lookups.
Images.List() on GPU/HPC/ML projects (deeplearning-platform-release etc.) forces GCP to scan the project's entire image history server side even with the deprecated filter. Replaces that with a curated project->family map resolved via Images.GetFromFamily(), fetched in parallel (same pattern as RegionZoneHandler.ListRegionZone()). A sequential first pass was slower than the List()-based baseline (31.5s vs 23.8s, 48 individual round trips add up); parallelizing brought it to 12.6s. A missing/retired family is skipped best-effort rather than failing the whole call, since this list is a hand-maintained catalog rather than raw GCP data. Verified against a live GCP account: same 246-image result set as the List()-based baseline, standard group order unchanged, extended group is the same 48 images as a set (internal order differs, expected since it's a different retrieval strategy). Completes the 3-stage plan for cb-spider#1184 (Stage 1: #196, Stage 3: #197).
|
Could you please check and revise Korean texts? cloud-control-manager/cloud-driver/drivers/gcp/resources/ImageHandler.go 18:+// "OS+가속기 조합"당 하나뿐이라 목록 자체는 작음(2026-08-07 기준 5개 프로젝트 합계 48개).
20:+// "최신 활성 이미지 1건"만 직접 조회 — GCP가 매번 프로젝트의 전체 이미지 이력을 스캔하지
98:+ // family 기반으로 "최신 활성 이미지 1건씩"만 직접 조회(cb-spider#1184 Stage 2).
121:+// 등록된 family(projectFamilyMap)마다 Images.GetFromFamily()로 "최신 활성 이미지 1건"만 |
Per review feedback on PR cloud-barista#1797: print/log statements and source comments should be in English.
Collaborator
|
b78050c (built on 6ee5155) replaced the extended (GPU/HPC/ML) group's Images.List() scan with per-family Images.GetFromFamily() lookups against a hardcoded arrExtendedImageFamilyMap, to cut ListImage() latency. That map is a snapshot of family names known at write time: GCP can add a new family at any point, and nothing here would notice. Review feedback on PR cloud-barista#1797 flagged exactly this. GetImage(imageIID), GetImageN(imageName) and GetImageByUrl all resolve images directly via Images.Get()/Images.List() and don't depend on the family map, so an unmapped family is still technically fetchable by name or URL. But this driver's normal usage pattern is ListImage() -> take an ImageIID from the result -> GetImage(imageIID), so anything missing from ListImage()'s output is undiscoverable in practice. Catalog correctness outweighs the latency win here. Restores ImageHandler.go to 5f842b5's version: a single arrImageProjectList and a plain per-project Images.List(projectId).Filter("NOT deprecated:*") scan for every project, standard and extended alike. Drops arrStandardImageProjectList, arrExtendedImageProjectList, arrExtendedImageFamilyMap, listImagesByProject() and listImagesByFamily(), which existed only to support the family-based path (6ee5155's own message: "Sets up cb-spider#1184 Stage 2"). Re-translates the one Korean comment 5f842b5 still carried, per the existing English-only comment convention on this branch. cb-spider#1184
Contributor
Author
|
말씀하신 우려 확인했고, 반영했습니다. GetImage(imageIID) / GetImageN() / GetImageByUrl은 Images.Get()을 직접 호출하기 때문에 arrExtendedImageFamilyMap에 의존하지 않아 문제가 없습니다. 하지만 실제 사용 패턴은 ListImage() → 원하는 ImageIID 선택 → GetImage(ImageIID) 흐름이라, ListImage()에서 빠지는 이미지는 애초에 선택지에 오르지 못합니다. 즉 신규 family가 목록에 등록되지 않으면 조용히 누락되는 구조라, 말씀하신 리스크가 실제로 컸습니다. {Project,Family} 기반 조회 부분을 되돌리고, 이전의 {Project} 기반 전체 스캔 방식(NOT deprecated:* 필터는 유지)으로 원복했습니다. 꼼꼼한 리뷰 감사합니다. |
Collaborator
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #1184 —
ListImage/GetImageon GCP only searched a limited set of projects, so images outside that set (e.g. some public/marketplace images) could not be looked up. Three incremental changes:NOT deprecatedfilter onListImageto cut latencyTest plan
ListImagelatency stays comparable to before the change