test-cnm: Support sending / polling multiple queues in one test run by reweeden · Pull Request #65 · asfadmin/cloud-tools

reweeden · 2026-04-10T22:07:01Z

Adds a new command tcnm configure that can be used to set the cnm ingest and response queues individually for each test.

Here is the configuration I set up for CIRRUS-ASF. There is currently no way to have a test execute on multiple queues, so the ASF_ECMWF_TROP still needs to be executed on the opera queue separately.

$ tcnm -e sbx configure
ARIA_S1_GUNW/1/S1-GUNW-D-R-163-tops-20250608_20250527-212909-00121E_00010S-PP-109d-v3_0_1:
    cnm_ingest_queue: aria-workflow-queue
    cnm_response_queue: aria-mock-response-queue
ASF_ECMWF_SMST/1/ECMWF_SMST_202411150000_202411150000_1:
    cnm_ingest_queue: ecmwf-nisar-ingest-queue
    cnm_response_queue: ecmwf-mock-response-queue
ASF_ECMWF_TROP/1/ECMWF_TROP_201501010000_201501010000_1:
    cnm_ingest_queue: ecmwf-nisar-ingest-queue
    cnm_response_queue: ecmwf-mock-response-queue
UAVSAR_POLSAR/1/CValle_13300_14086_012_140616_L090_CX_01:
    cnm_ingest_queue: uavsar-workflow-queue
    cnm_response_queue: uavsar-mock-response-queue
UAVSAR_RPI/1/ArgVol_24309_13045-001_14054-002_0401d_s01_L090_01:
    cnm_ingest_queue: uavsar-workflow-queue
    cnm_response_queue: uavsar-mock-response-queue

The tester module internals are refactored to support the TestExecutor polling multiple response queues at a time if tests are executing across multiple workflows simultaneously.

CIRRUS-ASF test run

The RPI fails with a key error due to the ingest not being implemented.

Executing tests on asf-cumulus-dev
Starting: ARIA_S1_GUNW/1/S1-GUNW-D-R-163-tops-20250608_20250527-212909-00121E_00010S-PP-109d-v3_0_1                          
Starting: ASF_ECMWF_SMST/1/ECMWF_SMST_202411150000_202411150000_1                                                            
Starting: ASF_ECMWF_TROP/1/ECMWF_TROP_201501010000_201501010000_1                                                            
Starting: OPERA_L2_CSLC-S1-STATIC_CALVAL_V1/OPERA_L2_CSLC-S1-STATIC_T143-305899-IW1_20140403_S1A_v1.0                        
Starting: OPERA_L2_CSLC-S1-STATIC_V1/OPERA_L2_CSLC-S1-STATIC_T001-000659-IW1_20140403_20230928T214220Z_S1B_v1.0              
Starting: OPERA_L2_CSLC-S1_V1/1.0/OPERA_L2_CSLC-S1_T081-172144-IW3_20250428T055424Z_20250429T073127Z_S1A_VV_v1.1             
Starting: OPERA_L2_RTC-S1-STATIC_CALVAL_V1/OPERA_L2_RTC-S1-STATIC_T004-006646-IW2_20140403_20231116T013537Z_S1B_30_v1.0      
Starting: OPERA_L2_RTC-S1-STATIC_V1/OPERA_L2_RTC-S1-STATIC_T001-000142-IW2_20140403_S1A_30_v1.0                              
Starting: OPERA_L2_RTC-S1-STATIC_V1/OPERA_L2_RTC-S1-STATIC_T001-000677-IW2_20140403_S1A_30_v1.0                              
Starting: OPERA_L2_RTC-S1_V1/OPERA_L2_RTC-S1_T001-000659-IW1_20210706T183013Z_20230928T161205Z_S1B_30_v1.0                   
Starting: OPERA_L2_RTC-S1_V1/OPERA_L2_RTC-S1_T035-073251-IW2_20240512T020817Z_20240512T122756Z_S1A_30_v1.0                   
Starting: OPERA_L2_RTC-S1_V1/OPERA_L2_RTC-S1_T087-185163-IW3_20250215T155253Z_20250217T171846Z_S1A_30_v1.0                   
Starting: OPERA_L3_DISP-S1-STATIC_V1/1/OPERA_L3_DISP-S1-STATIC_F21518_20140403_S1A_v1.0                                      
Starting: OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20201221T004213Z_20210619T004215Z_v1.0_20250418T131935Z        
Starting: OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20210619T004215Z_20210911T004220Z_v1.0_20250418T214032Z        
Starting: OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20210619T004215Z_20211017T004221Z_v1.0_20250418T214032Z        
Starting: OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20210619T004215Z_20211204T004219Z_v1.0_20250418T214032Z        
Starting: OPERA_L3_DIST-ALERT-S1_V1/1/OPERA_L3_DIST-ALERT-S1_T50MKT_20250710T220108Z_20260130T090919Z_S1A_30_v0.1            
Starting: OPERA_L4_TROPO-ZENITH_V1/1/OPERA_L4_TROPO-ZENITH_20160701T000000Z_20250918T000724Z_HRES_v1.0                       
Starting: UAVSAR_POLSAR/1/CValle_13300_14086_012_140616_L090_CX_01                                                           
Starting: UAVSAR_RPI/1/ArgVol_24309_13045-001_14054-002_0401d_s01_L090_01                                                    
SUCCESS | ARIA_S1_GUNW/1/S1-GUNW-D-R-163-tops-20250608_20250527-212909-00121E_00010S-PP-109d-v3_0_1                          
SUCCESS | ASF_ECMWF_SMST/1/ECMWF_SMST_202411150000_202411150000_1                                                            
SUCCESS | OPERA_L2_RTC-S1_V1/OPERA_L2_RTC-S1_T087-185163-IW3_20250215T155253Z_20250217T171846Z_S1A_30_v1.0                   
SUCCESS | OPERA_L2_CSLC-S1-STATIC_CALVAL_V1/OPERA_L2_CSLC-S1-STATIC_T143-305899-IW1_20140403_S1A_v1.0                        
SUCCESS | OPERA_L3_DISP-S1-STATIC_V1/1/OPERA_L3_DISP-S1-STATIC_F21518_20140403_S1A_v1.0                                      
SUCCESS | OPERA_L2_RTC-S1-STATIC_CALVAL_V1/OPERA_L2_RTC-S1-STATIC_T004-006646-IW2_20140403_20231116T013537Z_S1B_30_v1.0      
SUCCESS | OPERA_L2_RTC-S1-STATIC_V1/OPERA_L2_RTC-S1-STATIC_T001-000677-IW2_20140403_S1A_30_v1.0                              
SUCCESS | ASF_ECMWF_TROP/1/ECMWF_TROP_201501010000_201501010000_1                                                            
SUCCESS | OPERA_L2_RTC-S1-STATIC_V1/OPERA_L2_RTC-S1-STATIC_T001-000142-IW2_20140403_S1A_30_v1.0                              
SUCCESS | OPERA_L3_DIST-ALERT-S1_V1/1/OPERA_L3_DIST-ALERT-S1_T50MKT_20250710T220108Z_20260130T090919Z_S1A_30_v0.1            
SUCCESS | OPERA_L2_CSLC-S1_V1/1.0/OPERA_L2_CSLC-S1_T081-172144-IW3_20250428T055424Z_20250429T073127Z_S1A_VV_v1.1             
SUCCESS | OPERA_L2_CSLC-S1-STATIC_V1/OPERA_L2_CSLC-S1-STATIC_T001-000659-IW1_20140403_20230928T214220Z_S1B_v1.0              
SUCCESS | OPERA_L2_RTC-S1_V1/OPERA_L2_RTC-S1_T001-000659-IW1_20210706T183013Z_20230928T161205Z_S1B_30_v1.0                   
SUCCESS | OPERA_L2_RTC-S1_V1/OPERA_L2_RTC-S1_T035-073251-IW2_20240512T020817Z_20240512T122756Z_S1A_30_v1.0                   
SUCCESS | OPERA_L4_TROPO-ZENITH_V1/1/OPERA_L4_TROPO-ZENITH_20160701T000000Z_20250918T000724Z_HRES_v1.0                       
SUCCESS | OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20201221T004213Z_20210619T004215Z_v1.0_20250418T131935Z        
SUCCESS | OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20210619T004215Z_20211017T004221Z_v1.0_20250418T214032Z        
SUCCESS | OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20210619T004215Z_20211204T004219Z_v1.0_20250418T214032Z        
SUCCESS | OPERA_L3_DISP-S1_V1/1/OPERA_L3_DISP-S1_IW_F01093_VV_20210619T004215Z_20210911T004220Z_v1.0_20250418T214032Z        
SUCCESS | UAVSAR_POLSAR/1/CValle_13300_14086_012_140616_L090_CX_01                                                           
asf-cumulus-dev:  95%|█████████████████████████████████████████████████████████████████████████████████▉    | 20/21 finished 
PENDING | UAVSAR_RPI/1/ArgVol_24309_13045-001_14054-002_0401d_s01_L090_01                                                    
Totals: 20 Succeeded; 0 Failed; 1 Pending of 21 tests

Pull Request Checklist

I have:

performed a self review of my code I&A code style
- Resources and Data Structures are sorted by ABC or a defined sorting pattern
updated the documentation accordingly
verified required action checks are passing
bumped the version number as appropriate

This makes copy/pasting to `tcnm test <prefix>` easier

simultaneously

gjclark · 2026-04-17T16:19:32Z

Nice addition!

reweeden added 2 commits April 9, 2026 18:40

Print full test id in list command

5e14abe

This makes copy/pasting to `tcnm test <prefix>` easier

Move ChecksumReaderProxy and ChecksumWriter to uploader.py

fcbd4fc

reweeden force-pushed the rew/tcnm-multi-queue branch from 3fe89e9 to d76f1d2 Compare April 13, 2026 16:15

Refactor tester internals to support multiple cnm queues

0f0ddc6

simultaneously

reweeden force-pushed the rew/tcnm-multi-queue branch 5 times, most recently from f0b3dd5 to aa7a617 Compare April 13, 2026 17:07

reweeden added 4 commits April 13, 2026 13:32

Allow getting test specific config from the metadata.json file

84cbb4c

Add provider and trace as attributes on ExecutableTest

2fa0521

Only save metadata file if contents changed

7641891

Replace utcnow()

6739134

reweeden force-pushed the rew/tcnm-multi-queue branch 2 times, most recently from ff65b0c to 63ed333 Compare April 13, 2026 19:03

reweeden marked this pull request as ready for review April 13, 2026 19:33

reweeden requested review from Cameronsplaze, gjclark and mckadesorensen April 13, 2026 19:33

Cameronsplaze approved these changes Apr 15, 2026

View reviewed changes

Comment thread test-cnm/test_cnm/tester/collector.py

Comment thread test-cnm/pyproject.toml

gjclark approved these changes Apr 17, 2026

View reviewed changes

reweeden force-pushed the rew/tcnm-multi-queue branch from 63ed333 to 0052327 Compare April 22, 2026 18:14

reweeden added 2 commits April 22, 2026 19:02

Allow overriding dynamic config with cli args

e092b0d

Bump version number

a58f34c

reweeden force-pushed the rew/tcnm-multi-queue branch from 0052327 to a58f34c Compare April 22, 2026 23:03

reweeden merged commit e8b7732 into main Apr 27, 2026
6 checks passed

reweeden deleted the rew/tcnm-multi-queue branch April 27, 2026 14:42

Navigation Menu

Uh oh!

test-cnm: Support sending / polling multiple queues in one test run - #65

test-cnm: Support sending / polling multiple queues in one test run#65
reweeden merged 9 commits into
mainfrom
rew/tcnm-multi-queue

reweeden commented Apr 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

gjclark commented Apr 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

reweeden commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gjclark commented Apr 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

reweeden commented Apr 10, 2026 •

edited

Loading