[202405][GCU] Backport #3831 perf refactor + #4118/#4335/#4668 fixes (parts 1-8)

[rimunagala]

Summary

Backports upstream sonic-utilities #3831 (GCU perf refactor) to 202405 in 8 parts:

• Part 1/N — Config-threading refactor (fetch config once, thread through ChangeApplier)
• Part 2/N — PathAddressing xpath cache + batch find_ref_paths
• Part 3/N — patch_sorter: JsonMoveGroup + Bulk generators + validator refactor
• Part 4/N — Fix-up (dangling caller, missing imports, test fixtures)
• Part 5/N — Backport upstream #4118 (remove direct libyang dependency in gu_common.find_ref_paths)
• Part 6/N — Backport upstream #4335 (plan-aware RemoveCreateOnlyDependencyMoveGenerator — resolves the 400G DFS-explosion caveat below)
• Part 7/N — Backport upstream #4668 (safety guard in _validate_member for create-only leaf-list)
• Part 8/N — Wire BulkLeafListMoveGenerator into SortAlgorithmFactory.move_non_extendable_generators + test-fixture adaptations (part 6 commit message stated this was re-added, but the wiring edit did not land — this restores it)

Diff vs Azure/sonic-utilities.msft:202412 for ported prod files:

• change_applier.py, patch_sorter.py: 0 (identical)
• gu_common.py: -45 (illegal_dataacl_check method excluded — from unrelated #3668)
• generic_updater.py: -29 (skip_sort_tables + extract_scope multi-ASIC guard excluded — out of scope)

Requires coordinating buildimage-msft PR #2733 (Layers 2+3) to merge first — without it, GCU init hits AttributeError: 'SonicYang' object has no attribute 'find_schema_dependencies'.

Update — 400G caveat resolved

The prior "Known caveat — 400G config-apply" section (DFS explosion in RemoveCreateOnlyDependencyMoveValidator._validate_member on 400G port-add) is resolved by part 6 (backport of upstream #4335), which replaces the suboptimal RemoveCreateOnlyDependencyMoveGenerator with a plan-aware generator. Verified on the same DUT with the same C_400g_config_noBufferPG.json fixture: 400G dry-run now completes in 27 s / 25 moves (3 consecutive runs: 27s, 26s, 26s). No hangs, no KeyError, no CPU pegging.

Cherry-pick provenance

Not a clean cherry-pick. 202405's GCU has diverged from master — different method signatures, class structure and call sites — so git cherry-pick fails on most of these commits. Each upstream change was re-expressed against this branch: same logic, adapted to 202405's API. No new logic is introduced.

Parts 1–4 and 8 come from upstream #3831 (merge bd3de9da); part 5 from #4118 (4b787b0f); part 6 from #4335 (1580ccce); part 7 from #4668 (0552d0d2). The two review-driven commits are 1030de244 (restores the scope arg — the fixed line is identical to master gu_common.py:215) and e830cc965 (master's test that catches it, from #4219 2e9e81c1).

Net convergence vs master 1462eff8 — differing lines drop from 2063 → 491 (76% closed): change_applier.py 51→0 (now byte-identical), gu_common.py 771→74, patch_sorter.py 1084→266, generic_updater.py 157→151.

Intentionally left out (none perf-related): master's path-trace/diagnostics feature (trace_io, PatchSorterPath, (bool, reason) validator returns), list_checkpoints(includes_time), the multi-ASIC guard in extract_scope, illegal_dataacl_check (#3668), and loadData(quiet=True) (#4482 — needs a sonic-yang-mgmt kwarg 202405 lacks).

Correctness / non-perf testing done (fresh run against parts 1-8)

Full GCU unit-test suite (every file in tests/generic_config_updater/)

409 passed + 81 subtests passed = 490 test cases, 0 failed (26.44 s)

Test file	Result
gcu_feature_patch_application_test.py	2 passed, 2 subtests
generic_updater_test.py	39 passed
gu_common_test.py	74 passed
main_test.py	88 passed
multiasic_change_applier_test.py	6 passed
multiasic_generic_updater_test.py	1 passed
patch_sorter_test.py	197 passed + 79 subtests
service_validator_test.py	2 passed

Isolated add / replace / remove / mixed (3 runs each)

Purpose-built one-op patches that pass YANG validation and actually exercise the sorter:

Op	Patch target	Runs	rc	Moves	Avg time
add	/asic0/LOOPBACK_INTERFACE/Loopback0|10.99.0.1~132	3/3 ✅	0	1	4.5 s
replace	/asic0/ACL_TABLE/DATAACL/policy_desc	3/3 ✅	0	1	4.8 s
remove	/asic0/ACL_TABLE/DATAACL/ports/1 (leaf-list)	3/3 ✅	0	1	4.7 s
mixed (add+replace+remove)	all three in one patch	3/3 ✅	0	3	5.4 s

Real end-to-end patches (dry-run, 3 iterations each for perf-relevant ones)

Patch	Result	Time	Moves
100g_config.patch	PASS	26.3 s	25
400g_config.patch	PASS	26.4 s	25
400g_add_v2.patch	PASS	22.3 s	22
queue_add.patch	PASS	4.5 s	1
multi_asic.patch	PASS	6.1 s	2
multi_asic_revert.patch	PASS	5.2 s	0
stress.patch (80 ops)	PASS	16.6 s	2
stress_revert.patch	PASS	5.7 s	0

Prior-run functional coverage still holds:

• Real (non-dry-run) apply — 50 ACL rules: rc=0 in 7.92 s, all 50 rules in CONFIG_DB
• Real rollback of 49/50 rules: rc=0 in 6.78 s
• Multi-ASIC dry-run spanning /asic0/ + /asic1/: rc=0 in 4.94 s, both namespaces validated via extract_scope
• Negative cases (malformed JSON, invalid op, dangling leafref, schema violation): all produce graceful rc=2, no python tracebacks
• Rollback path via config replace: rc=0 in 4.10 s

Perf validation (str3-7800-lc3-1 chassis LC, asic0, SONiC.20240532.59) — parts 1-8

Scale battery

Scenario	Ops	Baseline	Parts 1-7 (previous run)	Parts 1-8 (this PR head)	Speedup vs baseline
scale_50_add (3 iters)	50	11.44 s	4.78 s	4.48 s	2.55×
scale_100_add (3 iters)	100	21.13 s	5.03 s	4.69 s	4.51×
scale_200_add (3 iters)	200	42.46 s	6.07 s	4.67 s	9.09×
variance_100 (5 iters)	100	21.42 s	4.92 s	4.57 s (σ≈0.10 s)	4.69×
scale_500	500	171.57 s	12.65 s	5.48 s	31.3×
scale_1000	1000	timeout (rc=2)	32.40 s	6.96 s	∞ (baseline never completes)

Small-op benchmarks (3 iters each)

Operation	Ops	Baseline avg	Parts 1-7	Parts 1-8
op_add_20	20	6.34 s	4.07 s	4.69 s ✅
op_replace_20	20	6.64 s	4.11 s	7.89 s *
op_remove_20	20	4.24 s	4.31 s	5.56 s *
op_mixed_30	30	5.72 s	4.04 s	7.62 s *

*Regenerated against live DUT config (ACL_TABLE.policy_desc + PORT.description + ACL_TABLE.ports leaf-lists) — different targets than the original synthetic fixtures. All complete cleanly, no hangs; timing delta reflects target complexity, not a regression.

Headline

• scale_1000 now completes in 7.0 s — 4.6× faster than parts 1-7's 32.4 s. Part 8's BulkLeafListMoveGenerator wire-up eliminates granular remove/add on leaf-list changes.
• scale_500 speedup vs baseline is now 31×.
• Every scale scenario matches or improves on parts-1-7 numbers.

Final on-chassis validation — clean-room 3-layer A/B (2026-07-30)

Independent end-to-end re-validation on a freshly installed image with a clean 202405-shape CONFIG_DB, measuring this PR's full stack against unmodified stock .59 GCU. This is the strongest available evidence: nothing carried over from prior sessions — image, config, and both code states were all built from scratch.

Environment

• DUT: str3-7800-lc4-1 — Arista 7800R3AK-36DM2-C36 chassis LC (broadcom-dnx, 2 ASICs), S/N SGD232207JY
• Image: SONiC.20240532.59 (release: 202405, branch: heads/20240532.59, Debian 12.12, Python 3.11, kernel 6.1.0-29-2-amd64) — installed fresh for this run
• CONFIG_DB: regenerated via config load_minigraph -y → 703 host keys + 1353 asic0 keys, no post-branch-cut drift keys
• All measurements are sudo config apply-patch <file> --dry-run, 3 iterations per scenario (safe on shared testbed — nothing committed)

States compared

	Contents	Source
A — stock .59	Unmodified .59 GCU + stock sonic-yang-mgmt + stock libyang	extracted directly from /host/image-20240532.59/fs.squashfs
B — this PR + #2733	L1 patch_sorter.py / gu_common.py / change_applier.py / generic_updater.py (this PR, parts 1-8) + L2 sonic_yang{,_ext,_path}.py (buildimage-msft #2733) + L3 rebuilt libyang / libyang-cpp / python3-yang 1.0.73 debs carrying libyang-leaf-must.patch	this PR head + #2733 branch

Results

Scenario	Ops	A — stock .59	B — PR #417 + #2733	Speedup
scale_50_add	50	11.62 s	4.96 s	2.34×
scale_100_add	100	22.13 s	5.19 s	4.26×
scale_200_add	200	43.26 s	4.99 s	8.67×
op_replace_20	40	7.30 s	4.62 s	1.58×
leaflist_replace	2	✗ not supported	4.98 s	new capability
op_remove_20	40	n/a¹	n/a¹	—

¹ Excluded on both sides: the generated patch adds 20 ACL rules then removes all 20, leaving ACL_RULE empty, which SONiC's ConfigDb "no empty tables" invariant rejects identically in State A and State B. Harness artifact, not a code difference — remove-op coverage is already provided by the isolated remove and multi_asic_revert / stress_revert runs above.

Raw per-iteration timings (s)

State A (stock .59)                    State B (PR #417 + #2733 L2+L3)
  scale_50_add:  12.366 11.221 11.280    scale_50_add:     5.422 4.625 4.838
  scale_100_add: 22.267 23.243 20.885    scale_100_add:    4.975 5.274 5.312
  scale_200_add: 43.215 44.032 42.546    scale_200_add:    4.953 5.082 4.949
  op_replace_20:  7.870  7.016  7.002    op_replace_20:    4.677 4.594 4.576
                                         leaflist_replace: 4.469 4.876 5.591

What this confirms

1. Reproduces the scale numbers above on independent hardware and a clean image — scale_50_add 2.34× (vs 2.55× measured on lc3-1), scale_100_add 4.26× (vs 4.51×). Within normal chassis-to-chassis variance.
2. State B is effectively flat across patch size — 4.96 s @ 50 ops → 4.99 s @ 200 ops, while stock .59 grows linearly (11.6 s → 43.3 s). This is the O(n²)→O(n) sorter win from #3831 showing up directly in wall-clock.
3. leaflist_replace only works under State B. Stock .59 cannot process a leaf-list replace at all; the #4478 logic bundled in this PR enables it. This is a functional gain, not just a perf one.

Merge-order proof (empirical)

The layer dependency asserted in buildimage-msft #2733 was verified by bisecting the stack on the live DUT — each layer produces a distinct, specific failure until the next one is added:

Stack on DUT	Result
Stock .59	✅ works (slow — this is State A)
Upstream #4592+#4593 files on stock .59	❌ DryRunConfigWrapper.apply_change_to_config_db() takes 2 positional arguments but 3 were given
This PR (parts 1-8), no #2733	❌ type object 'SonicYang' has no attribute 'configdb_path_split'
This PR + #2733 Layer 2 only	❌ 'Schema_Node_Leaf' object has no attribute 'must_size'
This PR + #2733 Layer 2 + Layer 3	✅ Patch applied successfully.

This confirms buildimage-msft #2733 (L2 + L3) must land before or together with this PR — merging this PR alone would break config apply-patch on 202405.

Layer 3 debs were rebuilt from sonic-buildimage/src/libyang (patch series incl. libyang-leaf-must.patch) inside a debian:12 container to match the DUT's Debian 12 / Python 3.11 runtime, producing libyang_1.0.73_amd64.deb, libyang-cpp_1.0.73_amd64.deb, and python3-yang_1.0.73_amd64.deb.

MOR add-cluster scaling A/B — test_max_mors_under_budget (2026-07-30)

Beyond synthetic patch batteries, this PR was measured against the real MOR add-cluster
workflow using test_max_mors_under_budget from sonic-mgmt PR
#26274 (@ c099743e), which answers the
operationally relevant question: how many MORs can be added inside a 30-minute budget?

Environment: DUT str3-7800-lc4-1, testbed vms62-t2-7800-2, image SONiC.20240532.59,
budget GCU_TIME_BUDGET_S=1800. State A = unmodified stock .59 GCU; State B = this PR
(parts 1-8) + buildimage-msft #2733 L2 + L3 — the same two states as the clean-room A/B above.

Six full test executions (A and B on each of three topologies), all PASSED, 24 measured
N-points total. Both ASICs were used to widen the range: the test enumerates MORs only from
PORTCHANNEL_MEMBER, so asic1's 14 routed links were converted to single-member PortChannels
reusing their existing IPs / DEVICE_NEIGHBOR / BGP_NEIGHBOR rows (no fabricated peers).

Results — asic1, widest measured range (N up to 14)

N	Input ops	A: changes	A: apply	B: changes	B: apply	Change ratio	Apply speedup
1	25	24	38.85 s	20	31.88 s	1.20×	1.22×
5	113	120	161.54 s	20	33.25 s	6.00×	4.86×
8	179	192	249.27 s	20	32.41 s	9.60×	7.69×
10	223	240	309.60 s	20	32.95 s	12.00×	9.40×
14	317	339	530.33 s	26	43.40 s	13.04×	12.22×
—	empty []	0	10.78 s	0	10.40 s	—	1.0× (control)

Whole-run wall clock: State A 51 m 39 s → State B 32 m 51 s.

Results — asic0, 10 MORs (mixed 2-member + 1-member LAGs)

N	Input ops	A: changes	A: apply	B: changes	B: apply	Change ratio	Apply speedup
1	36	34	51.51 s	23	35.35 s	1.48×	1.46×
5	156	170	222.79 s	23	36.24 s	7.39×	6.15×
8	238	265	341.62 s	23	37.18 s	11.52×	9.19×
10	301	315	472.39 s	32	50.94 s	9.84×	9.27×

Whole-run wall clock: State A 43 m 33 s → State B 16 m 39 s.

Analysis

1. The win is change-count collapse, not faster changes. Per-change cost is identical
   between states (A: 1.28–1.48 s/change; B: 1.52–1.58 s/change). What changes is how many
   changes the sorter emits: State A grows ~linearly with the input patch (24 → 339 on asic1),
   State B stays nearly flat (20 → 26). On asic1 the change count was literally unchanged at
   20 from N=1 through N=10 while the input patch grew ~9×. This is the bulk leaf-list
   generator folding per-member operations into single changes — exactly the #3831 mechanism.

2. Marginal cost per MOR: 37.8 s → 0.89 s on asic1 (42× slope reduction);
   46.8 s → 1.73 s on asic0 (27×). Extrapolated to the 1800 s budget: ~47 MORs (A) vs
   ~1990 MORs (B) on asic1. Flagged as extrapolation — the measured claim is that at N=14
   State A needs 530 s of apply time and State B needs 43 s.

3. The speedup widens with N (1.22× → 12.22×) and is still climbing at the edge of the
   measured range. The advantage is not a fixed constant factor; it grows with cluster size,
   which is the regime MRC isolation actually operates in.

4. Control validates attribution. An empty [] patch costs ~10.4–10.8 s in both states —
   pure fixed overhead (DB reads, YANG load, verification) that this PR does not touch, and does
   not regress. The measured gain is therefore attributable to the sorter, not to ambient
   differences between the two DUT states.

5. Independent reproduction of the sonic-mgmt PR's published baseline. Our State A numbers
   match PR #26274's published stock figures on a different image build — N=1: 34 changes /
   50.19 s (published 34 / 50.3 s); N=5: 170 changes / 217.35 s (published 170 / 232.0 s).
   This cross-validates the harness and the measurement method.

Note on end-to-end times: each N-point includes a fixed ~186 s stall that is not GCU and
not BGP convergence. It comes from a defect in the sonic-mgmt test's own
_verify_bgp_up_scaling helper, which greps show ip bgp summary -d all for the string
"Estab" — a string SONiC never prints (an Established session shows its prefix count, e.g.
257, in the State/PfxRcd column). The match therefore always fails and always burns its full
180 s timeout, reporting bgp_up=False regardless of actual BGP health. Proven by re-running the
identical State B sweep with all 24 IPv4 + 24 IPv6 peers genuinely Established (257 prefixes
each): e2e changed by less than 1 second (222.1 → 222.2 s at N=1, 231.8 → 231.9 s at N=8).
This is a harness defect that has now been fixed in sonic-net/sonic-mgmt#26274 (commit 5fd866d:
drop -d all and treat a peer as Established iff its State/PfxRcd column is numeric — verified on
str3-7800-lc4-1, 24/24 peers detected up where the old logic found 0). It affects both states equally
and so does not bias the A/B comparison. The apply-patch column is the PR-attributable
measurement; e2e is reported for completeness.

Reproducibility: three independent State B sweeps on identical topology gave N=1 222.1 / 222.9 /
222.2 s, N=5 224.4 / 224.5 / 224.0 s, N=8 231.8 / 231.6 / 231.9 s — max spread 0.8 s.

Grand total

536 test cases + 26 perf runs + 15 clean-room A/B runs + 24 MOR-scaling A/B data points
(6 full test_max_mors_under_budget executions) = 601 checks, 0 failures.

Plus a 5-step empirical merge-order proof (see previous section) establishing that buildimage-msft #2733 Layers 2+3 are hard prerequisites for this PR.

Signed-off-by: Rithvick Reddy Munagala rimunagala@microsoft.com

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR backports upstream Generic Config Updater (GCU) performance refactors and related fixes into the 202405 branch, primarily by introducing grouped/bulk move generation and reducing repeated YANG load/validation work, while also removing direct libyang usage from utilities.

Changes:

• Refactors patch sorting to operate on JsonMoveGroup and adds multiple bulk/non-extendable generators (key-level, low-level, leaf-list) to cut DFS exploration cost.
• Adds caching/ordering optimizations around config validation and dependency lookups (avoid redundant loadData() and improve key ordering via schema backlinks/musts).
• Removes direct yang/libyang imports (and wires Semgrep to enforce this), updating package manager/config tooling and extensive unit-test fixtures accordingly.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
generic_config_updater/patch_sorter.py	Introduces JsonMoveGroup, bulk generators, validator signature changes, and move simulation/undo plumbing.
generic_config_updater/gu_common.py	Refactors PathAddressing to use sonic_yang helpers; adds config validation caching and schema-based key ordering.
generic_config_updater/change_applier.py	Changes apply flow to be config-in/config-out and adds a post-apply delay workaround.
generic_config_updater/generic_updater.py	Threads the current config through change application instead of re-fetching each time.
config/config_mgmt.py	Removes direct libyang usage and adjusts SonicYang initialization/module parsing helpers.
sonic_package_manager/manager.py	Updates ConfigMgmt construction to no longer pass libyang options / direct libyang dependency.
README.md	Documents additional libyang3-related dependencies.
.semgrep/no-direct-libyang.yml	Adds a repo rule to ban direct yang/libyang imports outside tests.
.github/workflows/semgrep.yml	Updates Semgrep invocation to include local .semgrep/ rules.
tests/generic_config_updater/patch_sorter_test.py	Updates tests for move grouping (JsonMoveGroup) and new validator APIs.
tests/generic_config_updater/gutest_helpers.py	Adds mock side-effect helpers for new call signatures / grouped moves.
tests/generic_config_updater/gu_common_test.py	Adds tests for validation/loadData caching behavior and updated tokenization expectations.
tests/generic_config_updater/generic_updater_test.py	Updates expectations for config-threaded apply behavior.
tests/generic_config_updater/gcu_feature_patch_application_test.py	Updates applier tests to match new apply plumbing and mocking approach.
tests/generic_config_updater/change_applier_test.py	Updates dry-run/change applier tests for config-in/config-out apply signatures.
tests/generic_config_updater/multiasic_change_applier_test.py	Updates ChangeApplier tests for config-threaded apply and new get_config_db_as_json patch target.
tests/generic_config_updater/files/patch_sorter_test_success.json	Updates golden expected change sequences to reflect bulk/grouped move behavior and skip flags.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Vaibhav Hemant Dixit (vaibhavhd)]

rimunagala check this?

[Vaibhav Hemant Dixit (vaibhavhd)]

Seems blocking in nature.

[rimunagala]

Confirmed — this is a real bug and it was blocking. Fixed in 1030de244.

The bug

_invoke_validating_function was calling method_to_call(jsonpatch_element), but every validator in field_operation_validators.py on this branch takes (scope, patch_element) — rdma_config_update_validator, buffer_profile_config_update_validator, port_config_update_validator, read_statedb_entry. This PR does not touch that file, so the one-argument call raises TypeError for any patch touching a table wired up in gcu_field_operation_validators.conf.json — PFC_WD, BUFFER_POOL, BUFFER_PROFILE, WRED_PROFILE, PORT. The PORT entry has no validator_data, so every /PORT/... patch element hits it unconditionally.

This was a porting slip, not a deliberate change — validate_field_operation is an instance method and _invoke_validating_function is nested inside it, so self was always in scope. Master, the 202405 base branch and this PR all agree on the (scope, patch_element) signature. The fix restores master exactly:

return method_to_call(self.scope, jsonpatch_element)

byte-identical to master gu_common.py:215.

Why the on-chassis validation didn't catch it

The validation image (SONiC.20240532.59) predates the multi-ASIC scope work and ships the older single-argument validators, so on that image the 1-arg call matched the 1-arg validators and everything passed. The mismatch only surfaces once this branch's field_operation_validators.py is the one in use. A spy wrapper on the DUT confirmed the dispatch passed exactly one positional argument.

Why no unit test caught it, and what I did about it

Running the existing suite against the buggy commit: field_operation_validator_test.py → 42 passed, 0 failed. None of the three existing validate_field_operation tests on this branch reach the dispatch — two raise early via illegal_operations_to_fields_map, and the third uses LOOPBACK_INTERFACE, which has no validators wired. The port_config_update_validator tests call the validator directly with (scope, patch_element), bypassing the dispatch.

Master does not have this gap. Mutation-testing master (1462eff8) by injecting this exact 1-arg call into gu_common.py:215 fails test_validate_field_operation_illegal__buffer_pool with TypeError: rdma_config_update_validator() missing 1 required positional argument: 'patch_element' (124 passed clean → 1 failed / 123 passed mutated).

That test was added upstream in sonic-net/sonic-utilities#4219 and never backported, which is why this branch's copy of the test file is the older, smaller one. So rather than author a new test, I backported that one in e830cc965 — cherry-picked from #4219 with a single style adaptation (pytest.raises → self.assertRaises, since TestValidateFieldOperation subclasses unittest.TestCase here). No new test logic, and both prerequisites already exist on this branch: the BUFFER_POOL → rdma_config_update_validator conf wiring, and the 2-arg validator.

state	result
before 1030de244 (bug present)	1 failed — TypeError at gu_common.py:208 — 42 passed
after 1030de244 (fix applied)	43 passed

[rimunagala]

Verbatim from master — this whole test function is byte-identical to master's multiasic_change_applier_test.py (lines 183-205 there, scope=self at master:198). It arrived with #3831, which changed ChangeApplier.apply() to take the current config as a parameter; the base 202405 version of this test has no such call because the old apply(change) didn't need one.

It can't hide a scope-related bug here, though, because the scope under test isn't this argument. The patched get_config_db_as_json has a side effect that ignores its argument and returns a fixed dict — its only job is to produce a starting config to hand to applier.apply(...). The scope actually exercised comes from the constructor on the line above, and that's what the assertion checks:

applier = generic_config_updater.change_applier.ChangeApplier()      # default scope
...
mock_ConfigDBConnector.assert_called_once_with(use_unix_socket_path=True, namespace="")

The sibling test_apply_change_given_scope does the same for ChangeApplier(scope="asic0") / namespace="asic0", and passes scope="asic0" on the equivalent line — so the self here is an inconsistency in master rather than something intentional.

Agreed it would read better as scope="", but since it has no functional effect I've kept it identical to master so this branch doesn't diverge on a cosmetic point. Better fixed in master.

[rimunagala]

Verbatim from master #3831 — the same commit b5232ccb as the sleep above — and the comment immediately below this line is master's own acknowledgement of the question.

It also isn't a behaviour change. On the 202405 base the caller discarded apply()'s return value:

self.changeapplier.apply(change)     # 202405 base, generic_updater.py:143 — return not checked

so a failed _services_validate was already only logged, never propagated. _services_validate() itself is byte-identical across the base branch, this branch and master. The new signature returns upd_data because the config is now threaded through the loop instead of being re-read from Redis on every iteration.

On "partially-applied patches being treated as successful", to be precise about what is and isn't covered:

• A change that doesn't land in ConfigDB is caught. After the loop, generic_updater.py re-reads ConfigDB and compares it against the target, raising if they differ — byte-identical to the base branch:

  if not (self.patch_wrapper.verify_same_json(target_config, new_config)):
      raise GenericConfigUpdaterError(f"{scope}: after applying patch to config, there are still some parts not updated")

  This is the "final configuration comparison in PatchApplier 'just in case'" referred to in the comment above the sleep.
• A service-validation command that fails while the config did land is only logged. That's a real gap, but pre-existing and unchanged — _upd_data() performs the writes before _services_validate() runs, on both branches.

So no regression here, though I agree the propagation is worth improving — that's a master-level design question rather than something to diverge on in this backport.

[Vaibhav Hemant Dixit (vaibhavhd)]

Why 1s?

[rimunagala]

Not introduced by this backport — it's verbatim from master #3831, commit b5232ccb "config_db updates: work around race condition", byte-identical and at the same line (change_applier.py:173). Master's comment above it explains the choice: the sanity check that was removed had been hiding a real race (there's no feedback loop for when config_db changes are actually consumed), and it "would consume high CPU and would take a good amount of time (0.5s - 1s)". The sleep is "functionally equivalent in terms of preventing the race condition" — explicitly a workaround, with an upstream issue to follow.

The important part is that the sleep replaced work rather than adding to it. #3831 restructured ChangeApplier.apply(): the old path did two full get_config_db_as_json() dumps per change (one on entry, one for the sanity check); the new path threads the config through the loop and does none. Profiled on str3-7800-lc4-1, each dump costs ~0.48 s (it shells out via subprocess) — ~0.96 s per change, matching the 0.5-1 s in the comment.

Old and new are both called once per change (generic_updater.py:143), so this overhead scales with change count — which is exactly what this PR reduces. Measured on asic0, MOR add-cluster, N=8:

	changes	this overhead	apply total	s/change
Before	275	~264 s (2 dumps/change)	351.15 s	1.28
After	29	29 s (sleep)	44.08 s	1.52

Per-change cost is essentially flat, while in absolute terms the overhead drops ~9x (~264 s → 29 s) and its share of apply time goes from ~75% to ~66%. Removing it altogether needs the underlying race fixed rather than the sleep tuned, and that should land in master first so this branch stays aligned.

[Vaibhav Hemant Dixit (vaibhavhd)]

LGTM, other than a blocking comment from copilot review. That shoudl be an easy fix if the risk is agreed upon.

[rimunagala]

The blocking Copilot item is fixed.

1030de244 restores the scope argument when invoking field-operation validators — the line is now identical to master gu_common.py:215. e830cc965 adds the regression test that catches it, backported from upstream #4219 (2e9e81c1); it's the master test verbatim apart from pytest.raises → self.assertRaises to match this file's unittest style on 202405. Verified both directions on the branch — with the bug present it fails with the TypeError, with the fix applied the file passes 43/43.

On how it slipped through: 202405's field_operation_validator_test.py predates #4219 and had no test exercising the validator dispatch, so the branch stayed green with the bug in place. Master does catch it — I confirmed by injecting the same bug into master, where it fails 1 of 124. So this closes a pre-existing test gap rather than adding anything new.

The other three Copilot threads are answered in-thread, and I've added a Cherry-pick provenance section to the description covering the upstream source of each commit and what was deliberately left out.