Skip to content

P0 SMV: qualify SOURCE smpSynch AUTO within ~6–10s without forced metadata - #246

Open
masarray wants to merge 21 commits into
feature/studio-matched-ptp-sv-auto-20261017from
fix/studio-auto-sync-10s-evidence-20261018
Open

masarray wants to merge 21 commits into
feature/studio-matched-ptp-sv-auto-20261017from
fix/studio-auto-sync-10s-evidence-20261018

Conversation

@masarray

Copy link
Copy Markdown
Owner

Operator request

PTP SOURCE ON / smpSynch AUTO should gain measured LOCAL synchronization within approximately 10 seconds when the hardware clocks and canonical publisher are genuinely qualified, rather than taking nearly a minute.

Root cause

The current firmware only timestamps a canonical smpCnt=0 frame once per 16 sample epochs (seconds). The fail-closed gate requires a baseline followed by two measured consecutive pairs, hence first possible LOCAL qualification is ~48 seconds after SV START.

Targeted fix

  • One shared 2-epoch cadence constant is used by firmware descriptor sampling, portable continuity tracker, and AUTO freshness gate.
  • Expected evidence epochs at roughly 2, 4 and 6 seconds allow two coherent samples and qualification by ~6 seconds, subject to independent software phase ≤4 µs, source generation/session/fault continuity, physical MAC timestamp, and 250 µs canonical TX latency bound.
  • Shorten evidence age tolerance from 17 epochs to stride+1 (3) so stale confirmation is not retained for another 17 seconds.
  • Portable positive and negative regressions: no premature AUTO, no sync from MAC alone, reject high phase uncertainty, stale evidence, publisher fault, PTP generation mismatch. Existing 16s tracker tests remain explicit to ensure configurable stride and clock-step safety.
  • New Studio contract guard prevents accidentally reverting 2s firmware/portable cadence.

Non-regression / evidence boundaries

Do not touch GUI-to-SET channel mapping, IEC 61850 wire encoding, ESP32-P4 sample clock servo, PTP SOURCE generator, smpCnt, 4I+4V frame builder, diagnostic mirror (still OFF), or other GUI work already merged into #234 from #242–#245. All previous empirical 120s capture metrics are previous baseline; physical 10s acquisition and no-loss 4000 fps still require a fresh board capture. If TX timestamp takes >250 µs or source is not qualified, leave smpSynch=0 (do NOT force 1).

CI requirements

PTP core C++ tests, ESP32-P4 build, Studio Qt/Windows lifecycle and Native Preview installer from exact PR HEAD. Do not merge if checks fail; verify COM board capture before claiming hardware timing.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-18T08:31:26.315671Z fba4d7c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fba4d7c551

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// Baseline at ~2s, two coherent pairs at ~4s and ~6s permit AUTO to
// qualify within the 10s operator target IF independent software phase,
// PTP source and publisher continuity have also qualified. Never force 1.
inline constexpr std::uint32_t kSvTxEvidenceEpochStride = 2U;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the advertised descriptor polling cadence

With AR_SMV_HW_TX_EVIDENCE_EXPERIMENTAL enabled, this constant increases the blocking descriptor-timestamp operation from once per 16 epochs to once per 2 epochs, but embedded/esp32p4_smv_injector/main/Kconfig.projbuild and docs/PTP_SV_HARDWARE_TX_EVIDENCE_M2C.md still state that it runs once per 16 epochs. Because those descriptions are where users assess the feature's real-time scheduling risk, they now understate the polling frequency by 8×; update the Kconfig help and timing documentation with this change.

AGENTS.md reference: AGENTS.md:L230-L236

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Final CI / exact-commit handoff — 18 Oct 2026

HEAD: 4a795b1b5e7e4d560db342b7b907efac30fc55c9. Base: feature/studio-matched-ptp-sv-auto-20261017 (PR #234), not main.

ALL 10 HEAD WORKFLOWS PASS: Product Naming Guard, Embedded Profile CI, PTP Core CI, ESP32-P4 SMV Injector, ESP32-P4 PTP Lab Profile, ARStack Studio Qt, Security and Evidence, ARStack Studio Windows Lifecycle, C++ CI (GCC/Clang/MSVC), ARStack Studio Native Preview.

Exact-head Native Preview https://github.com/masarray/arstack61850/actions/runs/39111854061 — both firmware / build-esp32p4 and windows-preview PASS. Native smoke, Inno clean install, app-running upgrade, uninstall and public NSIS migration acceptance PASS.

Windows x64 Preview package: https://github.com/masarray/arstack61850/actions/runs/39111854061/artifacts/12131360475
Matched ESP32-P4 firmware: https://github.com/masarray/arstack61850/actions/runs/39111854061/artifacts/12130528867

Root cause & patch: 16s canonical MAC descriptor sampling required three epochs -> ~48s earliest AUTO qualification. Shared 2-second evidence stride + two measured pairs permits ~6s once software phase ≤4 µs, healthy current publisher and valid PTP SOURCE are actually qualified. No override to smpSynch=1; failures/stale evidence revoke qualification. Independent 80-sample generator DFT test PASS for Ua/Ub/Uc 0/−120/+120°, Ia/Ib/Ic −30/−145/+87° and unequal RMS currents.

Still NOT hardware-verified: No attached new COM15 serial/Pcap capture from this exact installer. Please test physically: install this Preview, flash its exact bundled firmware, connect 4I+4V, select PTP SOURCE/ON + smpSynch AUTO, click START; independently capture first ten seconds and a sustained 120s trace. Assert conditional AUTO LOCAL=1 by ~10s, phase/RMS as in table (not default 0/−120/+120), 4000fps, smpCnt modulo-4000 with 0 gaps/duplicates, 0 quality, intact PTP Sync/Follow_Up, no publisher TX misses. Keep PR #234 and #246 unmerged to main until bench evidence; do not claim IED/relay acceptance.

Copy link
Copy Markdown
Owner Author

Field follow-up (18 Oct 2026): LOCAL REF first observed by operator after >30s, not the 10s target. Updated PR #246 to commit 665f8d8. ESP32-P4 GPTimer is now enabled before the EMAC PTP clock snapshot to eliminate driver-enable latency in initial SV epoch arm; bounded phase servo and fail-closed smpSynch remain unchanged. Background observer now logs SV/PTP AUTO phase=... uncertainty=... slew=... tracking=... MACpairs=... LOCAL=... every ~2s initially, every 5s while acquiring, on transition and every 60s when stable. These logs are essential to distinguish slow phase acquisition from MAC evidence failures. New CI and installer in progress. Do not claim hardware <=10s without COM port build verification and a matching capture.

Copy link
Copy Markdown
Owner Author

FIELD REGRESSION — 18 Oct 2026: Operator reports intermittent NOT SYNC / LOCAL REF during early SOURCE AUTO, stabilizing only later. PR #246 HEAD d85c96c adds portable SvLocalRefAcquisition: before first LOCAL=1, requires 3 consecutive independently fully-qualified 1-second epochs (phase bound <=4us, 2 paired fresh real MAC descriptor timestamps, current session/fault/source generations and no missed samples). A single failed current qualification, source switch, missed epoch, session restart or publisher fault immediately invalidates; no forced smpSynch value, no grace period after faults. This prevents transient initial false promotions, but physical root-cause of repeated bad phase/MAC samples still requires startup COM/PCAP evidence. SV/PTP AUTO ... MACpairs=N clean=N/3 ... LOCAL=N now identifies readiness. CI exact SHA and matched Studio Windows installer requested; do not claim bench 10s accepted until real board evidence.

Copy link
Copy Markdown
Owner Author

18 Oct 2026 field request: initial NOT SYNC/LOCAL chatter is fixed, but AUTO LOCAL still takes >10s (sometimes ~30s). Latest source commit dbaeebe prewarms GPTimer while PTP SOURCE is actually ready and smpSynch mode is AUTO, before SV START. During preflight publisher notifications are disabled, so no SV frames or premature smpSynch claim. Phase observer independently tracks EMAC/GPTimer in the background. On START, when PTP lease/profile/rate are unchanged, handoff uses the already-disciplined GPTimer and starts first SV at next canonical second boundary. On mismatch, cold fallback preserves fail-closed behavior. MAC egress and 3 clean live epochs still required for LOCAL. Added pure policy tests and QML-source guard. CI / Native Preview building from exact PR HEAD. Hardware <=10 seconds is not yet verified; measure with fresh board COM log and PCAP.

Copy link
Copy Markdown
Owner Author

FINAL CI / HANDOFF (18 Oct 2026): exact commit dbaeebe, all 10 workflow checks passed. Native Preview workflow https://github.com/masarray/arstack61850/actions/runs/39123371988 SUCCESS. Windows x64 Inno installer and portable artifact https://github.com/masarray/arstack61850/actions/runs/39123371988/artifacts/12135915135 (contains firmware matched to GUI commit; manifest SHA verified). Matched standalone ESP32-P4 firmware https://github.com/masarray/arstack61850/actions/runs/39123371988/artifacts/12135317585. New SOURCE AUTO dry preflight starts PTP-phase GPTimer before SV START, transmits no SV during idle, and reuses same clock only with unchanged PTP source/profile/rate; fallback re-arms safely on mismatch. MAC evidence and three clean live epochs still required for LOCAL=1. Physical target <=10 seconds remains UNVERIFIED until bench testing: install/flash exact bundle, connect PTP SOURCE AUTO, wait ~10 seconds BEFORE pressing SV START, capture first 15 seconds of serial diagnostics and pcap, then compare with immediate-START cold baseline. Verify NO SV before explicit START, first smpCnt=0 at integer PTP epoch, 4000 fps, zero gaps/duplicates, valid GUI 4I4V setpoints, no forced local ref.

Copy link
Copy Markdown
Owner Author

P0 FIELD REGRESSION CAPTURE / ROOT CAUSE (18 Oct 2026, ~21:14 WIB). User's full 120s PCAP independently parsed: 480,466 total Ethernet frames, 480,087 IEC 61850 SV frames for APPID 0x4000 (all 480,087 advertise smpSynch=0), 120 each PTP Announce, Sync and Follow_Up, and 0 smpCnt modulo-4000 discontinuities across this PCAP. Live Process Bus Insight screenshots report 5–6 cumulative jumps and ~1,744–1,805 missing but those are NOT present in this 120s file; host/Npcap arrival jitter alone must not be represented as proven publisher loss. Root-cause in commit dbaeebe: silent preflight GPTimer ISR notifies observer, but sv_ptp_phase_observer_task only COMMITTED phase authority when decision.accepted && live_tx_running(); on every idle 1Hz preflight pass, it instead invalidated the controller and canceled steering. So the prewarm claim was misleading and could not converge while idle. Fix in new PR head: use SvAutoPreflightPolicy::can_steer to permit phase steering in verified preflight without transmitting SV; can_advertise_local ensures LOCAL only from actual live canonical MAC evidence, still requiring 3 clean epochs, no forced value. Added portable regressions and source guard. Run exact SHA firmware/installer CI then require fresh 120s capture + first 15s serial (SV/PTP AUTO phase=... slew=... tracking=... MACpairs=... clean=...) to prove recovered LOCAL; source code fix is not field acceptance. Do not upload user's raw capture to repository.

Copy link
Copy Markdown
Owner Author

P0 FIELD-REGRESSION CORRECTION / INSTALLER READY — 18 Oct 2026. Exact PR HEAD 7dc488c2c49bde0348dcdd649de654b2d6ba9017. All 10 CI workflows PASS, including ESP32-P4 SMV Injector, PTP Core, PTP Lab SOURCE/RECEIVER, Qt Studio, Security, C++ GCC/Clang/MSVC, Windows Lifecycle and Native Preview. Full native workflow https://github.com/masarray/arstack61850/actions/runs/39135571111 SUCCESS. Matched Windows Studio x64 installer+portable https://github.com/masarray/arstack61850/actions/runs/39135571111/artifacts/12140186093 ; firmware ESP32-P4 https://github.com/masarray/arstack61850/actions/runs/39135571111/artifacts/12139292984. Installer's bundled firmware was checked against exact GUI commit manifest and SHA-256. Native packaged GUI smoke, clean install/upgrade/uninstall and public NSIS v0.1.1 migration PASS. Root-cause fix: silent AUTO preflight used to trigger observer but discard every phase servo command due live_tx_running() gating; now preflight actually commits phase slew with SOURCE clock unchanged, while LOCAL always remains 0 until live canonical MAC descriptor pairs and three consecutive real qualified epochs. The uploaded 120s PCAP showed 480,087 SV, smpSynch=0 on all, 120 Announce+Sync+FollowUp pairs, 0 modulo-4000 smpCnt gaps. This is a software fix only: DO NOT claim hardware AUTO <=10s or relay interoperability until new matched-build bench capture. Test SOURCE/AUTO idle for ~10s, START, measure LOCAL transition and first 120s capture; collect serial SV/PTP AUTO phase, uncertainty, slew, tracking, MACpairs, clean and failure counters. If fails, revert immediately to last observed stable d85c96ce packaged build rather than force smpSynch=1. PR intentionally OPEN and unmerged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant