Skip to content

[dhcpmon] Validate configured DHCP server fan-out - #108

Closed
Xichen96 wants to merge 23 commits into
sonic-net:masterfrom
Xichen96:dev/xichenlin/validate-configured-server-fanout
Closed

[dhcpmon] Validate configured DHCP server fan-out#108
Xichen96 wants to merge 23 commits into
sonic-net:masterfrom
Xichen96:dev/xichenlin/validate-configured-server-fanout

Conversation

@Xichen96

@Xichen96 Xichen96 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Description of PR

Validate DHCP relay forward-path fan-out against the number of configured DHCP
servers.

Work item tracking
  • Microsoft ADO (number only): 38615655

Type of change

  • Bug fix

Approach

Read the current server count from CONFIG_DB on each health check:

  • DHCPv4 native: DHCPV4_RELAY|<VLAN> / dhcpv4_servers
  • DHCPv4 legacy fallback: VLAN|<VLAN> / dhcp_servers
  • DHCPv6: DHCP_RELAY|<VLAN> / dhcpv6_servers
  • DHCPv6 compatibility fallback: VLAN|<VLAN> / dhcpv6_servers

Both raw leaf-list fields with an @ suffix and unsuffixed fields are accepted.

This PR adds two separate group-3 magnitude states on Agg-<VLAN>. It does
not change the group-1 relay-flow checks or group-2 management checks.

The new checks require:

  • DHCPv4 Discover or Request TX delta equals its RX delta multiplied by the
    configured DHCPv4 server count.
  • DHCPv6 Relay-Forward TX delta equals the selected forward-input RX delta
    multiplied by the configured DHCPv6 server count.

If no supported server list is found or there is no relevant activity, the
server-fan-out state is INDETERMINATE.

Stack

Depends on #107. Review the final commit only:

12257b7 [dhcpmon]: Validate configured server fan-out

Verification

No local compilation was used. Azure PR CI will validate the branch.

Back port request

None. This targets master only.

Xichen96 added 14 commits July 26, 2026 16:39
Resolve nested Ethernet-to-PortChannel-to-VLAN membership and aggregate counters at each immediate parent without changing existing Dual-ToR attribution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Treat a self-mapped VLAN or PortChannel member as a root interface instead of creating a self aggregate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Treat configured context interfaces as roots and fail closed after the expected physical-to-PortChannel-to-VLAN depth, avoiding cyclic recursion in packet processing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Resolve direct VLAN membership before falling back through PortChannel membership for both context and immediate-parent lookup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Use the VLAN-first parent resolver for context traversal, aggregate selection, and tracked-interface detection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Generate immediate-child aggregate counter names through the device manager for initialization and packet accounting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Keep hierarchy traversal in the device manager while placing the pure aggregate-name formatter with the existing aggregate utilities.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Document unmapped parent lookup and root aggregate fallback without changing behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Keep the established get_agg_counter_ifname utility name while changing only its hierarchy semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Document root-interface parent semantics and the private context traversal helper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Populate one state per discovered VLAN or PortChannel check, derive aggregate ratios from topology, and format aggregate disparity errors with the interface and duration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Copilot AI review requested due to automatic review settings July 28, 2026 11:41
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens sonic-dhcpmon’s relay forward-path health validation by factoring in the number of DHCP servers configured in CONFIG_DB, so counter-based fan-out checks can detect partial relay forwarding (e.g., forwarding to only a subset of configured servers).

Changes:

  • Add a utility to read the configured DHCP server count (v4/v6) from CONFIG_DB with table/field fallbacks.
  • Update DHCPv4 positive health checks to require expected TX fan-out (RX * server_count) for Discover/Request when server configuration is available.
  • Update DHCPv6 positive health checks to apply configured-server fan-out validation to Relay-Forward output while keeping return-path checks activity-based.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
src/util.h Declares helper to retrieve configured DHCP server count from CONFIG_DB.
src/util.cpp Implements CONFIG_DB server-list counting and selection logic for v4/v6 (with fallbacks).
src/dhcp_device.cpp Applies configured-server fan-out validation to DHCPv4 and DHCPv6 positive health checks using counter deltas.

Comment thread src/dhcp_device.cpp
Comment on lines +129 to 134
static uint64_t get_counter_delta(const std::string &ifname, int sock, int msg_type)
{
return check_counter_not_transmitted(ifname, rx_sock, tx_sock, (const int *)monitored_msgs, monitored_msg_sz) ?
DHCP_MON_STATUS_UNHEALTHY : DHCP_MON_STATUS_HEALTHY;
const sock_info_t &sock_info = sock_mgr_get_sock_info(sock);
return sock_info.all_counters.at(ifname).at(msg_type) -
sock_info.all_counters_snapshot.at(ifname).at(msg_type);
}
Comment thread src/util.cpp
Comment on lines +52 to +58
size_t count = 1;
for (const char ch : *value) {
if (ch == ',') {
count++;
}
}
return count;
@Xichen96
Xichen96 force-pushed the dev/xichenlin/validate-configured-server-fanout branch from 135f3a3 to c746349 Compare July 28, 2026 13:35
Copilot AI review requested due to automatic review settings July 28, 2026 13:35
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/util.cpp:45

  • get_config_list_count() assumes the CONFIG_DB value is a comma-separated string and returns 1 for any non-empty value, which miscounts YANG leaf-list raw fields like [] (empty JSON array) as 1 server. That can cause false unhealthy results by enforcing fan-out against a non-existent server list. Consider parsing JSON arrays when the value starts with '[' and treating an empty array as 0 entries, falling back to comma-counting only for legacy comma-separated strings.
static size_t get_config_list_count(const std::string &table, const std::string &field)
{
    const std::string key = table + "|" + downstream_ifname;
    auto value = mConfigDbPtr->hget(key, field + "@");
    if (value == NULL) {

src/dhcp_device.cpp:134

  • get_counter_delta() subtracts snapshot from current using unsigned arithmetic. If a counter resets or wraps between snapshots (current < snapshot), this underflows and produces a huge delta, which can incorrectly mark the relay unhealthy. Clamp the delta to 0 when the counter did not increase.
static uint64_t get_counter_delta(const std::string &ifname, int sock, int msg_type)
{
    const sock_info_t &sock_info = sock_mgr_get_sock_info(sock);
    return sock_info.all_counters.at(ifname).at(msg_type) -
           sock_info.all_counters_snapshot.at(ifname).at(msg_type);
}

src/health_check.cpp:65

  • The aggregate error log message says "packets received but none transmitted", but the new v6 relay-flow check can fail for other reasons (e.g., fan-out mismatch or reply transformation missing). This can mislead operators when v6 health fails. Consider making the message generic so it remains accurate for both v4 and v6 checks.
static dhcp_mon_status_t check_relay_flow_health_v6()
{
    return dhcp_device_get_status(agg_dev_all, DHCP_DEVICE_CHECK_RELAY_FLOW_V6);
}

Copilot AI review requested due to automatic review settings July 28, 2026 14:40
@Xichen96
Xichen96 force-pushed the dev/xichenlin/validate-configured-server-fanout branch from c746349 to 1b7594e Compare July 28, 2026 14:40
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/util.cpp:59

  • get_config_list_count() counts commas to infer list length, but CONFIG_DB leaf-list fields with an '@' suffix are typically stored as JSON arrays (e.g., "[]", "['1.2.3.4']"). For an empty list "[]" this currently returns 1 (not 0), which will incorrectly force fan-out validation and can create false UNHEALTHY results. Consider parsing the '@' value as JSON and using the array size, then falling back to comma-counting only for legacy non-JSON fields.
static size_t get_config_list_count(const std::string &table, const std::string &field)
{
    const std::string key = table + "|" + downstream_ifname;
    auto value = mConfigDbPtr->hget(key, field + "@");
    if (value == NULL) {
        value = mConfigDbPtr->hget(key, field);
    }
    if (value == NULL || value->empty()) {
        return 0;
    }

    size_t count = 1;
    for (const char ch : *value) {
        if (ch == ',') {
            count++;
        }
    }
    return count;
}

Copilot AI review requested due to automatic review settings July 28, 2026 17:35
@Xichen96
Xichen96 force-pushed the dev/xichenlin/validate-configured-server-fanout branch from 4a28e04 to 48b958c Compare July 28, 2026 17:35
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@Xichen96
Xichen96 force-pushed the dev/xichenlin/validate-configured-server-fanout branch from 48b958c to ee6e1b4 Compare July 28, 2026 17:39
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/util.cpp:58

  • get_config_list_count() counts DHCP servers by counting commas and assuming the field is non-empty. This overcounts when the CONFIG_DB list contains empty entries (e.g., trailing commas) and can mis-handle bracket/quote wrappers, causing a wrong server_count and false health-check failures.
static size_t get_config_list_count(const std::string &table, const std::string &field)
{
    const std::string key = table + "|" + downstream_ifname;
    auto value = mConfigDbPtr->hget(key, field + "@");
    if (value == NULL) {
        value = mConfigDbPtr->hget(key, field);
    }
    if (value == NULL || value->empty()) {
        return 0;
    }

    size_t count = 1;
    for (const char ch : *value) {
        if (ch == ',') {
            count++;
        }
    }
    return count;

Copilot AI review requested due to automatic review settings July 28, 2026 17:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/util.cpp:56

  • get_config_list_count() counts servers by counting commas and assuming any non-empty string contains at least one entry. This miscounts empty list encodings like "[]" (returns 1) and can overcount when there are trailing/extra separators. That would make fan-out validation expect the wrong multiplier and can cause false health alerts.
    size_t count = 1;
    for (const char ch : *value) {
        if (ch == ',') {
            count++;
        }

Allow first-hop DHCPOFFER, DHCPACK, and DHCPNAK profile validation without a destination-IP assumption. Require each IPv4 VLAN TX packet to appear on either one direct member for unicast or every direct member for broadcast; keep all other parent/member comparisons exact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
Copilot AI review requested due to automatic review settings July 28, 2026 17:49
@Xichen96
Xichen96 force-pushed the dev/xichenlin/validate-configured-server-fanout branch from ee6e1b4 to 1b85476 Compare July 28, 2026 17:49
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/health_check.cpp:31

  • The relay disparity log message is now misleading: the updated positive-health checks can mark the aggregate unhealthy due to fan-out mismatch (TX != RX * configured server count), not only when there is “none transmitted”. This will make alerts harder to interpret during the new validation behavior.
static const char relay_disparity_error[] =
    "dhcpmon detected DHCPv4/v6 packets received but none transmitted for intf: %s. Duration: %d (sec)";

Read configured DHCP server counts from native and legacy CONFIG_DB schemas. Add separate IPv4 and IPv6 group-3 health states that compare relay forward input with configured server fan-out without changing group-1 relay-flow or group-2 management checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 39f979be-d826-4d5c-949a-f20abb58bb83
Signed-off-by: Xichen96 <lukelin0907@gmail.com>
@Xichen96
Xichen96 force-pushed the dev/xichenlin/validate-configured-server-fanout branch from 1b85476 to 12257b7 Compare July 29, 2026 05:03
Copilot AI review requested due to automatic review settings July 29, 2026 05:03
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/dhcp_device.cpp:228

  • Same as the IPv4 fan-out check: returning INDETERMINATE when the DHCPv6 server list is absent can allow a previous unhealthy count to continue increasing and alerting even though the strict fan-out validation is now unsupported. Returning HEALTHY when server_count==0 disables this check cleanly and restores the intended fallback behavior.
    const size_t server_count = get_configured_dhcp_server_count(true);
    if (server_count == 0) {
        return DHCP_MON_STATUS_INDETERMINATE;
    }

src/dhcp_device.cpp:203

  • When no supported DHCP server list is found, this check returns INDETERMINATE. In health_check.cpp, INDETERMINATE does not reset an existing unhealthy count, so a previous fan-out failure can keep counting/alerting even after the server list disappears, which contradicts the PR intent to fall back to the loose forward-path check when no list is present. Consider treating “no list present” as a disabled check that returns HEALTHY (so the state resets) rather than INDETERMINATE.

This issue also appears on line 225 of the same file.

    const size_t server_count = get_configured_dhcp_server_count(false);
    if (server_count == 0) {
        return DHCP_MON_STATUS_INDETERMINATE;
    }

src/dhcp_device.cpp:233

  • The PR description says DHCPv6 fan-out should use a selected forward-input RX delta, but the current implementation sums Solicit/Request/Relay-Forward RX deltas. If Relay-Forward packets can be received in addition to client-originated Solicit/Request, summing can over-count the forward input and falsely flag a disparity. Consider selecting one input signal (e.g., the larger of client-originated vs relay-forward RX deltas) instead of summing them.
    uint64_t rx_delta = 0;
    for (const auto msg_type : monitored_v6_forward_rx_msgs) {
        rx_delta += get_counter_delta(ifname, rx_sock_v6, msg_type);
    }

@Xichen96

Copy link
Copy Markdown
Contributor Author

Closing because configured DHCP server membership can change independently of the sampled packet window, so exact server-count fan-out is not a reliable health invariant.

@Xichen96 Xichen96 closed this Jul 31, 2026
@Xichen96
Xichen96 deleted the dev/xichenlin/validate-configured-server-fanout branch July 31, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants