Skip to content

[Cisco] Fix fboss2 interface traffic hangs and missing rates - #1626

Open
selekkala wants to merge 3 commits into
facebook:mainfrom
selekkala:fix/fboss2-interface-traffic-rates
Open

selekkala wants to merge 3 commits into
facebook:mainfrom
selekkala:fix/fboss2-interface-traffic-rates

Conversation

@selekkala

Copy link
Copy Markdown
Contributor

Summary

Fix fboss2 show interface traffic on OSS multi-switch systems where the command can hang indefinitely or return an empty/zero table while ports are carrying traffic.

This change:

  • uses deterministic synchronous RPCs for the command's sequential data collection;
  • exports the SUM and RATE types requested by MonotonicCounter callers, making port .rate.60 counters available from the HW agent;
  • fetches only the traffic-rate counter families needed by this command;
  • skips down ports, which may legitimately have no HW-agent rate counters;
  • reports a missing counter on an up port instead of silently converting it to zero; and
  • preserves fractional Kpps values and guards per-port zero bandwidth.

The aggregate zero-bandwidth fix from #1560 is already present on main and remains unchanged.

Root cause

There were three contributing problems:

  1. CmdShowInterfaceTraffic::queryClient() mixed future-based RPCs attached to a short-lived local IOThreadPoolExecutor with blocking HW-agent fanout, followed by an unconditional wait(). If the future did not reach a terminal state on a cold/multi-switch path, the command had no forward progress and remained blocked. The operation is sequential at the point the model is created, so the local executor did not provide useful concurrency.
  2. The OSS MonotonicCounter implementation created its backing stat with getLockableStatNoExport() and ignored both requested export types. Port counters were updated internally, but .rate.60 was therefore absent from FB303 output.
  3. The CLI read rate counters with std::map::operator[]. Missing counters were inserted as zero, making unavailable telemetry indistinguishable from a real zero traffic rate. It also attempted to read current rate counters for down ports.

Fix details and compatibility

  • The Thrift APIs and output schema are unchanged.
  • Both monolithic and multi-switch paths remain supported.
  • MonotonicCounter now honors its existing constructor contract. Its swap() implementation also transfers the state, name, and lockable stat so clear/rename operations retain the correct counter lifecycle.
  • The FB303 regex is anchored to exact .rate.60 suffixes, reducing the counter payload compared with fetching every agent counter.
  • Down ports are excluded before rate lookup. An up port without required telemetry fails explicitly instead of displaying misleading zeros.
  • Existing error-counter reporting is preserved for both up and down ports.

Unit-test coverage

  • synchronous query path and exact .rate.60 regex;
  • missing rate counter on an up port;
  • down port without rate counters;
  • fractional Kpps conversion;
  • zero-bandwidth utilization;
  • monotonic counter rate export, removal, and port-name reinitialization; and
  • existing empty-traffic aggregate output coverage from [Nexthop] Fix nan% issue in "show interface traffic" CLI output #1560.

Test Plan

  • pre-commit run --from-ref upstream/main --to-ref HEAD — passed, including clang-format and repository checks.
  • Built a full C8501 FBOSS image carrying the equivalent backport: fboss-gr2-stable-20260903-fbc276b.1-1-ge405635.
  • Validated on a Cisco C8501 / Morgan800CC lab switch:
    • enabled all 128 physical ports; two connected ports came up;
    • verified all 1,024 expected physical-port traffic .rate.60 counters (128 ports × 8 counters) were exported;
    • ran 70 no-traffic/below-threshold invocations and 30 active-traffic invocations: 100/100 returned successfully, with zero hangs or timeouts;
    • observed command latency between 32 ms and 64 ms;
    • generated a controlled traffic burst on eth1/1/1 and observed:
hwagent0  eth1/1/1.out_bytes.rate.60  52195

Interface Name  Peer  Intvl  InMbps  InPct  InKpps  OutMbps  OutPct  OutKpps
eth1/1/1        --    0:60   0.00    0.00%  0.00    0.42     0.00%   0.04
Total           --    --     0.00    0.00%  0.00    0.42     0.00%   0.04
  • Confirmed fboss_sw_agent, fboss_hw_agent@0, and qsfp_service remained active after validation.

@selekkala
selekkala requested review from a team as code owners September 11, 2026 14:24
@meta-cla meta-cla Bot added the CLA Signed label Sep 11, 2026
@github-actions github-actions Bot added the cisco label Sep 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant