Power proposal 2 - Enabling software based power measurements

[hanyunfan]

This topic has been discussed by the working group for several weeks, but there does not appear to be sufficient momentum to proceed at this time. As a result, we will move the discussion offline for now.

The detailed proposal can be found here (members only):
https://docs.google.com/document/d/1GMeMD4oXoL0KeXjXm4h-kpsbqNccEKvBvwHYOM7CUTw/edit?tab=t.0#heading=h.wt5klpyit9pz

Please leave a comment if you are interested, and we can continue the discussion offline. If we are unable to gather sufficient interest, this PR will be closed automatically.

[araghun]

@hanyunfan we bought this topic up in the PowerWG a few times I am hoping to start seeing engagement on this issue.

[guthrieg]

Software based measurements will make it significantly easier for organizations to demonstrate the power usage of their inference and other submissions. Instead of purchasing and adding an outside power measurement device, they can leverage the existing software to accurately measure power.

[Sreebhargavibalijaa]

@hanyunfan @araghun ,
The proposal states that aggregation rules will "ensure proportional weighting if nodes handle uneven workloads," but provides no concrete methodology. This is a critical gap for multi-node inference configurations. Specifically:

How will workload distribution be measured and logged, by token throughput, batch count, or wall-clock active time and who is responsible for instrumenting this per node?
If a node is partially idle during an inference run (e.g., a prefill-heavy vs. decode-heavy split), will its idle power be included in the submission total, and under what accounting rules?

The Working Group should define a normative aggregation specification, not leave it to submitters to prevent divergent interpretations that could make multi-node results incomparable across organizations.

[zhangbinbj2048-design]

The proposal will reduce the dependence on expensive physical power meters,but there are two issues that need attention:

1.When using software to read power, it is necessary to unify the sampling frequency, collect duration, and calculation method for data collection. This avoids excessive deviation of results caused by inconsistent collection logic among different customers.
2.It is required to distinguish the adaptation rules of software power consumption under DC and AC power supply, and mandatorily mark the power supply type in the reporting template.

[ShaohuiLiu]

Agree with @guthrieg. Software-based measurements could significantly lower the barrier for organizations to report power usage, especially when external power meters introduce additional cost and difficulty of deploy.

One caveat is that software measurements may introduce additional error or bias depending on the tool, firmware, driver stack, and platform-specific exposure. This may make standardization and cross-submission comparability more difficult. The practical difficulty also varies by platform: for example, IPMI availability/exposure is not always consistent; some HGX servers may require sudo/root access for RAPL or other telemetry; while GH200/GB200/GB300 platforms can often expose node-level power more conveniently through DCGM.

So I support enabling software-based measurements, but it would be helpful to define clear requirements around accepted telemetry sources, calibration/validation expectations, sampling frequency, and how uncertainty/error should be reported.

[hanyunfan]

Just my two cents:

1. We may need more clarity on how this can be implemented in details.

2. Hardware calibration with SPEC meters is not straightforward and still the roadblocker. It would be ideal if we could eliminate or simplify this validation requirement.

3. We need a reliable way to ensure that software-based measurements are trustworthy and consistent. i.e., that all tools measure the same metrics in the same way. Standardizing a single tool or methodology may help address this.

4. If the above criteria cannot be fully met, we should consider treating software-based measurements as reference-only data points, rather than allowing direct comparisons between systems, or maybe first allow it in Open division only.

[guthrieg]

@hanyunfan any recommendations on what is required to sufficiently validate the software-based approach? We have some initial data that suggested the software approach is >97% accurate (quoting from memory).

[hanyunfan]

@hanyunfan any recommendations on what is required to sufficiently validate the software-based approach? We have some initial data that suggested the software approach is >97% accurate (quoting from memory).

This is one of the more challenging aspects. We would either need to rely on IPMI/DCMI or Redfish implementations from system vendors, as it is ultimately their responsibility to ensure accurate power data is exposed to end users, or/and classify software-based measurements as unverified. We could consider automatically integrating a unified power command into inference submission runs, either in-band or out-of-band, where feasible.

While there is no perfect solution today due to the lack of standardized OOB power implementations, I believe this approach is simple enough to encourage power submissions without introducing significant overhead.

[dslik]

One approach is to cross-reference the power data reported by a RedFish-enabled PDU and reported by IPMI/RedFish enabled BMC for each of the servers to see if there is agreement within a certain bound.

I am working on a power data collector for RedFish that will be capable of doing this. The submitter would need to launch this on each host and provide read-only RedFish credentials for each host, plus the IP address(es) and credentials for the PDU(s).

We will also need to do a survey of how widely the RedFish SensorEnergykWhExcerpt LifetimeReading property is supported by BMC and PDU vendors, how many significant figures are provided, and if the vendors will provide/stand behind any accuracy numbers, since cumulative power measurements will be required, along with enough significant figures to accurately measure the consumed power.