What we are asking for
Ask 3 means the target, the rate a stream has to hold to pass, how many consecutive measurement
windows have to agree before the stream count changes, and how long a window is. Four numbers and
a sentence, all of them in hand at the point the result is written.
Acceptance criteria
- The per-stream target comes from the camera config's declared rate, with
TARGET_FPS still
able to override it.
- The result states the target, the pass mark and the measured rate as three separate numbers.
- The result states the pass rule, the window count and the window length in words.
- Every number in the result can be found in, or derived from, the raw log without reading source.
Context
Why 15, and what it does to the pass mark
Ask 1 gives a headline target of 15 taken from the camera's own declared rate, with a pass mark
of 14.25, which is 95% of it. That is the same arithmetic as today to within 0.05 fps, now a
stated rule instead of a constant. It also fixes a case nobody has hit yet: anyone benchmarking
a 30 fps camera today inherits a target derived from somebody else's 15 fps setup. TARGET_FPS
stays available for anyone who wants to override it.
What we found
A stream density result reports 14.95 fps. No camera achieved 14.95 fps. The raw log for the
same run contains 14.2025, and neither the summary nor the documentation explains how the two
relate, so a reader has to open the source to find out.
Three things combine to produce that:
The achieved per-stream rate is computed for the pass decision and then dropped before the
result is written. The documented return shape at stream_density.py:934-936
is (target_fps, container_name, num_pipelines, meet_target_fps): the target that was asked
for, and a boolean saying whether it was met. The rate the streams actually held is not in it,
so a run whose cameras held the pass mark is recorded as 14.95.
Separately, every camera config already declares "fps": 15. build_per_stream_target_fps at
stream_density.py:41 works out each stream's target, and at stream_density.py:70 it reads a different,
optional key called targetFps, falling back to the global 14.95 when that key is missing,
which it is in every config that ships.
Why it matters
Two independent reviews of the ASC benchmark could not reconcile the reported number against the
logs the same run produced. One of them took two days. A partner who hits that gap will conclude
the number was chosen rather than measured, and we do not get to explain afterwards.
What the result would say
Stream density result
-----------------------------------------------
Streams sustained 12
Target per camera 15.00 fps (declared source rate)
Pass mark per camera 14.25 fps (95% of source rate)
Measured, slowest camera __.__ fps
Measurement window ___ s (after settle)
A camera passes when it sustains at least 95% of the
rate its source publishes. Every camera must pass for
the stream count to be accepted. The stream count is
only changed after two consecutive windows agree.
Field names are a suggestion. The window length is blank because it is not defined anywhere
today, which is a separate issue.
Related
#245 changes the statistic the pass decision is taken on. This issue changes what the result
reports about that decision. They touch the same summary code and are easier to do together than
one after the other.
#232 asks for per-stream input FPS sourced from camera_to_workload_*.json for heterogeneous
camera sets. This issue is the narrower case of the same mechanism: read the declared rate
rather than a constant. Whoever picks either one up should look at both.
Line references link to performance-tools @ 144d3b6 and loss-prevention @ f865f5b, read 31 Aug 2026.
What we are asking for
Ask 3 means the target, the rate a stream has to hold to pass, how many consecutive measurement
windows have to agree before the stream count changes, and how long a window is. Four numbers and
a sentence, all of them in hand at the point the result is written.
Acceptance criteria
TARGET_FPSstillable to override it.
Context
Why 15, and what it does to the pass mark
Ask 1 gives a headline target of 15 taken from the camera's own declared rate, with a pass mark
of 14.25, which is 95% of it. That is the same arithmetic as today to within 0.05 fps, now a
stated rule instead of a constant. It also fixes a case nobody has hit yet: anyone benchmarking
a 30 fps camera today inherits a target derived from somebody else's 15 fps setup.
TARGET_FPSstays available for anyone who wants to override it.
What we found
A stream density result reports 14.95 fps. No camera achieved 14.95 fps. The raw log for the
same run contains 14.2025, and neither the summary nor the documentation explains how the two
relate, so a reader has to open the source to find out.
Three things combine to produce that:
loss-preventionloss-prevention Makefile:45performance-toolsstream_density.py:34The achieved per-stream rate is computed for the pass decision and then dropped before the
result is written. The documented return shape at stream_density.py:934-936
is
(target_fps, container_name, num_pipelines, meet_target_fps): the target that was askedfor, and a boolean saying whether it was met. The rate the streams actually held is not in it,
so a run whose cameras held the pass mark is recorded as 14.95.
Separately, every camera config already declares
"fps": 15.build_per_stream_target_fpsatstream_density.py:41 works out each stream's target, and at stream_density.py:70 it reads a different,
optional key called
targetFps, falling back to the global 14.95 when that key is missing,which it is in every config that ships.
Why it matters
Two independent reviews of the ASC benchmark could not reconcile the reported number against the
logs the same run produced. One of them took two days. A partner who hits that gap will conclude
the number was chosen rather than measured, and we do not get to explain afterwards.
What the result would say
Field names are a suggestion. The window length is blank because it is not defined anywhere
today, which is a separate issue.
Related
#245 changes the statistic the pass decision is taken on. This issue changes what the result
reports about that decision. They touch the same summary code and are easier to do together than
one after the other.
#232 asks for per-stream input FPS sourced from
camera_to_workload_*.jsonfor heterogeneouscamera sets. This issue is the narrower case of the same mechanism: read the declared rate
rather than a constant. Whoever picks either one up should look at both.
Line references link to
performance-tools@144d3b6andloss-prevention@f865f5b, read 31 Aug 2026.