Skip to content

Fix issue counts always returning zero - #830

Open
ruohai0925 wants to merge 1 commit into
ossf:mainfrom
ruohai0925:fix-issue-count-cursor-pagination
Open

Fix issue counts always returning zero#830
ruohai0925 wants to merge 1 commit into
ossf:mainfrom
ruohai0925:fix-issue-count-cursor-pagination

Conversation

@ruohai0925

Copy link
Copy Markdown

Summary

Fixes #816.

The three legacy issue signals — updated_issues_count, closed_issues_count
and issue_comment_frequency — currently report 0 for every repository.

GitHub has migrated /repos/{owner}/{repo}/issues to cursor-based pagination.
Its Link header now carries only a rel="next" entry with an opaque cursor,
and no rel="last":

Link: <https://api.github.com/repositories/150626842/issues?state=all&since=2026-05-04T00%3A00%3A00Z&per_page=1&after=Y3Vyc29yOnYyOpLPAAABn8D-mnjPAAAAASx2iuA%3D&page=2>; rel="next"

FetchIssueCount requests per_page=1 and reads the total off resp.LastPage,
which is now always 0, so it returns 0. Since issue_comment_frequency
divides the comment count by the updated-issues count, and IssuesSource.Get
short-circuits when that count is zero, the comment signal is zeroed as well.

Change

When the total cannot be read off the Link header, page through the results
and count them instead. The rel="last" fast path is kept for when the API
still provides it, so nothing changes for endpoints or deployments that still
return it, and the MaxIssuesLimit cap is preserved so large repositories stop
paging at 5000 rather than walking an unbounded number of pages.

FetchIssueCommentCount and FetchTotalContributors are deliberately left
unchanged — I checked both endpoints and they still return rel="last", so the
per_page=1 trick remains valid there:

# /issues/comments
Link: <...&per_page=1&page=2>; rel="next", <...&per_page=1&page=347>; rel="last"

# /contributors
Link: <...&per_page=1&page=2>; rel="next", <...&per_page=1&page=65>; rel="last"

Testing

make test and make lint both pass.

internal/collector/github/legacy had no test coverage at all, so this adds
issues_test.go with httptest-backed cases for the cursor-pagination
fallback, the rel="last" fast path, the single-result and empty cases, the
5xx → MaxIssuesLimit fallback, the paging cap (and its request count), and
the since lookback window.

Verified end to end against two repositories, with -depsdev-disable:

repo signal before after
BLAST-WarpX/warpx updated_issues_count 0 690
closed_issues_count 0 572
issue_comment_frequency 0 0.5
default_score 0.53654 0.62253
ruohai0925/IAMReX updated_issues_count 0 2
closed_issues_count 0 2
issue_comment_frequency 0 0
default_score 0.38916 0.40145

The counts match what the GitHub search API reports for the same 90-day window
(e.g. 690 issues + PRs updated for WarpX).

Relationship to #829

#829 fixes the same bug by switching to the search API, which is a cleaner and
cheaper approach — one request instead of up to 50. This PR takes the paging
route, which keeps the existing endpoint and its exact semantics and avoids the
search API's separate rate limit, and additionally adds test coverage for the
package.

I have no attachment to my version: if maintainers prefer #829, I'm happy to
close this and port the tests over to that PR instead. Opening this mainly
because #829 has been sitting unreviewed since July and the bug silently
distorts every published score.

GitHub migrated the repo issues list endpoint to cursor-based pagination.
Its Link header now carries only a rel="next" entry with an opaque cursor
and no rel="last", so the per_page=1 trick used by FetchIssueCount reads
LastPage as 0 and returns 0 for every repository.

This zeroes updated_issues_count and closed_issues_count, and because
issue_comment_frequency divides the comment count by the updated count,
that signal is zeroed too.

Fall back to paging through the results and counting them when the total
cannot be read off the Link header. The rel="last" fast path is kept for
when the API still provides it, and the MaxIssuesLimit cap is preserved so
large repositories do not page indefinitely.

The comments and contributors endpoints still return rel="last", so
FetchIssueCommentCount and FetchTotalContributors are unchanged.

Adds httptest-backed tests for FetchIssueCount, which had no coverage.

Signed-off-by: ruohai0925 <ruohai372@gmail.com>
@ruohai0925
ruohai0925 force-pushed the fix-issue-count-cursor-pagination branch from 76d4d77 to 7c6b1b9 Compare August 2, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

There's abnormal for command of 'criticality_score'

1 participant