Skip to content

fix(trending): decode HTML entities in scraped repo descriptions#408

Open
nankingjing wants to merge 3 commits into
gsscsd:mainfrom
nankingjing:fix/decode-html-entities-trending
Open

fix(trending): decode HTML entities in scraped repo descriptions#408
nankingjing wants to merge 3 commits into
gsscsd:mainfrom
nankingjing:fix/decode-html-entities-trending

Conversation

@nankingjing

Copy link
Copy Markdown

Problem

fetchGitHubTrending() in src/trending.ts scrapes repo descriptions and
languages from the github.com/trending HTML. It strips tags with
.replace(/<[^>]+>/g, "") but never decodes HTML entities. GitHub
HTML-escapes these fields, so entities such as &amp;, &#39;, &lt;,
&gt;, &quot; survive into TrendingRepo.description / .language,
flow verbatim into buildTrendingPrompt(), and end up rendered raw in the
generated briefings.

This is reproducible against the live page today — one of the current
trending descriptions is:

An agentic skills framework &amp; software development methodology that works.

which should read ... framework & software ....

Fix

Add a small decodeHtmlEntities() helper and apply it after tag stripping
for both description and language. It resolves named, decimal (&#NN;)
and hex (&#xNN;) references in a single regex pass, so already-decoded
text and unknown entities pass through untouched (e.g. &amp;lt; correctly
stays &lt;).

searchRepos come from the GitHub Search JSON API (already plain text), so
that path is intentionally left unchanged.

Verification (executed, not just read)

  • Fetched the live trending page and ran the existing parser: confirmed a
    real description containing &amp;.
  • Unit-tested the helper against 7 inputs (the live string, mixed
    named/decimal/hex, double-escaped, unknown entity, empty) — all pass.
  • prettier@3 --check with the repo .prettierrc: clean.
  • tsc --noEmit with the repo tsconfig.json (strict,
    noUncheckedIndexedAccess): exit 0.
  • eslint 9.39 + typescript-eslint 8.63 with the repo eslint.config.js:
    exit 0.

Minimal diff, no unrelated reformatting.

github.com/trending descriptions are HTML-escaped; the parser strips
tags but left entities like &amp;, &gsscsd#39;, &lt; intact, so they showed
up raw in the generated briefings. Decode named/decimal/hex entities
after tag stripping. Search API results are already plain text and
unaffected.
@nankingjing

Copy link
Copy Markdown
Author

Reviewed — decodeHtmlEntities handles named/decimal/hex entities in one pass, applied to both description and language fields. Ready for review.

@nankingjing

Copy link
Copy Markdown
Author

Re-reviewed and added a guard for out-of-range numeric entities (e.g. &#x110000;) so String.fromCodePoint cannot throw and silently discard a trending repo. All other behavior is unchanged. Local tsc --noEmit, eslint src, and focused decoder tests pass; CI run is still blocked pending workflow approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant