Skip to content

fix: decode numeric character references in text - #4076

Open
Kjubikstronk wants to merge 2 commits into
markedjs:masterfrom
Kjubikstronk:fix-numeric-char-refs
Open

fix: decode numeric character references in text#4076
Kjubikstronk wants to merge 2 commits into
markedjs:masterfrom
Kjubikstronk:fix-numeric-char-refs

Conversation

@Kjubikstronk

Copy link
Copy Markdown
Contributor

Marked version: 18.0.11 (53cb13f)

Markdown flavor: CommonMark

Description

CommonMark treats a numeric character reference as equivalent to the character it names, everywhere except code spans and code blocks. marked passes them through unchanged, so they reach the output exactly as written.

escapeTestNoEncode deliberately declines to escape anything shaped like an entity:

/[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/

which leaves it to the browser to resolve. That works for display but does not match the spec, and it is visible to anything reading the output as text rather than rendering it.

Expectation

&#35; &#1234;   ->  # Ӓ
&#X22; &#xcab;  ->  &quot; ಫ
foo&#10;&#10;bar -> foo, blank line, bar
&#9;foo         ->  a tab, then foo

Result

Each one is emitted as written: &#35;, &#X22;, &#10;, &#9;.

What was attempted

Decoding numeric references in inlineText, so the token carries the character and the existing escaping handles the rest. Zero, out of range, and surrogate values become the replacement character, per the spec. Raw HTML blocks keep whatever they were written with.

This deliberately does not touch named references. &copy; and friends need the HTML5 named table, which is the dependency question raised in #4050. The numeric half needs no table at all, only digit parsing, which is why it is separate.

Measured over the whole spec with the spec's own test/normalize.py: 28 failures before, 24 after. The four that move are 26, 27, 39 and 40, and nothing else changes, in both the CommonMark and GFM runs.

Worth noting why this was not caught: &#35; and # are equivalent once parsed as HTML, so html-differ reports them equal. All four of these already report as passing and carry no shouldFail. Same blind spot as #4073, #4074 and #4075, in a different corner.

Contributor

  • Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
  • no tests required for this PR.
  • If submitting new feature, it has been documented in the appropriate places.

Five tests in test/unit/marked.test.js under numeric character references. Three fail without the change; the other two are controls, an invalid reference that must stay literal and a code span that must not decode. Full spec suite (1789) and unit suite (196) pass.

Committer

In most cases, this should be a different person than the contributor.

CommonMark treats a numeric character reference as equivalent to the character
it names, everywhere except code. marked passed them through unchanged, so
`&markedjs#35;` reached the output as written instead of `#`.

Fixes CommonMark examples 26, 27, 39 and 40. Named references still pass
through, since resolving those needs the HTML5 table.

The spec suite could not see this: `&markedjs#35;` and `#` are equivalent once parsed
as HTML, so html-differ reports them equal and all four already passed.
@vercel

vercel Bot commented Aug 30, 2026

Copy link
Copy Markdown

@Kjubikstronk is attempting to deploy a commit to the MarkedJS Team on Vercel.

A member of the Team first needs to authorize it.

@UziTech UziTech left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move specs to test/specs/new with renderExact

Uses renderExact so the fixture compares exactly, which htmlIsEqual
does not, and keeps the assertions out of marked.test.js.
@Kjubikstronk

Copy link
Copy Markdown
Contributor Author

Moved to test/specs/new/numeric_character_references.md with renderExact in e612c78, and marked.test.js is untouched again.

All five cases in one fixture: decimal, hex in both cases, the replacement character for zero, an invalid reference left alone, and the code span control. Reverting the decode fails it.

@UziTech UziTech left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! 💯

@vercel

vercel Bot commented Sep 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
marked-website Ready Ready Preview Sep 3, 2026 3:47am UTC

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants