fix(rich-markdown): demote links whose URL Telegram would mangle - #25
Merged
Conversation
Telegram's own server-side markdown parser HTML-escapes `&` (-> `&`) and `'` (-> `'`) inside a link destination, so an ordinary query-string link arrives pointing at `?action=view&handbook=235` — a URL the target server reads as a parameter named `amp;handbook`. One real note went out with 19 broken links. `scripts/spike_rich_link_escaping.py` (new) proved there is no spelling of the link that avoids it: `&` in the source comes back double-escaped, `\&` as `\&`, an autolink and an inline `<a href>` are escaped exactly like a plain link, `%26` survives but changes what the target parses, and a pointy-bracket destination makes no link at all. Its `--mode chars` run drew the character list: only `&` and `'` are mangled; `+`, `%20`, `#`, `~`, `|`, `_`, `*` and non-ASCII survive untouched. A bare URL in the text is the one form that works — the parser stores no link entity for it and the clients autodetect it. So `unwrap_unsafe_links()` rewrites `[269 - AWRA](https://x/?a=1&k=2)` to `269 - AWRA: https://x/?a=1&k=2`, keeping the anchor text because it is usually the only thing naming the target. Links whose URL carries neither character stay markdown links — they work and read better. Media embeds are excluded by the pattern's leading `(?<!!)`, and code spans are skipped by containment rather than overlap, the rule the wikilink and media sweeps already use. Like the wikilink pass it has no knob and is not CLI-only: the parser doing the damage is Telegram's. It can only shrink the text, so it stays out of the over-limit `grew_by` list. Verified live (Saved Messages, msg 409106): the same note read back through `messages.getRichMessage` now carries 0 occurrences of `&`, 18 intact `&handbook=235&key=` URLs and 0 `TextUrl` entities. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Telegram's own server-side markdown parser HTML-escapes
&(→&) and'(→') inside a link destination. An ordinary query-string link therefore arrives pointing at?action=view&handbook=235, which the target server reads as a parameter namedamp;handbook. Nothing on our side rewrote it — one real note went out with 19 broken links.What the spike proved
scripts/spike_rich_link_escaping.py(new) sends one article per mode to Saved Messages and reads it back throughmessages.getRichMessage, printing theTextUrl.urlthe server stored.--mode escaping— no spelling of the link avoids the escaping:[a](…&handbook=235)…&handbook=235[a](…&handbook=235)…&amp;handbook=235[a](…\&handbook=235)…\&handbook=235<https://…&handbook=235>…&handbook=235<a href="https://…&handbook=235">…&handbook=235[a](<https://…>)…%26handbook=235--mode chars— only&and'are mangled.+,%20,#,~,|,_,*and non-ASCII survive untouched;<,>,"prevent a link from forming at all (a separate, pre-existing markdown-level issue, out of scope).A bare URL in the text is the one form that works: the parser stores no link entity for it and the clients autodetect it.
The fix
unwrap_unsafe_links()inmessages/rich_markdown.pyrewrites[269 - AWRA](https://x/?a=1&k=2)→269 - AWRA: https://x/?a=1&k=2.UNSAFE_LINK_URL_CHARS(&,') — a working markdown link reads better than a bare URL, so clean links are untouched.) is excluded by the pattern's leading(?<!!); code spans are skipped by containment, not overlap — the rule the wikilink and media sweeps already use.grew_bylist.--dry-runreportsrich_markdown_unwrapped_links.Verification
pytest— 2275 passed (2240 + 35 new: 32 intests/test_rich_markdown_links.py, plus CLI dry-run, HTTP and MCP surface tests)ruff check src tests scripts— clean&, 18 intact&handbook=235&key=URLs, 0TextUrlentitiesManual checks for a reviewer
messages send --rich-markdown <note-with-query-links> --dry-runreports a non-zerorich_markdown_unwrapped_links&and confirm they are still proper markdown links (blue anchor text, not a raw URL)🤖 Generated with Claude Code