diff --git a/CHANGELOG.md b/CHANGELOG.md index 509db77..8e394c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,12 @@ # Changelog +## Unreleased + +### Miscellaneous + +- Regenerate changelog + ## v0.11.1 - 2026-08-01 ### Build diff --git a/CLAUDE.md b/CLAUDE.md index 279e1fa..667359f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,6 +32,7 @@ All runtime state — `config.yml`, Telethon session, SQLite DB, bearer token - Rich-**media** spike (same preconditions and exit codes, **uploads and sends a real file**): `.venv/bin/python scripts/spike_rich_media.py --file [--entity me] [--only tg-scheme]` — it sends one article per candidate reference syntax, which is how the `tg://` dialect was proven; `--dry-run` prints the candidate articles without uploading. - Rich **line-break** spike (same preconditions and exit codes, **sends one real text-only article per candidate**): `.venv/bin/python scripts/spike_rich_line_breaks.py [--entity me] [--only blank-line]` — it is how the dialect's handling of a single newline inside a paragraph was proven (folded into a space; two trailing spaces / `\` / `
` all give a real in-paragraph break; a blank line gives two tightly rendered paragraphs, which is what the shipped pass emits); `--dry-run` prints the candidates without sending. - Rich **group-caption** spike (same preconditions and exit codes, **uploads two generated 1×1 PNGs and sends one article per candidate**): `.venv/bin/python scripts/spike_rich_collage_caption.py [--entity me] [--only figcaption-block]` — it is how the `
` spelling of a ``/`` caption was proven; `--dry-run` prints the candidate articles without uploading. +- Rich **link-escaping** spike (same preconditions and exit codes, **sends one real article per mode**): `.venv/bin/python scripts/spike_rich_link_escaping.py [--entity me] [--mode escaping|chars]` — it is how the `&`/`'` escaping in link destinations was proven and how the surviving-characters list was drawn; it reads each sent article back through `messages.getRichMessage` (the message's own `rich_message` is a truncated `part=True` preview) and prints the `TextUrl.url` the server stored. `--dry-run` prints the article without sending. - Rich **GIF** spike (same preconditions and exit codes, plus a missing `ffmpeg`, **uploads an animated GIF and sends one real article per candidate**): `.venv/bin/python scripts/spike_rich_gif.py --file [--entity me] [--only gif-stub,gif-probed,mp4-converted]` — it is how the undocumented server-side transcoding size threshold and the ffmpeg-converted-mp4 fallback were proven (see the `.gif` wire facts under "Local media in an article is CLI-only" below); `--dry-run` prints the candidates without uploading. The Telethon session is created **only** by `telegram-assistant auth` (interactive — prompts for phone, code, optional 2FA). There is no HTTP endpoint for login. Re-running `auth` for an authorized session prints the bound account and exits without re-prompting. @@ -58,7 +59,7 @@ Six cross-cutting behaviours in `messages/` that surfaces must not re-implement: - **Rich sends bypass the high-level client.** `SendMessageRequest.rich_markdown` (CLI `--rich-markdown `, HTTP/MCP `rich_markdown` string) sends a Telegram *article* — headings, tables, quotes, code, media by public HTTPS URL or (CLI only) by uploaded local file — parsed **server-side**, so nothing here builds a PageBlock tree and the 4096-char split never applies. `send_message` validates it (non-empty, `<= MAX_RICH_MARKDOWN_CHARS` = 32 768 inclusive, mutually exclusive with `text`/attachments) and passes it down through the **only-when-set** `extra` kwargs block, so backends predating rich sends keep working — that contract is pinned by a legacy-signature fake on every surface (`LegacySendBackend`, `CliLegacyMessageBackend`, `test_mcp_plain_send_omits_rich_markdown_kwarg`) rather than by asserting a recorded `None`. Telethon's `client.send_message` has no `rich_message` parameter, so `TelethonMessageBackend._send_rich_message` issues a raw `functions.messages.SendMessageRequest(message="", rich_message=InputRichMessageMarkdown(...))` and reads the id back from the `UpdateMessageID` whose `random_id` matches the request's own — an `Updates` container may carry another request's updates, and a *keyed* update naming a different `random_id` is never used as a fallback: its id would be returned as the article's and recorded in the `SentMessageRegistry`, handing the session edit/delete rights over a message it never sent (Telethon's own sender keys strictly on `random_id` for the same reason). An unkeyed entry, or any entry at all when the request's own `random_id` is unreadable, is still accepted. From there it falls back to `UpdateNewMessage`/`UpdateNewChannelMessage`/`UpdateNewScheduledMessage` — **minus** the ids named by those rejected foreign-keyed `UpdateMessageID`s, since Telegram pairs each `UpdateMessageID` with the `UpdateNew*Message` for the *same* message and an unfiltered scan would hand back the very id the `random_id` check just refused (only that id is excluded: another `UpdateNew*Message` may still be ours when our own `UpdateMessageID` is missing) — and finally to a bare `UpdateShortSentMessage`/`UpdateShort` envelope. That last read is gated on the type name being `UpdateShortSentMessage`: `UpdateShortMessage`/`UpdateShortChatMessage` are *incoming*-message envelopes carrying someone else's `id` and expose neither `.updates` nor `.update`, so an ungated `getattr(result, "id")` would claim it (Telethon's own extractor returns nothing for them). `InputRichMessageMarkdown` arrived in **telethon 1.44 (layer 227)**, so it is imported through a version-tolerant probe (`_import_rich_markdown_type`, the `topics/telethon_backend.py` pattern) — an older Telethon fails only the rich send, with a version message, not the whole import. Mass mode has no rich path at all (`MassSendRequest` has no such field), so that exclusivity lives on the surfaces (HTTP `MessageSendBody._shape`, the CLI's `--mass` check), and there is deliberately **no** silent fallback to a plain-text send: a server-rejected article surfaces through the normal send-error path and the caller decides. Two rich-specific failures deliberately stay **out** of the `MessageSendFailed` class, because that one means "a previous attempt with this idempotency key failed" and every surface renders it as `previous_attempt_failed` (HTTP/MCP 409, CLI exit 2): an old Telethon raises `RichMessageUnsupported` (a plain `RuntimeError`, so it lands on 500 / exit 1 and names the version, not a phantom prior attempt — the HTTP route maps it explicitly to a 500 whose body carries `{"error": "rich_message_unsupported", "message": ...}`, because an unmapped `RuntimeError` would surface as Starlette's *empty* 500 and drop the version hint, and `send_message` **drops the just-opened operation row** rather than failing it: the check runs before any RPC, so the retry after the upgrade — same `operation_id` — must send instead of replaying `previous_attempt_failed` or a stuck `pending`), and an envelope with no readable id raises `MessageSendUnconfirmed`, which `send_message` catches next to `FloodWaitError` and marks **`needs_review`** — the request reached Telegram, so the article may be delivered, and a terminal `failed` would tell the caller nothing happened and invite a duplicate re-send. -- **`messages/rich_markdown.py` owns every rewrite of an article, and every pass edits only the line ranges — or, for wikilinks, the in-line spans — it owns.** There is no markdown parser in the loop (the Telegram dialect diverges from CommonMark — `==mark==`, `||spoiler||`, ``, footnotes — and a parse→render round-trip would rewrite the author's text): `scan_blocks()` is a line-based scanner producing typed `Block`s, and every pass edits only the line ranges (or, for `strip_wikilinks`, the character spans within a line) it owns, so each pass returns the **input string by identity** when it changes nothing — that is what keeps CRLF and the trailing newline intact for a byte-for-byte send. Nesting is bounded by `MAX_BLOCK_NESTING` (64): past that depth `_scan_nested()` returns `()` and the quote/HTML container is emitted as a **leaf** whose `lines` still carry the whole body (so the media sweep, which skips only lines a *child* owns, still resolves references buried inside it). The scanner recurses once per level, and a single line of `"> " * 600` is 1.2 KB — far under `MAX_RICH_MARKDOWN_CHARS` — so without the bound it raises `RecursionError`; that scan runs on the event loop *ahead* of the WRITE gate, where a `RuntimeError` subclass misses every `except ValueError` on the surfaces and lands on an unmapped, empty 500 for a caller who may hold no write grant. YAML frontmatter is **not** the scanner's business: `strip_yaml_frontmatter()` is called by the CLI at the file-read boundary, next to the `utf-8-sig` BOM strip and for the same reason (that is where "this is a note file" is known), because the scanner reads a note's opening `---` as a divider and its `key: value` lines as a *setext heading* underlined by the closing `---` — an Obsidian note would otherwise open with a rule and a large heading reading its own metadata, with a spacer inserted inside it. It strips only an exact `---` first line closed by a matching `---` **whose enclosed lines read as YAML** (`_is_frontmatter_body`: a mapping entry first, then only mapping entries, `- ` items, indented continuations, comments or blank lines — a comment is accepted only *after* the first line, since its syntax is an ATX heading's and the mapping-entry-first rule is the whole fence against swallowing an article that opens with a rule and a heading; rejecting the block over one would leave the fences in and send the note's metadata as a rule plus a large heading) — the fences alone are not enough, since an article that opens with a `---` rule and uses `---` dividers later would otherwise have its whole first section silently dropped, and nothing downstream would report it (the CLI never echoes the body). It slices the remainder out of the original text (so CRLF and the trailing newline survive), returns the input by identity otherwise, and runs *before* the CLI's emptiness check so a file that is only frontmatter is reported empty. HTTP/MCP take a markdown string an agent composed rather than a note, so their input is never rewritten this way. `normalize_rich_markdown()` runs **strip → group → space → split → count** in that order, so the reported block count and the 500-block rollback see the text with wikilinks already expanded, the containers, *and* the paragraphs the send actually carries. *Stripping* (`strip_wikilinks`/`_expand_wikilink`) expands every Obsidian `[[target]]`/`[[target|alias]]` to plain text: one rule, no special cases — the alias wins when present, otherwise the target reads as Obsidian renders it, with a leading `#` (a link into the current note) dropped and every other `#` replaced by ` > ` (block references, `note#^blk`, fall out of that same rule unchanged, needing no branch of their own). Only the **first** `|` separates target from alias, so further pipes stay in the alias (`[[A|B|C]]` → `B|C`); an empty half falls back to the other (`[[Note|]]` → `Note`, `[[|Стас]]` → `Стас`); and a link with **both** halves empty (`[[]]`, `[[|]]`) is not a link at all — it ships verbatim, since silently deleting characters the author typed is worse than leaving a curiosity in the article. `![[…]]` embeds are untouched — the absent `!` is the whole discriminator, because that syntax is media and belongs to `scan_media`, not this pass — and so is anything inside a fenced code block (via `scan_blocks`'s own `code` blocks) or an inline code span (reusing `_CODE_SPAN_RE` rather than a second matcher); the code-span check is **containment, not overlap** (`span_start <= start and end <= span_end`), the same rule `iter_line_media_refs` uses, so a backtick merely touching a link's alias does not shield the link. It runs **first**, ahead of `scan_blocks` and every other pass, so the reported block count, the media sweep, and the spacer/split/group passes all see the text Telegram will actually receive rather than literal brackets — and a table cell whose wikilink pipe would otherwise split the row is settled before the table is even scanned. A link nested inside its own target/alias (`[[[[a]]]]`) needs more than one pass to fully resolve — the regex body excludes brackets, so one call only unwraps the innermost pair — so a single call loops up to `MAX_WIKILINK_PASSES` (8, each pass a full document rescan on the event loop ahead of the WRITE gate) times; it is idempotent up to that nesting depth (a second pass over its own output finds no `[[` left to expand, unless the source nested past the cap, in which case the leftover ships verbatim like a degenerate `[[]]`) and, like every pass here, returns the input string by identity when there was nothing to expand. There is **no knob** for it: a literal `[[…]]` reaching a Telegram reader is always a defect, never a style choice worth preserving. And unlike `strip_yaml_frontmatter` and `scan_media`, which answer "this came from a vault file" and are therefore CLI-only, this pass is **not** CLI-only — a wikilink is meaningless in Telegram no matter which surface submitted the markdown, so it runs for CLI, HTTP and MCP alike. *Splitting* (`_split_paragraph_lines`, knob `line_breaks`) inserts a blank line between the lines of a top-level paragraph, because Telegram's parser folds a single newline into a space and an Obsidian note's two lines under one another would arrive as one run-on line; the resulting paragraphs render tight, which is the whole point, so it runs **after** spacing — the spacer pass must never see the pairs it produces. `scripts/spike_rich_line_breaks.py` (2026-07-27, Saved Messages) proved that a real in-paragraph hard break (two trailing spaces, a trailing `\`, `
`) is *also* accepted and keeps the pair in one block; the split is emitted anyway because it reads better in the clients, and the cost — the one place this module gives up idempotency — is that a split pair is indistinguishable from two author paragraphs, so re-normalising this pass's own output would let spacing push them apart (nothing in a send does that: `send_message` normalises once, from the author's source). *Spacing* inserts a U+00A0-only line as its own block between two consecutive top-level paragraphs, before any heading, and after any medium — a `media` block or the ``/`` a run was grouped into, since the container *is* the media as far as the article's vertical rhythm goes, while `
` is not — but never *before* media (an embed's lead-in line belongs with it), never after a heading, never adjacent to an existing spacer, never inside code/table/list/quote/html — U+00A0 is why `_is_blank()` is "whitespace **and** no NBSP", or the scanner would swallow the module's own spacers and double them on every re-normalisation, and why the block count is honest: Telegram charges a spacer as a `PageBlockParagraph`. *Grouping* wraps a run of 2+ consecutive top-level media blocks in ``/``; media inside a quote or an author-written group lives in `Block.children` and is therefore never re-grouped without a special case. A wrapped run also gets a **group caption**: Telegram's clients render no caption under an individual medium *inside* a group, only the container's own `PageBlockCollage.caption` (the item captions do reach the server — a read-back of a grouped article has `PageBlockPhoto.caption` populated inside the collage — it is the rendering that ignores them), so grouping would silently swallow the captions the author wrote — `_group_caption()` joins the run's non-empty `MediaRef.caption`s with `", "` and `_wrap_media_runs` emits them as a `
` line inside the tag, reported per run as `MediaGroup.caption` (for every run, including one grouped `none`, so a surface can show what the caption *would* be). `scripts/spike_rich_collage_caption.py` (2026-07-27, Saved Messages) proved the wire facts the dialect reference documents for HTML only: `
` inside a **markdown** container populates the group caption (in any of the three placements tried), a bare text line inside the tag does **not** — it leaks out as a paragraph *after* the group — and `` is silently ignored. `figcaption` is deliberately **not** in `HTML_BLOCK_TAGS` (it is one line inside its container, not a nested block) and `_consume_html` gives it **weight 0**, since the server folds it into the container's caption field rather than charging a block; counting it would over-report the budget by one per captioned group and could roll the spacer pass back over blocks Telegram never charges. Limits are **warnings, never rejections** (Telegram is the authority) with one exception: spacing that would cross 500 blocks rolls itself back — cosmetics must not break a send. `send_message` normalises once and rebinds `request = replace(request, rich_markdown=…)` before the length check, so the operation payload, `MAX_RICH_MARKDOWN_CHARS` and the backend kwarg can never disagree about what was sent, and the over-limit `ValueError` names the pass that grew it — but the **source** is bounded by the same `MAX_RICH_MARKDOWN_CHARS` *before* normalisation runs: grouping, spacing and splitting only ever grow the text, and the check runs ahead of wikilink stripping too, so it is deliberately conservative rather than exact — a source that is over the limit only because of `[[…]]` brackets that stripping would later remove is still rejected, since normalisation as a whole is a full line-by-line scan of caller input on the event loop that runs ahead of the WRITE gate; neither HTTP nor MCP bounds the field, so without that pre-check a token holder with no write grant could block the loop for seconds on a send it was never authorized to make. `spaced_paragraphs`/`line_breaks`/`media_grouping`/`media_groups` are **request-level** knobs, never backend kwargs (the legacy-signature fakes stay untouched); surfaces read the config default through `spaced_paragraphs_default(config)` / `line_breaks_default(config)` / `media_grouping_default(config)` and a surface field is `bool | None` — `None` means *not set*, since a `True` default could not tell "asked for spacing" from "sent a plain message" and would fail the "reject it without `rich_markdown`" rule on every plain send. +- **`messages/rich_markdown.py` owns every rewrite of an article, and every pass edits only the line ranges — or, for wikilinks, the in-line spans — it owns.** There is no markdown parser in the loop (the Telegram dialect diverges from CommonMark — `==mark==`, `||spoiler||`, ``, footnotes — and a parse→render round-trip would rewrite the author's text): `scan_blocks()` is a line-based scanner producing typed `Block`s, and every pass edits only the line ranges (or, for `strip_wikilinks`, the character spans within a line) it owns, so each pass returns the **input string by identity** when it changes nothing — that is what keeps CRLF and the trailing newline intact for a byte-for-byte send. Nesting is bounded by `MAX_BLOCK_NESTING` (64): past that depth `_scan_nested()` returns `()` and the quote/HTML container is emitted as a **leaf** whose `lines` still carry the whole body (so the media sweep, which skips only lines a *child* owns, still resolves references buried inside it). The scanner recurses once per level, and a single line of `"> " * 600` is 1.2 KB — far under `MAX_RICH_MARKDOWN_CHARS` — so without the bound it raises `RecursionError`; that scan runs on the event loop *ahead* of the WRITE gate, where a `RuntimeError` subclass misses every `except ValueError` on the surfaces and lands on an unmapped, empty 500 for a caller who may hold no write grant. YAML frontmatter is **not** the scanner's business: `strip_yaml_frontmatter()` is called by the CLI at the file-read boundary, next to the `utf-8-sig` BOM strip and for the same reason (that is where "this is a note file" is known), because the scanner reads a note's opening `---` as a divider and its `key: value` lines as a *setext heading* underlined by the closing `---` — an Obsidian note would otherwise open with a rule and a large heading reading its own metadata, with a spacer inserted inside it. It strips only an exact `---` first line closed by a matching `---` **whose enclosed lines read as YAML** (`_is_frontmatter_body`: a mapping entry first, then only mapping entries, `- ` items, indented continuations, comments or blank lines — a comment is accepted only *after* the first line, since its syntax is an ATX heading's and the mapping-entry-first rule is the whole fence against swallowing an article that opens with a rule and a heading; rejecting the block over one would leave the fences in and send the note's metadata as a rule plus a large heading) — the fences alone are not enough, since an article that opens with a `---` rule and uses `---` dividers later would otherwise have its whole first section silently dropped, and nothing downstream would report it (the CLI never echoes the body). It slices the remainder out of the original text (so CRLF and the trailing newline survive), returns the input by identity otherwise, and runs *before* the CLI's emptiness check so a file that is only frontmatter is reported empty. HTTP/MCP take a markdown string an agent composed rather than a note, so their input is never rewritten this way. `normalize_rich_markdown()` runs **strip → unwrap → group → space → split → count** in that order, so the reported block count and the 500-block rollback see the text with wikilinks already expanded and unsafe links demoted, the containers, *and* the paragraphs the send actually carries. *Stripping* (`strip_wikilinks`/`_expand_wikilink`) expands every Obsidian `[[target]]`/`[[target|alias]]` to plain text: one rule, no special cases — the alias wins when present, otherwise the target reads as Obsidian renders it, with a leading `#` (a link into the current note) dropped and every other `#` replaced by ` > ` (block references, `note#^blk`, fall out of that same rule unchanged, needing no branch of their own). Only the **first** `|` separates target from alias, so further pipes stay in the alias (`[[A|B|C]]` → `B|C`); an empty half falls back to the other (`[[Note|]]` → `Note`, `[[|Стас]]` → `Стас`); and a link with **both** halves empty (`[[]]`, `[[|]]`) is not a link at all — it ships verbatim, since silently deleting characters the author typed is worse than leaving a curiosity in the article. `![[…]]` embeds are untouched — the absent `!` is the whole discriminator, because that syntax is media and belongs to `scan_media`, not this pass — and so is anything inside a fenced code block (via `scan_blocks`'s own `code` blocks) or an inline code span (reusing `_CODE_SPAN_RE` rather than a second matcher); the code-span check is **containment, not overlap** (`span_start <= start and end <= span_end`), the same rule `iter_line_media_refs` uses, so a backtick merely touching a link's alias does not shield the link. It runs **first**, ahead of `scan_blocks` and every other pass, so the reported block count, the media sweep, and the spacer/split/group passes all see the text Telegram will actually receive rather than literal brackets — and a table cell whose wikilink pipe would otherwise split the row is settled before the table is even scanned. A link nested inside its own target/alias (`[[[[a]]]]`) needs more than one pass to fully resolve — the regex body excludes brackets, so one call only unwraps the innermost pair — so a single call loops up to `MAX_WIKILINK_PASSES` (8, each pass a full document rescan on the event loop ahead of the WRITE gate) times; it is idempotent up to that nesting depth (a second pass over its own output finds no `[[` left to expand, unless the source nested past the cap, in which case the leftover ships verbatim like a degenerate `[[]]`) and, like every pass here, returns the input string by identity when there was nothing to expand. There is **no knob** for it: a literal `[[…]]` reaching a Telegram reader is always a defect, never a style choice worth preserving. And unlike `strip_yaml_frontmatter` and `scan_media`, which answer "this came from a vault file" and are therefore CLI-only, this pass is **not** CLI-only — a wikilink is meaningless in Telegram no matter which surface submitted the markdown, so it runs for CLI, HTTP and MCP alike. *Unwrapping* (`unwrap_unsafe_links`, constant `UNSAFE_LINK_URL_CHARS`) rewrites `[text](url)` to `text: url` when the URL contains a character Telegram's **own** markdown parser HTML-escapes inside a link destination — `&` → `&` and `'` → `'`, proven live 2026-08-02 by `scripts/spike_rich_link_escaping.py` (Saved Messages, msg 409105, read back through `messages.getRichMessage`) — so an ordinary query-string link no longer arrives pointing at `?action=view&handbook=235`, which the target server reads as a parameter named `amp;handbook`. The same spike proved there is **no spelling of the link** that avoids it: `&` in the source comes back double-escaped, `\&` as `\&`, an autolink `` and an inline `` are escaped exactly like a plain link, `%26` survives but changes what the target server parses, and a pointy-bracket destination (`[a]()`) produces no link at all. A **bare** URL in the text is the one form that works — the parser stores no link entity for it and the clients autodetect it — which is why the demoted form is text-then-URL rather than another link syntax. Characters that survive untouched (`+`, `%20`, `#`, `~`, `|`, `_`, `*`, non-ASCII) deliberately do **not** trigger it: a working markdown link reads better than a bare URL, so only broken ones are rewritten. The anchor text is kept (`[269 - AWRA](…)` → `269 - AWRA: …`) because in a list of handbook entries it is the only thing naming the target; a link whose text is empty or already equals the URL collapses to the URL alone, and a title is dropped (Telegram renders it nowhere). Media embeds are excluded by the pattern's leading `(?`) is *also* accepted and keeps the pair in one block; the split is emitted anyway because it reads better in the clients, and the cost — the one place this module gives up idempotency — is that a split pair is indistinguishable from two author paragraphs, so re-normalising this pass's own output would let spacing push them apart (nothing in a send does that: `send_message` normalises once, from the author's source). *Spacing* inserts a U+00A0-only line as its own block between two consecutive top-level paragraphs, before any heading, and after any medium — a `media` block or the ``/`` a run was grouped into, since the container *is* the media as far as the article's vertical rhythm goes, while `
` is not — but never *before* media (an embed's lead-in line belongs with it), never after a heading, never adjacent to an existing spacer, never inside code/table/list/quote/html — U+00A0 is why `_is_blank()` is "whitespace **and** no NBSP", or the scanner would swallow the module's own spacers and double them on every re-normalisation, and why the block count is honest: Telegram charges a spacer as a `PageBlockParagraph`. *Grouping* wraps a run of 2+ consecutive top-level media blocks in ``/``; media inside a quote or an author-written group lives in `Block.children` and is therefore never re-grouped without a special case. A wrapped run also gets a **group caption**: Telegram's clients render no caption under an individual medium *inside* a group, only the container's own `PageBlockCollage.caption` (the item captions do reach the server — a read-back of a grouped article has `PageBlockPhoto.caption` populated inside the collage — it is the rendering that ignores them), so grouping would silently swallow the captions the author wrote — `_group_caption()` joins the run's non-empty `MediaRef.caption`s with `", "` and `_wrap_media_runs` emits them as a `
` line inside the tag, reported per run as `MediaGroup.caption` (for every run, including one grouped `none`, so a surface can show what the caption *would* be). `scripts/spike_rich_collage_caption.py` (2026-07-27, Saved Messages) proved the wire facts the dialect reference documents for HTML only: `
` inside a **markdown** container populates the group caption (in any of the three placements tried), a bare text line inside the tag does **not** — it leaks out as a paragraph *after* the group — and `` is silently ignored. `figcaption` is deliberately **not** in `HTML_BLOCK_TAGS` (it is one line inside its container, not a nested block) and `_consume_html` gives it **weight 0**, since the server folds it into the container's caption field rather than charging a block; counting it would over-report the budget by one per captioned group and could roll the spacer pass back over blocks Telegram never charges. Limits are **warnings, never rejections** (Telegram is the authority) with one exception: spacing that would cross 500 blocks rolls itself back — cosmetics must not break a send. `send_message` normalises once and rebinds `request = replace(request, rich_markdown=…)` before the length check, so the operation payload, `MAX_RICH_MARKDOWN_CHARS` and the backend kwarg can never disagree about what was sent, and the over-limit `ValueError` names the pass that grew it — but the **source** is bounded by the same `MAX_RICH_MARKDOWN_CHARS` *before* normalisation runs: grouping, spacing and splitting only ever grow the text, and the check runs ahead of wikilink stripping too, so it is deliberately conservative rather than exact — a source that is over the limit only because of `[[…]]` brackets that stripping would later remove is still rejected, since normalisation as a whole is a full line-by-line scan of caller input on the event loop that runs ahead of the WRITE gate; neither HTTP nor MCP bounds the field, so without that pre-check a token holder with no write grant could block the loop for seconds on a send it was never authorized to make. `spaced_paragraphs`/`line_breaks`/`media_grouping`/`media_groups` are **request-level** knobs, never backend kwargs (the legacy-signature fakes stay untouched); surfaces read the config default through `spaced_paragraphs_default(config)` / `line_breaks_default(config)` / `media_grouping_default(config)` and a surface field is `bool | None` — `None` means *not set*, since a `True` default could not tell "asked for spacing" from "sent a plain message" and would fail the "reject it without `rich_markdown`" rule on every plain send. - **Local media in an article is CLI-only, and its reference syntax is `tg://`.** `scan_media(markdown, base_dir=, vault_dir=, overrides=)` resolves each local media reference (relative/absolute path, or an Obsidian `![[file.png|caption|size]]` embed) in this order: `--rich-file` override (keyed by the target as written, its URL-decoded form, or its bare file name) → absolute path → relative to the article's directory → nearest by-name match under `vault_dir` (`os.walk`, since an Obsidian name may contain `[`/`*`; a tie raises `AmbiguousMediaError`). "Nearest" is counted in path steps from `base_dir`, so `base_dir` is `.resolve()`d up front — `Path("note.md").parent` is `Path(".")`, whose `parts` is empty, and an unresolved base would silently degrade the distance to the candidate's absolute depth and upload a file from another directory. Every media *reference* is resolved, not just every media *block*, and not just references that occupy a whole line: a media line directly followed by prose is deliberately not a block (it opens a paragraph — the common Obsidian "embed then caption line" shape), and an embed is just as at home in a bullet list, a table cell, a footnote or mid-sentence, so `_iter_media_refs()` sweeps every line of every non-`code` block with `iter_line_media_refs()` (the unanchored twin of `parse_media_line`, which stays whole-line because that is the question the *block* scanner asks — both share one pattern so they cannot drift), or the article would go out carrying a literal local path. For the same reason `_MEDIA_MD_PATTERN` accepts the two CommonMark forms a naive pattern stops short on — **balanced parentheses** in a bare destination (`![](Screenshot(1).png)`) and a **backslash-escaped quote** inside a title (`![](a.png "he said \"hi\"")`) — since a reference the pattern does not recognise at all is not resolved, not reported, and shipped verbatim; `_unescape_markdown` then undoes the escapes on the target and the title, restricted to ASCII punctuation so a Windows-style `sub\shot.png` keeps its separators. Inline **code spans** are masked out of that sweep (`_CODE_SPAN_RE`) for the same reason a fence is opaque to the block scanner: an article documenting the dialect writes `` `![](shot.png)` `` and means the text, so an unmasked sweep would either upload a file nobody asked for or hard-fail the send on a path that was never meant to exist. The mask tests **containment, not overlap**: a code span inside a reference's own caption (``![](a.png "run `make` first")``, ``![[a.png|`make` output]]``) only overlaps it, and skipping on overlap would leave that reference unresolved and ship the local path verbatim — the one silent drop `scan_media` promises never to make. The converse guard lives in the block scanner: **indented code never interrupts a paragraph** (`_interrupts_paragraph`), because an indented line right under prose is continuation text, not a code block — classifying it as `code` would make the sweep skip it and ship an indented `![[shot.png]]` as a literal local path, the one silent drop `scan_media` promises never to make. After a blank line, a heading, or a spacer it is still indented code. A `media` block counts as a paragraph for that rule — it *is* one, a paragraph whose only line is a media reference — so an embed followed by an *indented* embed does not read as media-then-code and ship the second one as a literal local path. In a block with `children` (a quote, a ``, a `
`), the lines a child **owns** — matched by document-absolute `start`/`end`, their own lines being the de-prefixed body — are covered by that child and never swept twice, but the lines it does *not* own still are: `_consume_html` puts only the body *between* the tags into `children`, so skipping the parent wholesale would silently drop a reference sitting on the opening tag line (`
![](a.png)`) — the one silent drop `scan_media` promises never to make. It rewrites each reference by **splicing at `MediaRef.span`** — the half-open range the sweep matched — inside the *document* line, so the rest of that line (a `> ` quote marker, a list bullet, table pipes, surrounding prose) is untouched. The span, not a `str.replace(ref.raw, …, 1)` search, is what makes the code-span masking actually work: an article that documents the dialect writes `` `![](shot.png)` `` and then embeds the same file for real, and a first-occurrence search would rewrite the *masked* copy and ship the real one as a literal local path. For the same reason `_iter_media_refs` sweeps the document lines rather than `Block.lines` (a quote child's lines are the de-prefixed body, so a span taken against them would land short by the prefix) and a `media` block is swept like any other line — `Block.media` comes from `parse_media_line`, which reports the *stripped* line and so carries no usable span. The rewritten form is `![alt](tg://photo?id= "caption")`, `tg://video?id=`, or `tg://audio?id=`, and returns `RichFile(id, path, caption, kind)` per file (deduplicated by resolved path). **The scheme must match the upload**: a photo named through `tg://video` fails `RICH_MESSAGE_VIDEO_INVALID`, so `media_kind()` decides the kind once from the suffix (`.gif` is an animation ⇒ document ⇒ `tg://video`; an unknown suffix raises rather than guessing, the dialect has no fourth scheme). Ids are `[A-Za-z0-9_-]+` (`make_rich_file_id`) — a dot or a non-ASCII character is `RICH_MESSAGE_FILE_ID_INVALID`, which fires *before* the URL check. Every non-`tg://` form is rejected by the server (bare id, path, `file://`, `attach://`, `` via `InputRichMessageHTML`), and an unresolvable reference or an override matching nothing is an **error naming the file**, never a silent drop. `_validate_rich_files` runs before the operation row is opened and *does* touch the filesystem (unlike the pure `_validate_attachment_refs`): the ids are already in the markdown, so a missing file would send an article pointing at nothing, and failing early leaves the idempotency key free. The backend resolves the peer **once** (`messages.uploadMedia` binds the upload to the destination), uploads in markdown order, and passes `files=` only when non-empty. Four wire facts proven by `scripts/spike_rich_media.py` (2026-07-27, Saved Messages) that Telethon's stubs do **not** document, so they are recorded here rather than re-derived: `InputRichFilePhoto(id: str, photo)` / `InputRichFileDocument(id: str, document)` take a **caller-chosen** id string, not a Telegram file id; captions survive (`PageBlockPhoto.caption` is populated for both `![](tg://photo?id=x)` and `![alt](tg://photo?id=x "cap")`, and the read-back carries a real `PageBlockPhoto(photo_id=…)` — the article embeds the media, it does not link it); a video needs **no** thumbnail (`InputMediaUploadedDocument` + `DocumentAttributeVideo` is accepted), but an `.mp3` is only reachable through `tg://audio` when it carries a `DocumentAttributeAudio`, which Telethon's `utils.get_attributes()` omits entirely without `hachoir` — hence `_document_attributes()` appending `DocumentAttributeAudio(duration=0)` for `kind == "audio"`; and a `files=` list does **not** intercept http(s) URLs, so uploaded and remote media compose freely in one article. Media attributes are **not** left to Telethon or to the server, and video/audio/animation share one path. Without a metadata library `utils.get_attributes()` returns a stub `DocumentAttributeVideo(duration=0, w=1, h=1, supports_streaming=False)` for every mp4 and no `DocumentAttributeAudio` at all; Telegram repairs the metadata by re-parsing the upload, but only for smaller files — measured live 2026-07-29 (Saved Messages, msg 407429/407430), seven videos up to 6.30 MB came back with real duration, dimensions, `thumbs=2` and `supports_streaming=True`, while three from 12.72 MB up kept `duration=0, w=1, h=1, thumbs=None` and rendered as an **empty rectangle** in the clients. The threshold is undocumented, so nothing keys off file size: `messages/media_probe.py` (no Telethon imports, plain data out) runs `ffprobe` once per non-photo file and `_document_attributes()` builds the one video/audio/animation attribute from it — replacing Telethon's, never joining it, since two `DocumentAttributeVideo` in a document is a malformed request. Videos additionally get an `ffmpeg`-generated preview frame (10% in, frame 0 of a real recording is often black) as `thumb=`; the missing preview is what makes the empty rectangle, so it is generated for every video rather than for large ones only. A **cover-art** stream (`disposition.attached_pic`) is not a video stream — an `.mp3` with artwork would otherwise be shaped like a video and take the artwork's dimensions. `ffprobe`/`ffmpeg` are **optional external binaries, not pip dependencies**: a failed probe is never an error (the pre-probe stub goes out, with a `WARNING` naming the file), because a box with no ffmpeg must still be able to send. An **animated `.gif`** turns out to have the same undocumented size threshold as the empty-rectangle case above, proven live 2026-07-29 by `scripts/spike_rich_gif.py` (Saved Messages, msg 407434-407437): a 98 KB `image/gif` upload is transcoded server-side regardless of what we attach — the read-back document came back `mime=video/mp4`, 98 KB → 21 KB, carrying Telegram's *own* `DocumentAttributeVideo`/`DocumentAttributeAnimated`/`thumbs=2`, renamed `loop.gif.mp4`, and rendered as a real `PageBlockVideo` — but a 21.2 MB `image/gif` upload is not: it kept `mime=image/gif`, had `thumbs=None` and `DocumentAttributeAnimated` dropped, and the article send then failed outright with `RICH_MESSAGE_VIDEO_INVALID`, while the same spike's ffmpeg-converted mp4 attached correctly at both sizes (22.2 MB → 9.6 MB, real `DocumentAttributeVideo`/`DocumentAttributeAnimated`, `thumbs=2`, `PageBlockVideo`). Since the threshold is undocumented and the small-gif case the server already handles is not worth a second code path, `convert_gif_to_mp4()` unconditionally uploads a converted mp4 in its place (temp file removed in a `finally`, including the failure path) and the original `.gif` name with an `.mp4` suffix is written back as the filename attribute so the temp name never reaches the article. Since conversion is what makes a `.gif` of any size reliably attachable, a `.gif` with no `ffmpeg` on PATH is rejected by `_validate_rich_files` **before the operation row is opened**, leaving the idempotency key free for the retry after the install. Every probe/convert/thumbnail call is a blocking subprocess and runs through `asyncio.to_thread`, never on the event loop. The markdown body is untouched by all of this: a `.gif` is still referenced as `tg://video?id=…` and `media_kind()` is unchanged. `RichMediaForbidden` (a `ValueError`, so it lands on 400 / exit 2 with its message rather than an empty 500) wraps both the upload loop and the send RPC for `ChatSend*ForbiddenError` — an article's media is part of its body, so there is no media-less half to retry. That match is by class name **and** by raw RPC string (`CHAT_SEND_*_FORBIDDEN` via `_MEDIA_RIGHTS_RPC_RE`, read off `exc.message`): telethon 1.44 generates no class for `CHAT_SEND_DOCS_FORBIDDEN`/`CHAT_SEND_AUDIOS_FORBIDDEN`, which arrive as a bare `ForbiddenError`, and those two cover exactly the kinds local uploads added (a video and a `.gif` are documents, `tg://audio` is an audio document) — a name-only set would leave the newest sends on the unmapped exit 1 / 500. A named error's `message` is a bare `FORBIDDEN`, so the string path cannot steal `ChatSendPlainForbiddenError`, which is about text rather than media. `MediaCaptionTooLongError` is deliberately **not** in that set: it is a plain `ValueError` naming the caption, since "this chat forbids media" would send the operator to check admin rights over a fixable caption. Both rely on `messages send` catching `ValueError` → exit 2 (the same handler every other `messages` command has), so a domain rejection never reads as the internal-error exit 1. HTTP/MCP never call `scan_media`, so a remote caller cannot name a server-side path. diff --git a/README.md b/README.md index c9c47c0..c3273b1 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,8 @@ Member references in `members`/`admins` (group create) and in `members bulk-add` - **Dialect** (parsed server-side): `#`…`######` headings, tables with alignment, task lists, `>` quotes, fenced code with a language, `---` dividers, `~~strike~~`, `==marked==`, `||spoiler||`, footnotes, math, `
`, and media as a standalone block — by public HTTPS URL (`![](https://…jpg "caption")`, fetched by the server) or, **on the CLI only**, by local file (uploaded and referenced as `tg://photo?id=…` / `tg://video?id=…` / `tg://audio?id=…`). Headings, lists, tables, quotes, code, dividers, URL media and uploaded local media are verified over MTProto; the remaining constructs are documented for the Bot API twin of the same server feature and are unverified here. - **Wikilinks** (always on, no knob): Obsidian `[[target]]` / `[[target|alias]]` links are expanded to plain text before anything else runs — the alias wins when present, otherwise the target reads as Obsidian renders it (a leading `#`, a link into the current note, drops; every other `#` becomes ` > `; block references like `note#^blk` fall out of that same rule unchanged). Only the first `|` separates target from alias, so further pipes stay in the alias (`[[A|B|C]]` → `B|C`); an empty half falls back to the other (`[[Note|]]` → `Note`, `[[|Стас]]` → `Стас`); a link with both halves empty (`[[]]`, `[[|]]`) is not a link and ships verbatim rather than collapsing to nothing. `![[…]]` embeds are untouched (that syntax is media, resolved as described below) and so is anything inside inline code or a fenced code block. Unlike frontmatter stripping and local-media resolution, this is **not** CLI-only — a literal `[[…]]` reaching a Telegram reader is a defect whichever surface sent it, so it runs on the CLI, HTTP, and MCP alike. A link nested inside its own target/alias (`[[[[a]]]]`) expands too, up to a bounded nesting depth — an unrealistic depth beyond that ships the remainder verbatim rather than rescanning the document without limit. `--dry-run` reports the count as `rich_markdown_wikilinks`. -- **Paragraph spacing** (on by default): the server renders neighbouring paragraphs tight against each other, so the markdown is rewritten before sending — a U+00A0-only spacer paragraph is inserted between two consecutive paragraphs, before every heading, and after every medium (a photo/video/audio block, or the ``/`` a run was grouped into), but never *before* media, never after a heading, never inside code/tables/lists/quotes/HTML blocks, and never next to a spacer the author already wrote. Turn it off with `--no-spaced-paragraphs` (HTTP/MCP: `spaced_paragraphs: false`) to keep the author's own spacing; the default also comes from `telegram.defaults.rich_markdown_spaced_paragraphs`. Note that this switches off *only* the spacer pass — media grouping and local-media rewriting are independent, so a truly byte-for-byte send also needs `--no-line-breaks`, `telegram.defaults.rich_markdown_grouping: none` (or `--media-group =none` per run), an article with no local media, and no `[[wikilinks]]`, which are always expanded (no knob). Spacers count toward both limits below: if spacing would push the article past 500 blocks it is sent unspaced with the warning `spaced_paragraphs disabled: N blocks would exceed the 500-block limit`. +- **Links with `&` in the URL** (always on, no knob): Telegram's own markdown parser HTML-escapes `&` (→ `&`) and `'` (→ `'`) inside a link destination, so `[Справочник](https://example.com/?action=view&handbook=235)` arrives pointing at `…&handbook=235` — a URL the target server reads as a parameter named `amp;handbook`. No spelling of the link avoids it (`&` in the source is escaped twice, `\&` becomes `\&`, autolinks and inline `` are escaped the same way), and the one form that survives is a **bare URL in the text** — the parser stores no link for it and the clients autodetect it. Such links are therefore rewritten to `text: url` (`[269 - AWRA](https://…&key=269)` → `269 - AWRA: https://…&key=269`); an empty anchor text, or one identical to the URL, collapses to the URL alone, and a link title is dropped (Telegram renders it nowhere). Links whose URL carries none of those characters are left as markdown links — `+`, `%20`, `#`, `~`, `|`, `_`, `*` and non-ASCII all survive the parser intact. `![…](…)` media is never demoted, nor is anything inside inline code or a fenced code block. Like wikilinks this runs on the CLI, HTTP and MCP alike — the parser doing the damage is Telegram's. `--dry-run` reports the count as `rich_markdown_unwrapped_links`. +- **Paragraph spacing** (on by default): the server renders neighbouring paragraphs tight against each other, so the markdown is rewritten before sending — a U+00A0-only spacer paragraph is inserted between two consecutive paragraphs, before every heading, and after every medium (a photo/video/audio block, or the ``/`` a run was grouped into), but never *before* media, never after a heading, never inside code/tables/lists/quotes/HTML blocks, and never next to a spacer the author already wrote. Turn it off with `--no-spaced-paragraphs` (HTTP/MCP: `spaced_paragraphs: false`) to keep the author's own spacing; the default also comes from `telegram.defaults.rich_markdown_spaced_paragraphs`. Note that this switches off *only* the spacer pass — media grouping and local-media rewriting are independent, so a truly byte-for-byte send also needs `--no-line-breaks`, `telegram.defaults.rich_markdown_grouping: none` (or `--media-group =none` per run), an article with no local media, no `[[wikilinks]]` (always expanded, no knob) and no link whose URL contains `&` or `'` (always demoted to a bare URL, no knob). Spacers count toward both limits below: if spacing would push the article past 500 blocks it is sent unspaced with the warning `spaced_paragraphs disabled: N blocks would exceed the 500-block limit`. - **Line breaks** (on by default): Telegram parses the markdown itself and, like CommonMark, folds a *single* newline inside a paragraph into a space — so an Obsidian note's ``` Фотоальбом - https://… @@ -147,7 +148,7 @@ Member references in `members`/`admins` (group create) and in `members bulk-add` - **Obsidian frontmatter** (**CLI-only**): a leading `---` … `---` YAML block is dropped when the file is read, next to the BOM strip and for the same reason — this dialect has no notion of frontmatter, so the article would otherwise open with a divider and a large heading reading `tags: [...] date: ...`. Only an exact `---` on the first line starts a block, only a matching `---` ends one, and the lines between them must read as YAML (a `key: value` entry first, then only entries, `- ` items, indented continuations or blank lines), so a note that merely begins with a horizontal rule keeps it — even when a later `---` divider would otherwise close the pair; a file that is *nothing but* frontmatter is reported as empty. HTTP/MCP take a markdown string an agent composed rather than a note file, so their input is passed through untouched. - **Limits**: 1..32 768 characters (validated locally after normalization, inclusive), ~500 blocks and 50 media attachments (counted locally and reported as warnings — Telegram is the authority); the server additionally caps nesting and table columns and reports its own errors (`RICH_MESSAGE_MARKDOWN_INVALID`, `RICH_MESSAGE_TEXT_TOO_LONG`, …). - **Exclusivity**: `--rich-markdown` is a targeted-send-only alternative to the message body — it cannot be combined with `--text`, `--file`, `--file-url` (HTTP/MCP: `text`, `file_urls`, `base64_files`) or with mass mode. `--spaced-paragraphs`/`--no-spaced-paragraphs`, `--line-breaks`/`--no-line-breaks`, `--rich-file`, `--vault-dir` and `--media-group` are errors without `--rich-markdown` (CLI exit 2; HTTP `spaced_paragraphs`/`line_breaks` without `rich_markdown` is a `422`). -- Everything else is unchanged: entity resolution, the WRITE gate, `--operation-id` idempotency, topic/reply targeting (`--topic-id`/`--reply-to`), and scheduling (`--schedule-at`/`--delay`) all work. `--dry-run` reports the article as markers (`rich_markdown`, post-normalization `rich_markdown_chars`, `rich_markdown_blocks`, `rich_markdown_media`, `rich_markdown_wikilinks`, `rich_markdown_file`, `spaced_paragraphs`, `spaced`, `line_breaks`, `media_grouping`, `rich_markdown_groups`, `rich_files`) rather than echoing a 32k body — the listed files are never read. Normalization warnings go to stderr as `warning: ...` on a real send and ride the result JSON as `warnings`. +- Everything else is unchanged: entity resolution, the WRITE gate, `--operation-id` idempotency, topic/reply targeting (`--topic-id`/`--reply-to`), and scheduling (`--schedule-at`/`--delay`) all work. `--dry-run` reports the article as markers (`rich_markdown`, post-normalization `rich_markdown_chars`, `rich_markdown_blocks`, `rich_markdown_media`, `rich_markdown_wikilinks`, `rich_markdown_unwrapped_links`, `rich_markdown_file`, `spaced_paragraphs`, `spaced`, `line_breaks`, `media_grouping`, `rich_markdown_groups`, `rich_files`) rather than echoing a 32k body — the listed files are never read. Normalization warnings go to stderr as `warning: ...` on a real send and ride the result JSON as `warnings`. - A chat that forbids media rejects the **whole** article: Telegram's `ChatSendMediaForbiddenError` (and its per-type siblings) becomes a `RichMediaForbidden` error naming the chat (HTTP `400`, CLI exit 2). There is no media-less fallback — an article's media is part of its body. - If Telegram accepts the request but the response carries no readable message id, the send is **not** marked failed — the operation goes to `needs_review` (the queue never auto-retries it), because the article may well have been delivered. Check the chat before `operations retry`; a blind re-send under a fresh key would duplicate it. - A failed rich send is **not** silently retried as plain text — it surfaces as the normal send error, and the caller decides. Requires `telethon >= 1.44` (layer 227), now the project's minimum pin; if an older Telethon is force-installed anyway, only the rich send fails, with an explicit version error (HTTP `500 {"error": "rich_message_unsupported"}`, CLI exit 1, MCP error message) — and the idempotency key is left free, so the same `--operation-id` sends normally once Telethon is upgraded. diff --git a/scripts/spike_rich_link_escaping.py b/scripts/spike_rich_link_escaping.py new file mode 100644 index 0000000..e40b025 --- /dev/null +++ b/scripts/spike_rich_link_escaping.py @@ -0,0 +1,309 @@ +#!/usr/bin/env python3 +"""Spike: keep an ``&`` inside a link URL intact in a rich article. + +A note written in Obsidian carries ordinary query-string links:: + + [Конкурсы и ассоциации (235)](https://example.com/?action=handbooklist&handbook=235) + +Sent as an article the link arrives **broken**: the ``&`` comes back +HTML-escaped as ``&``, so the target server sees a parameter named +``amp;handbook`` instead of ``handbook``. Nothing on our side rewrites it — +``normalize_rich_markdown`` leaves the text byte-for-byte — so the escaping +happens inside Telegram's own server-side markdown parser. + +This spike asks which spelling of ``&`` survives that parser. It sends **one** +article to Saved Messages carrying one paragraph per candidate, reads the +article back through ``messages.getRichMessage`` (the message's own +``rich_message`` is a truncated ``part=True`` preview), and prints the +``TextUrl.url`` the server actually stored for each one. A candidate is a +**pass** when the stored URL equals the URL the note meant. + +It is a *spike*, not part of the shipped surface: it talks to the real account +and shares ``scripts/spike_rich_media.py``'s precondition of an authorized +Telethon session. + +Usage:: + + .venv/bin/python scripts/spike_rich_link_escaping.py + .venv/bin/python scripts/spike_rich_link_escaping.py --dry-run + .venv/bin/python scripts/spike_rich_link_escaping.py --entity me + +Exit codes: 0 = the article was accepted (or dry run), 2 = precondition +missing (no session, no config, Telethon too old), 3 = the server rejected the +send. +""" + +from __future__ import annotations + +import argparse +import asyncio +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +REPO_ROOT = Path(__file__).resolve().parent.parent +SRC = REPO_ROOT / "src" +if str(SRC) not in sys.path: # allow running without an editable install + sys.path.insert(0, str(SRC)) + +DEFAULT_ENTITY = "me" + +# The shape that breaks: two query parameters joined by a bare ``&``. +WANTED = "https://example.com/?action=handbookdataview&handbook=235&key=269" + + +@dataclass(frozen=True) +class Candidate: + """One markdown spelling of a URL carrying a bare ``&``.""" + + name: str + syntax: str + markdown: str + # The URL the note meant. A candidate passes when the server stored this. + wanted: str = WANTED + + +def build_candidates() -> tuple[Candidate, ...]: + escaped = WANTED.replace("&", "&") + backslashed = WANTED.replace("&", "\\&") + percent = WANTED.replace("&", "%26") + return ( + Candidate( + name="plain", + syntax="bare & in a markdown link (what a note writes today)", + markdown=f"[plain]({WANTED})", + ), + Candidate( + name="entity", + syntax="& entity in the destination", + markdown=f"[entity]({escaped})", + ), + Candidate( + name="backslash", + syntax="backslash-escaped \\& in the destination", + markdown=f"[backslash]({backslashed})", + ), + Candidate( + name="angle-dest", + syntax="CommonMark pointy-bracket destination <...>", + markdown=f"[angle-dest](<{WANTED}>)", + ), + Candidate( + name="autolink", + syntax="CommonMark autolink ", + markdown=f"<{WANTED}>", + ), + Candidate( + name="bare-url", + syntax="bare URL, no link syntax at all (autodetected)", + markdown=WANTED, + ), + Candidate( + name="html-a", + syntax="inline inside the markdown dialect", + markdown=f'html-a', + ), + Candidate( + name="percent", + syntax="%26 instead of & (changes the target's parsing)", + markdown=f"[percent]({percent})", + ), + ) + + +def build_char_candidates() -> tuple[Candidate, ...]: + """One link per character that the parser might mangle inside a URL. + + ``&`` is the known-broken baseline; the rest are the characters an Obsidian + note plausibly carries in a query string, plus the two markdown emphasis + markers, which a naive destination scanner could also eat. + """ + + probes = ( + ("amp", "&"), + ("lt", "<"), + ("gt", ">"), + ("dquote", '"'), + ("squote", "'"), + ("plus", "+"), + ("percent20", "%20"), + ("hash", "#"), + ("tilde", "~"), + ("pipe", "|"), + ("underscore", "_"), + ("asterisk", "*"), + ("cyrillic", "тест"), + ) + return tuple( + Candidate( + name=name, + syntax=f"URL containing {char!r}", + markdown=f"[{name}](https://example.com/?q=a{char}b)", + wanted=f"https://example.com/?q=a{char}b", + ) + for name, char in probes + ) + + +def build_article(candidates: tuple[Candidate, ...]) -> str: + lines = ["# Link escaping spike", ""] + for candidate in candidates: + lines.append(f"{candidate.name}: {candidate.markdown}") + lines.append("") + return "\n".join(lines) + + +def _fail(message: str, code: int = 2) -> int: + print(f"FAIL: {message}", file=sys.stderr) + return code + + +def _iter_urls(node: Any) -> list[tuple[str, str]]: + """Collect every ``(anchor text, url)`` pair in a ``RichText`` tree.""" + + if node is None or isinstance(node, str): + return [] + found: list[tuple[str, str]] = [] + url = getattr(node, "url", None) + if url is not None: + found.append((_flatten(getattr(node, "text", None)), url)) + for part in list(getattr(node, "texts", None) or []): + found.extend(_iter_urls(part)) + inner = getattr(node, "text", None) + if inner is not None and not isinstance(inner, str): + found.extend(_iter_urls(inner)) + return found + + +def _flatten(node: Any) -> str: + if node is None: + return "" + if isinstance(node, str): + return node + parts = getattr(node, "texts", None) + if parts: + return "".join(_flatten(part) for part in parts) + text = getattr(node, "text", None) + if text is not None: + return _flatten(text) + return "" + + +def _report(rich: Any, candidates: tuple[Candidate, ...]) -> None: + by_name = {candidate.name: candidate for candidate in candidates} + print(f"\nread-back: part={getattr(rich, 'part', None)!r}") + seen: dict[str, list[str]] = {} + for block in list(getattr(rich, "blocks", None) or []): + text = _flatten(getattr(block, "text", None)) + for anchor, url in _iter_urls(getattr(block, "text", None)): + name = text.split(":", 1)[0].strip() + seen.setdefault(name if name in by_name else anchor, []).append(url) + + print() + for candidate in candidates: + urls = seen.get(candidate.name) or [] + if not urls: + print(f" [{candidate.name:<10}] NO LINK ({candidate.syntax})") + continue + for url in urls: + verdict = "PASS" if url == candidate.wanted else "FAIL" + suffix = "" if verdict == "PASS" else f" (wanted {candidate.wanted})" + print(f" [{candidate.name:<10}] {verdict} {url}{suffix}") + + +async def _run(args: argparse.Namespace) -> int: + try: + from telethon.tl import functions, types + except ImportError as exc: # pragma: no cover - spike script + return _fail(f"Telethon is not importable: {exc}") + if getattr(types, "InputRichMessageMarkdown", None) is None: + return _fail("This Telethon build has no InputRichMessageMarkdown (layer < 227).") + if "rich_message" not in functions.messages.SendMessageRequest.__init__.__annotations__: + return _fail("SendMessageRequest has no rich_message parameter. Install telethon>=1.44.") + + candidates = build_char_candidates() if args.mode == "chars" else build_candidates() + article = build_article(candidates) + + if args.dry_run: + print(article) + print("\ndry-run: not sending") + return 0 + + from telegram_assistant.config.loader import ConfigError, load_config + from telegram_assistant.entities.service import CachingEntityResolver, EntityError + from telegram_assistant.entities.telethon_backend import TelethonResolverBackend + from telegram_assistant.telegram_client.session import TelethonSessionManager + + try: + config = load_config(args.config) + except ConfigError as exc: + return _fail(str(exc)) + + manager = TelethonSessionManager(config.telegram) + if not manager.session_path_exists(): + return _fail( + f"No Telethon session at {manager.session_path}. Run `telegram-assistant auth` first." + ) + + client = await manager.get_client() + try: + if not await client.is_user_authorized(): + return _fail(f"Session {manager.session_path} exists but is not authorized.") + resolver = CachingEntityResolver(TelethonResolverBackend(client)) + try: + resolved = await resolver.resolve(args.entity) + except EntityError as exc: + return _fail(f"Could not resolve {args.entity!r}: {exc}") + print(f"resolved: chat_id={resolved.chat_id} title={resolved.title!r}") + + peer = await client.get_input_entity(resolved.chat_id) + request = functions.messages.SendMessageRequest( + peer=peer, + message="", + rich_message=types.InputRichMessageMarkdown(markdown=article), + ) + try: + result = await client(request) + except Exception as exc: # noqa: BLE001 - the taxonomy is the finding + print(f"REJECTED: {type(exc).__name__}: {exc}") + return 3 + + from telegram_assistant.messages.telethon_backend import _extract_rich_message_id + + message_id = _extract_rich_message_id( + result, random_id=getattr(request, "random_id", None) + ) + print(f"ACCEPTED: message_id={message_id}") + if message_id is None: + print("(no readable message id — cannot read the article back)") + return 0 + + full = await client(functions.messages.GetRichMessageRequest(peer=peer, id=message_id)) + message = list(getattr(full, "messages", None) or [None])[0] + rich = getattr(message, "rich_message", None) + if rich is None: + print("(read-back carried no rich_message)") + return 0 + _report(rich, candidates) + return 0 + finally: + await manager.disconnect() + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--entity", default=DEFAULT_ENTITY, help="target chat reference") + parser.add_argument("--config", default=None, help="path to config.yml") + parser.add_argument( + "--mode", + choices=("escaping", "chars"), + default="escaping", + help="escaping: spellings of & in one URL; chars: one URL per suspect character", + ) + parser.add_argument("--dry-run", action="store_true", help="print the article, send nothing") + return asyncio.run(_run(parser.parse_args())) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/telegram-assistant/SKILL.md b/skills/telegram-assistant/SKILL.md index 1ebe35f..67f99a1 100644 --- a/skills/telegram-assistant/SKILL.md +++ b/skills/telegram-assistant/SKILL.md @@ -575,7 +575,8 @@ command rather than after `folder_cache_ttl` seconds. send time, and any reply target. For a rich send it echoes the article as markers only — `rich_markdown: true`, `rich_markdown_chars` (post-normalization), `rich_markdown_blocks`, `rich_markdown_media`, - `rich_markdown_wikilinks`, `rich_markdown_file`, `spaced_paragraphs` + `rich_markdown_wikilinks`, `rich_markdown_unwrapped_links`, + `rich_markdown_file`, `spaced_paragraphs` (the effective decision), `spaced` (what the pass actually did), `line_breaks`, `media_grouping`, `rich_markdown_groups` and `rich_files` — never the body; show the human those, plus the file @@ -607,6 +608,18 @@ command rather than after `folder_cache_ttl` seconds. (`[[[[a]]]]`) expands too, up to a bounded depth — an unrealistic note past that ships the remainder verbatim. The dry-run reports the count as `rich_markdown_wikilinks`. +- Links with `&` in the URL (**always on, no flag**): Telegram's own + parser HTML-escapes `&` (→ `&`) and `'` (→ `'`) inside a link + destination, so `[Справочник](https://…/?action=view&handbook=235)` + would arrive pointing at `…&handbook=235` — a broken link. No + spelling of the link avoids it, and a bare URL in the text is the one + form that survives, so such links are rewritten to `text: url` + (`[269 - AWRA](https://…&key=269)` → `269 - AWRA: https://…&key=269`). + Links whose URL has none of those characters stay markdown links; + `![…](…)` media and anything inside code are never touched. Runs on + every surface. The dry-run reports the count as + `rich_markdown_unwrapped_links`. If the human asks why a link in their + note lost its title, this is why — say so and do not "fix" it back. - Paragraph spacing (**on by default**): the server renders neighbouring paragraphs tight against each other, so the CLI/HTTP/MCP insert a U+00A0-only spacer paragraph between two paragraphs and before every @@ -615,8 +628,9 @@ command rather than after `folder_cache_ttl` seconds. e.g. because they hand-tuned the spacing. It switches off the spacer pass only — media grouping and local-media rewriting are independent, so mention `--media-group =none` if they want the source truly - byte-for-byte, and note wikilinks are always expanded regardless (no - knob) — a `[[…]]`-bearing article can never go byte-for-byte. The flag is an error + byte-for-byte, and note wikilinks are always expanded and links with `&`/`'` in the URL + always demoted regardless (no knob) — an article carrying either can + never go byte-for-byte. The flag is an error (exit 2 / 422) without `--rich-markdown`. The default also comes from `telegram.defaults.rich_markdown_spaced_paragraphs`. Spacers count toward both the character and the block limit; if spacing would push @@ -1512,6 +1526,7 @@ Request: «Опубликуй в чате Клиент / проект стать 4. Show the resolved chat id and the article markers from the dry-run JSON (`rich_markdown: true`, `rich_markdown_chars`, `rich_markdown_blocks`, `rich_markdown_media`, `rich_markdown_wikilinks`, +`rich_markdown_unwrapped_links`, `rich_markdown_file`, `spaced_paragraphs`, `spaced`, `line_breaks`, `media_grouping`, `rich_markdown_groups`, plus `rich_files` when the article carries local media) — the body is deliberately not echoed, so diff --git a/src/telegram_assistant/cli/rich_send.py b/src/telegram_assistant/cli/rich_send.py index 8b7be38..161f367 100644 --- a/src/telegram_assistant/cli/rich_send.py +++ b/src/telegram_assistant/cli/rich_send.py @@ -289,6 +289,9 @@ def rich_dry_run_markers( "rich_markdown_wikilinks": ( normalization.wikilinks if normalization is not None else None ), + "rich_markdown_unwrapped_links": ( + normalization.unwrapped_links if normalization is not None else None + ), "rich_markdown_file": (str(rich_markdown) if is_rich else None), # Local media the real send would upload. The files are listed, # never read — a dry run touches no bytes and no network. diff --git a/src/telegram_assistant/messages/rich_markdown.py b/src/telegram_assistant/messages/rich_markdown.py index f61e58f..8a5a48d 100644 --- a/src/telegram_assistant/messages/rich_markdown.py +++ b/src/telegram_assistant/messages/rich_markdown.py @@ -73,6 +73,22 @@ #: exactly like a degenerate ``[[]]``. MAX_WIKILINK_PASSES = 8 +#: Characters Telegram's own markdown parser HTML-escapes inside a link +#: destination, turning a working URL into a broken one. Proven live +#: 2026-08-02 (Saved Messages, ``scripts/spike_rich_link_escaping.py``, read +#: back through ``messages.getRichMessage``): ``&`` is stored as ``&`` and +#: ``'`` as ``'``, so ``?action=view&handbook=235`` reaches the target +#: server as a parameter named ``amp;handbook``. Everything else a query +#: string plausibly carries — ``+``, ``%20``, ``#``, ``~``, ``|``, ``_``, +#: ``*``, non-ASCII — survives untouched. No spelling of the link escapes it: +#: ``&`` in the source comes back double-escaped (``&amp;``), ``\&`` +#: as ``\&``, and an autolink or an inline ```` is escaped +#: exactly like a plain markdown link. A **bare** URL in the text is the one +#: form that survives, because the parser stores no link entity for it at all +#: and the clients autodetect it — which is what :func:`unwrap_unsafe_links` +#: rewrites to. +UNSAFE_LINK_URL_CHARS = ("&", "'") + #: HTML container tags the dialect defines; their contents are scanned as #: nested blocks so grouping can tell author-written groups from runs it may #: wrap itself. @@ -190,6 +206,22 @@ #: down to prose before anything could upload it. _WIKILINK_RE = re.compile(r"(?[^\[\]]*)\]\]") +#: An inline markdown link, ``[text](destination "title")``. The destination +#: half is :data:`_MEDIA_MD_PATTERN`'s verbatim — the two dialects are the same +#: one, so they cannot drift — and the leading ``(?[^\[\]]*)\]\(\s* + (?:<(?P[^>]*)> + |(?P(?:[^()\s\\]|\\.|\((?:[^()\s\\]|\\.)*\))+)) + (?:\s+(?:"(?P(?:[^"\\]|\\.)*)" + |'(?P(?:[^'\\]|\\.)*)' + |\((?P(?:[^()\\]|\\.)*)\)))? + \s*\)""", + re.VERBOSE, +) + #: A CommonMark backslash escape: a backslash before ASCII punctuation. A #: backslash before anything else is a literal backslash, which is what keeps a #: Windows-style ``C:\Users\me\a.png`` target intact. @@ -303,6 +335,9 @@ class RichMarkdownNormalization: #: flags above this is a count, because it is what a surface reports to the #: operator — there is no knob to explain, only a number. wikilinks: int = 0 + #: How many links :func:`unwrap_unsafe_links` demoted to a bare URL. A count + #: for the same reason ``wikilinks`` is one. + unwrapped_links: int = 0 class MediaResolutionError(ValueError): @@ -523,6 +558,108 @@ def _strip_line_wikilinks(line: str) -> tuple[str, int]: return "".join(out), count +def _unwrap_link(text: str, url: str) -> str: + """Return the plain-text form of one link: its text, then the bare URL. + + The anchor text is kept because it is usually the only thing that names + what the link points at (``[269 - AWRA](…)`` in a list of handbook + entries), and dropping it would leave a column of indistinguishable URLs. + A link whose text *is* the URL, or has no text at all, collapses to the + URL alone rather than repeating it. + + A title (``[a](url "tip")``) is dropped: Telegram renders it nowhere, so + carrying it into the text would show the reader a tooltip that never + existed. + """ + + text = text.strip() + if not text or text == url: + return url + return f"{text}: {url}" + + +def _unwrap_line_links(line: str) -> tuple[str, int]: + """Demote every unsafe-URL link on one line, leaving inline code spans alone.""" + + code_spans = [match.span() for match in _CODE_SPAN_RE.finditer(line)] + out: list[str] = [] + cursor = 0 + count = 0 + for match in _LINK_MD_RE.finditer(line): + start, end = match.span() + # Containment, not overlap — the rule _strip_line_wikilinks and + # iter_line_media_refs both use. A backtick inside the anchor text + # merely overlaps the link and must not shield it, or the broken URL + # ships. + if any(span_start <= start and end <= span_end for span_start, span_end in code_spans): + continue + url = match.group("angle") + if url is None: + url = match.group("plain") or "" + if not any(char in url for char in UNSAFE_LINK_URL_CHARS): + continue + out.append(line[cursor:start]) + out.append(_unwrap_link(match.group("text"), url)) + cursor = end + count += 1 + if not count: + return line, 0 + out.append(line[cursor:]) + return "".join(out), count + + +def unwrap_unsafe_links(markdown: str) -> tuple[str, int]: + """Demote links whose URL Telegram would mangle to text + bare URL; report how many. + + Telegram's server-side markdown parser HTML-escapes + :data:`UNSAFE_LINK_URL_CHARS` inside a link destination, so an ordinary + query-string link arrives pointing at ``?action=view&handbook=235`` — + a URL the target server reads as a parameter called ``amp;handbook``. No + spelling of the link avoids it (see :data:`UNSAFE_LINK_URL_CHARS`); only a + bare URL in the text survives, because the parser stores no link entity for + it and the clients autodetect it instead. + + So ``[269 - AWRA](https://x/?a=1&k=2)`` becomes + ``269 - AWRA: https://x/?a=1&k=2``. Links whose URL carries none of those + characters are left as markdown links — they work, and they read better. + Like :func:`strip_wikilinks` this has **no knob**: a link that opens the + wrong page is always a defect, and it is one on every surface, since the + parser is Telegram's rather than the vault's. + + The rewrite can only shrink the text (``[`` ``]`` ``(`` ``)`` become + ``: ``, and a title is dropped), so unlike grouping/spacing/splitting it + can never push an article over + :data:`~telegram_assistant.messages.service.MAX_RICH_MARKDOWN_CHARS`. + + Media embeds are untouched — ``![alt](a.png)`` is a file for + :func:`scan_media`, not a link — and so is anything inside a fenced code + block or an inline code span, for the reason an article documenting this + dialect needs: it writes the markdown and means the characters. + + Returns ``(markdown, 0)`` by identity when nothing changed, which is what + keeps CRLF and the trailing newline intact for a byte-for-byte send. + """ + + if "](" not in markdown or not any(char in markdown for char in UNSAFE_LINK_URL_CHARS): + return markdown, 0 + + code_lines = _code_line_indices(scan_blocks(markdown)) + lines = split_lines(markdown) + out: list[str] = [] + total = 0 + for index, line in enumerate(lines): + if index in code_lines or "](" not in line: + out.append(line) + continue + rewritten, count = _unwrap_line_links(line) + out.append(rewritten) + total += count + if not total: + return markdown, 0 + text = "\n".join(out) + return (text + "\n" if markdown.endswith(("\n", "\r")) else text), total + + def _code_line_indices(blocks: tuple[Block, ...] | list[Block]) -> set[int]: """Document line indices covered by a code block, nesting included.""" @@ -863,6 +1000,10 @@ def normalize_rich_markdown( # the text that will actually be sent. It also settles a table cell whose # wikilink pipe would otherwise split it. markdown, wikilinks = strip_wikilinks(markdown) + # Second, for the same reason and on the same footing: another in-line + # rewrite whose output every later pass must see. It runs *after* wikilink + # expansion because that pass can put a URL where a `[[…]]` stood. + markdown, unwrapped_links = unwrap_unsafe_links(markdown) blocks = scan_blocks(markdown) warnings: list[str] = [] @@ -930,6 +1071,7 @@ def _split(text: str, text_blocks: tuple[Block, ...]) -> str: spacers_added=spacers_added, lines_split=unspaced is not grouped, wikilinks=wikilinks, + unwrapped_links=unwrapped_links, ) diff --git a/tests/test_mcp_tools.py b/tests/test_mcp_tools.py index f671b40..0aedb4c 100644 --- a/tests/test_mcp_tools.py +++ b/tests/test_mcp_tools.py @@ -741,6 +741,32 @@ def test_mcp_send_rich_markdown_expands_wikilinks( assert "[[" not in sent +def test_mcp_send_rich_markdown_demotes_links_telegram_would_mangle( + minimal_config_yaml: str, tmp_path: Path +) -> None: + """Same proof for the MCP tool surface — the mangling is Telegram's, not the vault's.""" + backend = FakeRichMessageBackend() + with _client(minimal_config_yaml, tmp_path, message_backend=backend) as client: + token = _mint_token(client) + _initialize(client, token) + + result = _call_tool( + client, + token, + "telegram_messages_send", + { + "telegram_chat_id": -100123, + "rich_markdown": "[269 - AWRA](https://example.com/?action=view&key=269)\n", + "operation_id": "mcp-rich-link", + }, + ) + + assert result["isError"] is False, result + sent = backend.sent[0]["rich_markdown"] + assert "269 - AWRA: https://example.com/?action=view&key=269" in sent + assert "](" not in sent + + def test_mcp_send_rich_markdown_never_resolves_a_local_media_path( minimal_config_yaml: str, tmp_path: Path ) -> None: diff --git a/tests/test_messages_rich_spacing_flag.py b/tests/test_messages_rich_spacing_flag.py index 3fc893e..b5286a2 100644 --- a/tests/test_messages_rich_spacing_flag.py +++ b/tests/test_messages_rich_spacing_flag.py @@ -395,6 +395,35 @@ def test_cli_dry_run_reports_expanded_wikilinks( assert backend.sent == [] +def test_cli_dry_run_reports_unwrapped_links( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + config_file, md_file, backend = _cli_setup( + tmp_path, + monkeypatch, + markdown="[269 - AWRA](https://example.com/?action=view&key=269)\n", + ) + + result = _run_cli( + [ + "messages", + "send", + "--chat-id", + "-100", + "--rich-markdown", + str(md_file), + "--dry-run", + "--config", + str(config_file), + ] + ) + + assert result.exit_code == 0, _cli_output(result) + resolved = json.loads(result.stdout.strip().splitlines()[-1])["resolved"] + assert resolved["rich_markdown_unwrapped_links"] == 1 + assert backend.sent == [] + + def test_cli_dry_run_plain_send_has_no_wikilink_marker( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -418,6 +447,7 @@ def test_cli_dry_run_plain_send_has_no_wikilink_marker( assert result.exit_code == 0, _cli_output(result) resolved = json.loads(result.stdout.strip().splitlines()[-1])["resolved"] assert resolved["rich_markdown_wikilinks"] is None + assert resolved["rich_markdown_unwrapped_links"] is None def test_cli_dry_run_reports_block_limit_warnings( diff --git a/tests/test_messages_rich_surfaces.py b/tests/test_messages_rich_surfaces.py index b71c52b..1bbe0b5 100644 --- a/tests/test_messages_rich_surfaces.py +++ b/tests/test_messages_rich_surfaces.py @@ -172,6 +172,25 @@ def test_http_rich_send_expands_wikilinks() -> None: assert "[[" not in sent +def test_http_rich_send_demotes_links_telegram_would_mangle() -> None: + """Also not CLI-only: the parser that breaks the URL is Telegram's.""" + backend = RecordingMessageBackend() + client = _client(backend) + resp = client.post( + "/telegram/messages", + json={ + "telegram_chat_id": -100, + "rich_markdown": "[269 - AWRA](https://example.com/?action=view&key=269)\n", + "operation_id": "rich-http-link", + }, + headers=AUTH, + ) + assert resp.status_code == 200, resp.text + sent = backend.sent[0]["rich_markdown"] + assert "269 - AWRA: https://example.com/?action=view&key=269" in sent + assert "](" not in sent + + def test_http_rich_send_allows_topic_reply_and_schedule() -> None: backend = RecordingMessageBackend() client = _client(backend) diff --git a/tests/test_rich_markdown_links.py b/tests/test_rich_markdown_links.py new file mode 100644 index 0000000..3bc2ca7 --- /dev/null +++ b/tests/test_rich_markdown_links.py @@ -0,0 +1,193 @@ +"""Demoting links whose URL Telegram's markdown parser would mangle. + +Telegram HTML-escapes ``&`` and ``'`` inside a link destination, so an +ordinary query-string link arrives pointing at ``?a=1&k=2``. Only a bare +URL in the text survives (proven live — see ``UNSAFE_LINK_URL_CHARS``), so +``unwrap_unsafe_links`` rewrites ``[text](url)`` to ``text: url`` for exactly +those links and leaves every other link a markdown link. +""" + +from __future__ import annotations + +import pytest + +from telegram_assistant.messages.rich_markdown import ( + UNSAFE_LINK_URL_CHARS, + normalize_rich_markdown, + unwrap_unsafe_links, +) + +UNSAFE = "https://example.com/?action=view&handbook=235" +SAFE = "https://example.com/plain/page" + + +def test_link_with_ampersand_is_demoted_to_text_and_bare_url() -> None: + text, count = unwrap_unsafe_links(f"[269 - AWRA]({UNSAFE})") + assert text == f"269 - AWRA: {UNSAFE}" + assert count == 1 + + +def test_link_with_apostrophe_is_demoted() -> None: + url = "https://example.com/?q=o'brien" + text, count = unwrap_unsafe_links(f"[name]({url})") + assert text == f"name: {url}" + assert count == 1 + + +def test_safe_link_is_returned_by_identity() -> None: + source = f"See [the page]({SAFE}) for details.\n" + text, count = unwrap_unsafe_links(source) + assert text is source + assert count == 0 + + +def test_document_without_unsafe_characters_is_returned_by_identity() -> None: + source = "Just [a link](https://example.com/a) and prose.\r\n" + text, count = unwrap_unsafe_links(source) + assert text is source + assert count == 0 + + +def test_only_the_unsafe_link_is_rewritten() -> None: + source = f"[safe]({SAFE}) and [unsafe]({UNSAFE})" + text, count = unwrap_unsafe_links(source) + assert text == f"[safe]({SAFE}) and unsafe: {UNSAFE}" + assert count == 1 + + +def test_empty_anchor_text_collapses_to_the_url() -> None: + text, count = unwrap_unsafe_links(f"[]({UNSAFE})") + assert text == UNSAFE + assert count == 1 + + +def test_anchor_text_equal_to_the_url_is_not_repeated() -> None: + text, count = unwrap_unsafe_links(f"[{UNSAFE}]({UNSAFE})") + assert text == UNSAFE + assert count == 1 + + +def test_title_is_dropped() -> None: + text, count = unwrap_unsafe_links(f'[name]({UNSAFE} "a tooltip")') + assert text == f"name: {UNSAFE}" + assert count == 1 + + +def test_angle_bracket_destination_is_unwrapped_without_its_brackets() -> None: + text, count = unwrap_unsafe_links(f"[name](<{UNSAFE}>)") + assert text == f"name: {UNSAFE}" + assert count == 1 + + +def test_media_embed_is_never_demoted() -> None: + source = "![alt](https://example.com/a.png?w=1&h=2)" + text, count = unwrap_unsafe_links(source) + assert text is source + assert count == 0 + + +def test_obsidian_media_embed_is_untouched() -> None: + source = "![[shot.png|a & b]]" + text, count = unwrap_unsafe_links(source) + assert text is source + assert count == 0 + + +def test_inline_code_span_containing_a_link_is_untouched() -> None: + source = f"Write `[name]({UNSAFE})` to link." + text, count = unwrap_unsafe_links(source) + assert text is source + assert count == 0 + + +def test_code_span_inside_the_anchor_text_does_not_shield_the_link() -> None: + # Containment, not overlap: the span only overlaps the link, so skipping + # would ship the broken URL — the defect this pass exists to remove. + text, count = unwrap_unsafe_links(f"[run `make` first]({UNSAFE})") + assert text == f"run `make` first: {UNSAFE}" + assert count == 1 + + +def test_fenced_code_block_is_untouched() -> None: + source = f"```\n[name]({UNSAFE})\n```\n" + text, count = unwrap_unsafe_links(source) + assert text is source + assert count == 0 + + +def test_links_inside_lists_quotes_and_tables_are_rewritten_in_place() -> None: + source = ( + f"- item [a]({UNSAFE}) tail\n" + f"> quoted [b]({UNSAFE}) tail\n" + f"| cell [c]({UNSAFE}) | second |\n" + ) + text, count = unwrap_unsafe_links(source) + assert count == 3 + assert text == ( + f"- item a: {UNSAFE} tail\n" + f"> quoted b: {UNSAFE} tail\n" + f"| cell c: {UNSAFE} | second |\n" + ) + + +def test_two_links_on_one_line_are_both_rewritten() -> None: + text, count = unwrap_unsafe_links(f"[a]({UNSAFE}), [b]({UNSAFE})") + assert text == f"a: {UNSAFE}, b: {UNSAFE}" + assert count == 2 + + +def test_trailing_newline_and_crlf_survive_a_rewrite() -> None: + text, count = unwrap_unsafe_links(f"[a]({UNSAFE})\r\nsecond line\r\n") + assert count == 1 + assert text == f"a: {UNSAFE}\nsecond line\n" + + +def test_rewrite_never_grows_the_text() -> None: + source = f'[a very long anchor]({UNSAFE} "and a title")\n' + text, _ = unwrap_unsafe_links(source) + assert len(text) <= len(source) + + +def test_pass_is_idempotent() -> None: + once, first = unwrap_unsafe_links(f"[a]({UNSAFE})") + twice, second = unwrap_unsafe_links(once) + assert twice is once + assert first == 1 + assert second == 0 + + +@pytest.mark.parametrize("char", UNSAFE_LINK_URL_CHARS) +def test_every_unsafe_character_triggers_the_rewrite(char: str) -> None: + url = f"https://example.com/?q=a{char}b" + _, count = unwrap_unsafe_links(f"[name]({url})") + assert count == 1 + + +@pytest.mark.parametrize("char", ["+", "#", "~", "|", "_", "*", "%20", "тест"]) +def test_characters_telegram_preserves_do_not_trigger_the_rewrite(char: str) -> None: + source = f"[name](https://example.com/?q=a{char}b)" + text, count = unwrap_unsafe_links(source) + assert text is source + assert count == 0 + + +def test_normalize_reports_the_count_and_rewrites_the_article() -> None: + result = normalize_rich_markdown(f"# Title\n\n[269 - AWRA]({UNSAFE})\n") + assert result.unwrapped_links == 1 + assert f"269 - AWRA: {UNSAFE}" in result.markdown + assert "](" not in result.markdown + + +def test_normalize_reports_zero_for_an_article_without_unsafe_links() -> None: + result = normalize_rich_markdown(f"Just [a link]({SAFE}).\n") + assert result.unwrapped_links == 0 + assert f"[a link]({SAFE})" in result.markdown + + +def test_wikilink_expansion_runs_before_this_pass() -> None: + # A wikilink alias can carry the anchor text of a link, so the wikilink + # pass must have finished before the link pattern sees the line. + result = normalize_rich_markdown(f"[[Note|Alias]] and [x]({UNSAFE})\n") + assert result.wikilinks == 1 + assert result.unwrapped_links == 1 + assert f"Alias and x: {UNSAFE}" in result.markdown