diff --git a/CHANGELOG.md b/CHANGELOG.md index 61fd403..aa55bc7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ ### Features +- rich-markdown: Expand wikilinks on every surface +- rich-markdown: Expand Obsidian wikilinks to plain text - rich-markdown: Log every rich-media upload - rich-markdown: Reject a GIF up front when ffmpeg is missing - rich-markdown: Upload an animated GIF as a converted mp4 @@ -16,6 +18,7 @@ ### Bug Fixes +- rich-markdown: Fixpoint-expand nested wikilinks, name the pass in grew_by - rich-markdown: Correct gif-attach docstrings, log converted gif's original name - spike: Make spike_rich_gif.py exit 3 when every candidate is rejected - access: Keep a skipped rule's restrictive session-only override @@ -23,6 +26,10 @@ ### Documentation +- rich-markdown: Report and document wikilink expansion +- plan: Implementation plan for Obsidian wikilink stripping +- spec: Design Obsidian wikilink stripping for rich markdown +- readme: Note the Docker image ships without ffmpeg - rich-markdown: Record the ffprobe-backed media attribute path - rich-markdown: Implementation plan for the ffprobe-backed media path diff --git a/CLAUDE.md b/CLAUDE.md index 951be2f..caebd63 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -58,7 +58,7 @@ Six cross-cutting behaviours in `messages/` that surfaces must not re-implement: - **Rich sends bypass the high-level client.** `SendMessageRequest.rich_markdown` (CLI `--rich-markdown `, HTTP/MCP `rich_markdown` string) sends a Telegram *article* — headings, tables, quotes, code, media by public HTTPS URL or (CLI only) by uploaded local file — parsed **server-side**, so nothing here builds a PageBlock tree and the 4096-char split never applies. `send_message` validates it (non-empty, `<= MAX_RICH_MARKDOWN_CHARS` = 32 768 inclusive, mutually exclusive with `text`/attachments) and passes it down through the **only-when-set** `extra` kwargs block, so backends predating rich sends keep working — that contract is pinned by a legacy-signature fake on every surface (`LegacySendBackend`, `CliLegacyMessageBackend`, `test_mcp_plain_send_omits_rich_markdown_kwarg`) rather than by asserting a recorded `None`. Telethon's `client.send_message` has no `rich_message` parameter, so `TelethonMessageBackend._send_rich_message` issues a raw `functions.messages.SendMessageRequest(message="", rich_message=InputRichMessageMarkdown(...))` and reads the id back from the `UpdateMessageID` whose `random_id` matches the request's own — an `Updates` container may carry another request's updates, and a *keyed* update naming a different `random_id` is never used as a fallback: its id would be returned as the article's and recorded in the `SentMessageRegistry`, handing the session edit/delete rights over a message it never sent (Telethon's own sender keys strictly on `random_id` for the same reason). An unkeyed entry, or any entry at all when the request's own `random_id` is unreadable, is still accepted. From there it falls back to `UpdateNewMessage`/`UpdateNewChannelMessage`/`UpdateNewScheduledMessage` — **minus** the ids named by those rejected foreign-keyed `UpdateMessageID`s, since Telegram pairs each `UpdateMessageID` with the `UpdateNew*Message` for the *same* message and an unfiltered scan would hand back the very id the `random_id` check just refused (only that id is excluded: another `UpdateNew*Message` may still be ours when our own `UpdateMessageID` is missing) — and finally to a bare `UpdateShortSentMessage`/`UpdateShort` envelope. That last read is gated on the type name being `UpdateShortSentMessage`: `UpdateShortMessage`/`UpdateShortChatMessage` are *incoming*-message envelopes carrying someone else's `id` and expose neither `.updates` nor `.update`, so an ungated `getattr(result, "id")` would claim it (Telethon's own extractor returns nothing for them). `InputRichMessageMarkdown` arrived in **telethon 1.44 (layer 227)**, so it is imported through a version-tolerant probe (`_import_rich_markdown_type`, the `topics/telethon_backend.py` pattern) — an older Telethon fails only the rich send, with a version message, not the whole import. Mass mode has no rich path at all (`MassSendRequest` has no such field), so that exclusivity lives on the surfaces (HTTP `MessageSendBody._shape`, the CLI's `--mass` check), and there is deliberately **no** silent fallback to a plain-text send: a server-rejected article surfaces through the normal send-error path and the caller decides. Two rich-specific failures deliberately stay **out** of the `MessageSendFailed` class, because that one means "a previous attempt with this idempotency key failed" and every surface renders it as `previous_attempt_failed` (HTTP/MCP 409, CLI exit 2): an old Telethon raises `RichMessageUnsupported` (a plain `RuntimeError`, so it lands on 500 / exit 1 and names the version, not a phantom prior attempt — the HTTP route maps it explicitly to a 500 whose body carries `{"error": "rich_message_unsupported", "message": ...}`, because an unmapped `RuntimeError` would surface as Starlette's *empty* 500 and drop the version hint, and `send_message` **drops the just-opened operation row** rather than failing it: the check runs before any RPC, so the retry after the upgrade — same `operation_id` — must send instead of replaying `previous_attempt_failed` or a stuck `pending`), and an envelope with no readable id raises `MessageSendUnconfirmed`, which `send_message` catches next to `FloodWaitError` and marks **`needs_review`** — the request reached Telegram, so the article may be delivered, and a terminal `failed` would tell the caller nothing happened and invite a duplicate re-send. -- **`messages/rich_markdown.py` owns every rewrite of an article, and it only ever inserts or wraps lines.** There is no markdown parser in the loop (the Telegram dialect diverges from CommonMark — `==mark==`, `||spoiler||`, ``, footnotes — and a parse→render round-trip would rewrite the author's text): `scan_blocks()` is a line-based scanner producing typed `Block`s, and every pass edits only the line ranges it owns, so each pass returns the **input string by identity** when it changes nothing — that is what keeps CRLF and the trailing newline intact for a byte-for-byte send. Nesting is bounded by `MAX_BLOCK_NESTING` (64): past that depth `_scan_nested()` returns `()` and the quote/HTML container is emitted as a **leaf** whose `lines` still carry the whole body (so the media sweep, which skips only lines a *child* owns, still resolves references buried inside it). The scanner recurses once per level, and a single line of `"> " * 600` is 1.2 KB — far under `MAX_RICH_MARKDOWN_CHARS` — so without the bound it raises `RecursionError`; that scan runs on the event loop *ahead* of the WRITE gate, where a `RuntimeError` subclass misses every `except ValueError` on the surfaces and lands on an unmapped, empty 500 for a caller who may hold no write grant. YAML frontmatter is **not** the scanner's business: `strip_yaml_frontmatter()` is called by the CLI at the file-read boundary, next to the `utf-8-sig` BOM strip and for the same reason (that is where "this is a note file" is known), because the scanner reads a note's opening `---` as a divider and its `key: value` lines as a *setext heading* underlined by the closing `---` — an Obsidian note would otherwise open with a rule and a large heading reading its own metadata, with a spacer inserted inside it. It strips only an exact `---` first line closed by a matching `---` **whose enclosed lines read as YAML** (`_is_frontmatter_body`: a mapping entry first, then only mapping entries, `- ` items, indented continuations, comments or blank lines — a comment is accepted only *after* the first line, since its syntax is an ATX heading's and the mapping-entry-first rule is the whole fence against swallowing an article that opens with a rule and a heading; rejecting the block over one would leave the fences in and send the note's metadata as a rule plus a large heading) — the fences alone are not enough, since an article that opens with a `---` rule and uses `---` dividers later would otherwise have its whole first section silently dropped, and nothing downstream would report it (the CLI never echoes the body). It slices the remainder out of the original text (so CRLF and the trailing newline survive), returns the input by identity otherwise, and runs *before* the CLI's emptiness check so a file that is only frontmatter is reported empty. HTTP/MCP take a markdown string an agent composed rather than a note, so their input is never rewritten this way. `normalize_rich_markdown()` runs **group → space → split → count** in that order, so the reported block count and the 500-block rollback see the containers *and* the paragraphs the send actually carries. *Splitting* (`_split_paragraph_lines`, knob `line_breaks`) inserts a blank line between the lines of a top-level paragraph, because Telegram's parser folds a single newline into a space and an Obsidian note's two lines under one another would arrive as one run-on line; the resulting paragraphs render tight, which is the whole point, so it runs **after** spacing — the spacer pass must never see the pairs it produces. `scripts/spike_rich_line_breaks.py` (2026-07-27, Saved Messages) proved that a real in-paragraph hard break (two trailing spaces, a trailing `\`, `
`) is *also* accepted and keeps the pair in one block; the split is emitted anyway because it reads better in the clients, and the cost — the one place this module gives up idempotency — is that a split pair is indistinguishable from two author paragraphs, so re-normalising this pass's own output would let spacing push them apart (nothing in a send does that: `send_message` normalises once, from the author's source). *Spacing* inserts a U+00A0-only line as its own block between two consecutive top-level paragraphs, before any heading, and after any medium — a `media` block or the ``/`` a run was grouped into, since the container *is* the media as far as the article's vertical rhythm goes, while `
` is not — but never *before* media (an embed's lead-in line belongs with it), never after a heading, never adjacent to an existing spacer, never inside code/table/list/quote/html — U+00A0 is why `_is_blank()` is "whitespace **and** no NBSP", or the scanner would swallow the module's own spacers and double them on every re-normalisation, and why the block count is honest: Telegram charges a spacer as a `PageBlockParagraph`. *Grouping* wraps a run of 2+ consecutive top-level media blocks in ``/``; media inside a quote or an author-written group lives in `Block.children` and is therefore never re-grouped without a special case. A wrapped run also gets a **group caption**: Telegram's clients render no caption under an individual medium *inside* a group, only the container's own `PageBlockCollage.caption` (the item captions do reach the server — a read-back of a grouped article has `PageBlockPhoto.caption` populated inside the collage — it is the rendering that ignores them), so grouping would silently swallow the captions the author wrote — `_group_caption()` joins the run's non-empty `MediaRef.caption`s with `", "` and `_wrap_media_runs` emits them as a `
` line inside the tag, reported per run as `MediaGroup.caption` (for every run, including one grouped `none`, so a surface can show what the caption *would* be). `scripts/spike_rich_collage_caption.py` (2026-07-27, Saved Messages) proved the wire facts the dialect reference documents for HTML only: `
` inside a **markdown** container populates the group caption (in any of the three placements tried), a bare text line inside the tag does **not** — it leaks out as a paragraph *after* the group — and `` is silently ignored. `figcaption` is deliberately **not** in `HTML_BLOCK_TAGS` (it is one line inside its container, not a nested block) and `_consume_html` gives it **weight 0**, since the server folds it into the container's caption field rather than charging a block; counting it would over-report the budget by one per captioned group and could roll the spacer pass back over blocks Telegram never charges. Limits are **warnings, never rejections** (Telegram is the authority) with one exception: spacing that would cross 500 blocks rolls itself back — cosmetics must not break a send. `send_message` normalises once and rebinds `request = replace(request, rich_markdown=…)` before the length check, so the operation payload, `MAX_RICH_MARKDOWN_CHARS` and the backend kwarg can never disagree about what was sent, and the over-limit `ValueError` names the pass that grew it — but the **source** is bounded by the same `MAX_RICH_MARKDOWN_CHARS` *before* normalisation runs, since both passes only ever grow the text (so an over-limit source can never come back under it) and normalisation is a full line-by-line scan of caller input on the event loop that runs ahead of the WRITE gate; neither HTTP nor MCP bounds the field, so without that pre-check a token holder with no write grant could block the loop for seconds on a send it was never authorized to make. `spaced_paragraphs`/`line_breaks`/`media_grouping`/`media_groups` are **request-level** knobs, never backend kwargs (the legacy-signature fakes stay untouched); surfaces read the config default through `spaced_paragraphs_default(config)` / `line_breaks_default(config)` / `media_grouping_default(config)` and a surface field is `bool | None` — `None` means *not set*, since a `True` default could not tell "asked for spacing" from "sent a plain message" and would fail the "reject it without `rich_markdown`" rule on every plain send. +- **`messages/rich_markdown.py` owns every rewrite of an article, and every pass edits only the line ranges — or, for wikilinks, the in-line spans — it owns.** There is no markdown parser in the loop (the Telegram dialect diverges from CommonMark — `==mark==`, `||spoiler||`, ``, footnotes — and a parse→render round-trip would rewrite the author's text): `scan_blocks()` is a line-based scanner producing typed `Block`s, and every pass edits only the line ranges (or, for `strip_wikilinks`, the character spans within a line) it owns, so each pass returns the **input string by identity** when it changes nothing — that is what keeps CRLF and the trailing newline intact for a byte-for-byte send. Nesting is bounded by `MAX_BLOCK_NESTING` (64): past that depth `_scan_nested()` returns `()` and the quote/HTML container is emitted as a **leaf** whose `lines` still carry the whole body (so the media sweep, which skips only lines a *child* owns, still resolves references buried inside it). The scanner recurses once per level, and a single line of `"> " * 600` is 1.2 KB — far under `MAX_RICH_MARKDOWN_CHARS` — so without the bound it raises `RecursionError`; that scan runs on the event loop *ahead* of the WRITE gate, where a `RuntimeError` subclass misses every `except ValueError` on the surfaces and lands on an unmapped, empty 500 for a caller who may hold no write grant. YAML frontmatter is **not** the scanner's business: `strip_yaml_frontmatter()` is called by the CLI at the file-read boundary, next to the `utf-8-sig` BOM strip and for the same reason (that is where "this is a note file" is known), because the scanner reads a note's opening `---` as a divider and its `key: value` lines as a *setext heading* underlined by the closing `---` — an Obsidian note would otherwise open with a rule and a large heading reading its own metadata, with a spacer inserted inside it. It strips only an exact `---` first line closed by a matching `---` **whose enclosed lines read as YAML** (`_is_frontmatter_body`: a mapping entry first, then only mapping entries, `- ` items, indented continuations, comments or blank lines — a comment is accepted only *after* the first line, since its syntax is an ATX heading's and the mapping-entry-first rule is the whole fence against swallowing an article that opens with a rule and a heading; rejecting the block over one would leave the fences in and send the note's metadata as a rule plus a large heading) — the fences alone are not enough, since an article that opens with a `---` rule and uses `---` dividers later would otherwise have its whole first section silently dropped, and nothing downstream would report it (the CLI never echoes the body). It slices the remainder out of the original text (so CRLF and the trailing newline survive), returns the input by identity otherwise, and runs *before* the CLI's emptiness check so a file that is only frontmatter is reported empty. HTTP/MCP take a markdown string an agent composed rather than a note, so their input is never rewritten this way. `normalize_rich_markdown()` runs **strip → group → space → split → count** in that order, so the reported block count and the 500-block rollback see the text with wikilinks already expanded, the containers, *and* the paragraphs the send actually carries. *Stripping* (`strip_wikilinks`/`_expand_wikilink`) expands every Obsidian `[[target]]`/`[[target|alias]]` to plain text: one rule, no special cases — the alias wins when present, otherwise the target reads as Obsidian renders it, with a leading `#` (a link into the current note) dropped and every other `#` replaced by ` > ` (block references, `note#^blk`, fall out of that same rule unchanged, needing no branch of their own). Only the **first** `|` separates target from alias, so further pipes stay in the alias (`[[A|B|C]]` → `B|C`); an empty half falls back to the other (`[[Note|]]` → `Note`, `[[|Стас]]` → `Стас`); and a link with **both** halves empty (`[[]]`, `[[|]]`) is not a link at all — it ships verbatim, since silently deleting characters the author typed is worse than leaving a curiosity in the article. `![[…]]` embeds are untouched — the absent `!` is the whole discriminator, because that syntax is media and belongs to `scan_media`, not this pass — and so is anything inside a fenced code block (via `scan_blocks`'s own `code` blocks) or an inline code span (reusing `_CODE_SPAN_RE` rather than a second matcher); the code-span check is **containment, not overlap** (`span_start <= start and end <= span_end`), the same rule `iter_line_media_refs` uses, so a backtick merely touching a link's alias does not shield the link. It runs **first**, ahead of `scan_blocks` and every other pass, so the reported block count, the media sweep, and the spacer/split/group passes all see the text Telegram will actually receive rather than literal brackets — and a table cell whose wikilink pipe would otherwise split the row is settled before the table is even scanned. A link nested inside its own target/alias (`[[[[a]]]]`) needs more than one pass to fully resolve — the regex body excludes brackets, so one call only unwraps the innermost pair — so a single call loops up to `MAX_WIKILINK_PASSES` (8, each pass a full document rescan on the event loop ahead of the WRITE gate) times; it is idempotent up to that nesting depth (a second pass over its own output finds no `[[` left to expand, unless the source nested past the cap, in which case the leftover ships verbatim like a degenerate `[[]]`) and, like every pass here, returns the input string by identity when there was nothing to expand. There is **no knob** for it: a literal `[[…]]` reaching a Telegram reader is always a defect, never a style choice worth preserving. And unlike `strip_yaml_frontmatter` and `scan_media`, which answer "this came from a vault file" and are therefore CLI-only, this pass is **not** CLI-only — a wikilink is meaningless in Telegram no matter which surface submitted the markdown, so it runs for CLI, HTTP and MCP alike. *Splitting* (`_split_paragraph_lines`, knob `line_breaks`) inserts a blank line between the lines of a top-level paragraph, because Telegram's parser folds a single newline into a space and an Obsidian note's two lines under one another would arrive as one run-on line; the resulting paragraphs render tight, which is the whole point, so it runs **after** spacing — the spacer pass must never see the pairs it produces. `scripts/spike_rich_line_breaks.py` (2026-07-27, Saved Messages) proved that a real in-paragraph hard break (two trailing spaces, a trailing `\`, `
`) is *also* accepted and keeps the pair in one block; the split is emitted anyway because it reads better in the clients, and the cost — the one place this module gives up idempotency — is that a split pair is indistinguishable from two author paragraphs, so re-normalising this pass's own output would let spacing push them apart (nothing in a send does that: `send_message` normalises once, from the author's source). *Spacing* inserts a U+00A0-only line as its own block between two consecutive top-level paragraphs, before any heading, and after any medium — a `media` block or the ``/`` a run was grouped into, since the container *is* the media as far as the article's vertical rhythm goes, while `
` is not — but never *before* media (an embed's lead-in line belongs with it), never after a heading, never adjacent to an existing spacer, never inside code/table/list/quote/html — U+00A0 is why `_is_blank()` is "whitespace **and** no NBSP", or the scanner would swallow the module's own spacers and double them on every re-normalisation, and why the block count is honest: Telegram charges a spacer as a `PageBlockParagraph`. *Grouping* wraps a run of 2+ consecutive top-level media blocks in ``/``; media inside a quote or an author-written group lives in `Block.children` and is therefore never re-grouped without a special case. A wrapped run also gets a **group caption**: Telegram's clients render no caption under an individual medium *inside* a group, only the container's own `PageBlockCollage.caption` (the item captions do reach the server — a read-back of a grouped article has `PageBlockPhoto.caption` populated inside the collage — it is the rendering that ignores them), so grouping would silently swallow the captions the author wrote — `_group_caption()` joins the run's non-empty `MediaRef.caption`s with `", "` and `_wrap_media_runs` emits them as a `
` line inside the tag, reported per run as `MediaGroup.caption` (for every run, including one grouped `none`, so a surface can show what the caption *would* be). `scripts/spike_rich_collage_caption.py` (2026-07-27, Saved Messages) proved the wire facts the dialect reference documents for HTML only: `
` inside a **markdown** container populates the group caption (in any of the three placements tried), a bare text line inside the tag does **not** — it leaks out as a paragraph *after* the group — and `` is silently ignored. `figcaption` is deliberately **not** in `HTML_BLOCK_TAGS` (it is one line inside its container, not a nested block) and `_consume_html` gives it **weight 0**, since the server folds it into the container's caption field rather than charging a block; counting it would over-report the budget by one per captioned group and could roll the spacer pass back over blocks Telegram never charges. Limits are **warnings, never rejections** (Telegram is the authority) with one exception: spacing that would cross 500 blocks rolls itself back — cosmetics must not break a send. `send_message` normalises once and rebinds `request = replace(request, rich_markdown=…)` before the length check, so the operation payload, `MAX_RICH_MARKDOWN_CHARS` and the backend kwarg can never disagree about what was sent, and the over-limit `ValueError` names the pass that grew it — but the **source** is bounded by the same `MAX_RICH_MARKDOWN_CHARS` *before* normalisation runs: grouping, spacing and splitting only ever grow the text, and the check runs ahead of wikilink stripping too, so it is deliberately conservative rather than exact — a source that is over the limit only because of `[[…]]` brackets that stripping would later remove is still rejected, since normalisation as a whole is a full line-by-line scan of caller input on the event loop that runs ahead of the WRITE gate; neither HTTP nor MCP bounds the field, so without that pre-check a token holder with no write grant could block the loop for seconds on a send it was never authorized to make. `spaced_paragraphs`/`line_breaks`/`media_grouping`/`media_groups` are **request-level** knobs, never backend kwargs (the legacy-signature fakes stay untouched); surfaces read the config default through `spaced_paragraphs_default(config)` / `line_breaks_default(config)` / `media_grouping_default(config)` and a surface field is `bool | None` — `None` means *not set*, since a `True` default could not tell "asked for spacing" from "sent a plain message" and would fail the "reject it without `rich_markdown`" rule on every plain send. - **Local media in an article is CLI-only, and its reference syntax is `tg://`.** `scan_media(markdown, base_dir=, vault_dir=, overrides=)` resolves each local media reference (relative/absolute path, or an Obsidian `![[file.png|caption|size]]` embed) in this order: `--rich-file` override (keyed by the target as written, its URL-decoded form, or its bare file name) → absolute path → relative to the article's directory → nearest by-name match under `vault_dir` (`os.walk`, since an Obsidian name may contain `[`/`*`; a tie raises `AmbiguousMediaError`). "Nearest" is counted in path steps from `base_dir`, so `base_dir` is `.resolve()`d up front — `Path("note.md").parent` is `Path(".")`, whose `parts` is empty, and an unresolved base would silently degrade the distance to the candidate's absolute depth and upload a file from another directory. Every media *reference* is resolved, not just every media *block*, and not just references that occupy a whole line: a media line directly followed by prose is deliberately not a block (it opens a paragraph — the common Obsidian "embed then caption line" shape), and an embed is just as at home in a bullet list, a table cell, a footnote or mid-sentence, so `_iter_media_refs()` sweeps every line of every non-`code` block with `iter_line_media_refs()` (the unanchored twin of `parse_media_line`, which stays whole-line because that is the question the *block* scanner asks — both share one pattern so they cannot drift), or the article would go out carrying a literal local path. For the same reason `_MEDIA_MD_PATTERN` accepts the two CommonMark forms a naive pattern stops short on — **balanced parentheses** in a bare destination (`![](Screenshot(1).png)`) and a **backslash-escaped quote** inside a title (`![](a.png "he said \"hi\"")`) — since a reference the pattern does not recognise at all is not resolved, not reported, and shipped verbatim; `_unescape_markdown` then undoes the escapes on the target and the title, restricted to ASCII punctuation so a Windows-style `sub\shot.png` keeps its separators. Inline **code spans** are masked out of that sweep (`_CODE_SPAN_RE`) for the same reason a fence is opaque to the block scanner: an article documenting the dialect writes `` `![](shot.png)` `` and means the text, so an unmasked sweep would either upload a file nobody asked for or hard-fail the send on a path that was never meant to exist. The mask tests **containment, not overlap**: a code span inside a reference's own caption (``![](a.png "run `make` first")``, ``![[a.png|`make` output]]``) only overlaps it, and skipping on overlap would leave that reference unresolved and ship the local path verbatim — the one silent drop `scan_media` promises never to make. The converse guard lives in the block scanner: **indented code never interrupts a paragraph** (`_interrupts_paragraph`), because an indented line right under prose is continuation text, not a code block — classifying it as `code` would make the sweep skip it and ship an indented `![[shot.png]]` as a literal local path, the one silent drop `scan_media` promises never to make. After a blank line, a heading, or a spacer it is still indented code. A `media` block counts as a paragraph for that rule — it *is* one, a paragraph whose only line is a media reference — so an embed followed by an *indented* embed does not read as media-then-code and ship the second one as a literal local path. In a block with `children` (a quote, a ``, a `
`), the lines a child **owns** — matched by document-absolute `start`/`end`, their own lines being the de-prefixed body — are covered by that child and never swept twice, but the lines it does *not* own still are: `_consume_html` puts only the body *between* the tags into `children`, so skipping the parent wholesale would silently drop a reference sitting on the opening tag line (`
![](a.png)`) — the one silent drop `scan_media` promises never to make. It rewrites each reference by **splicing at `MediaRef.span`** — the half-open range the sweep matched — inside the *document* line, so the rest of that line (a `> ` quote marker, a list bullet, table pipes, surrounding prose) is untouched. The span, not a `str.replace(ref.raw, …, 1)` search, is what makes the code-span masking actually work: an article that documents the dialect writes `` `![](shot.png)` `` and then embeds the same file for real, and a first-occurrence search would rewrite the *masked* copy and ship the real one as a literal local path. For the same reason `_iter_media_refs` sweeps the document lines rather than `Block.lines` (a quote child's lines are the de-prefixed body, so a span taken against them would land short by the prefix) and a `media` block is swept like any other line — `Block.media` comes from `parse_media_line`, which reports the *stripped* line and so carries no usable span. The rewritten form is `![alt](tg://photo?id= "caption")`, `tg://video?id=`, or `tg://audio?id=`, and returns `RichFile(id, path, caption, kind)` per file (deduplicated by resolved path). **The scheme must match the upload**: a photo named through `tg://video` fails `RICH_MESSAGE_VIDEO_INVALID`, so `media_kind()` decides the kind once from the suffix (`.gif` is an animation ⇒ document ⇒ `tg://video`; an unknown suffix raises rather than guessing, the dialect has no fourth scheme). Ids are `[A-Za-z0-9_-]+` (`make_rich_file_id`) — a dot or a non-ASCII character is `RICH_MESSAGE_FILE_ID_INVALID`, which fires *before* the URL check. Every non-`tg://` form is rejected by the server (bare id, path, `file://`, `attach://`, `` via `InputRichMessageHTML`), and an unresolvable reference or an override matching nothing is an **error naming the file**, never a silent drop. `_validate_rich_files` runs before the operation row is opened and *does* touch the filesystem (unlike the pure `_validate_attachment_refs`): the ids are already in the markdown, so a missing file would send an article pointing at nothing, and failing early leaves the idempotency key free. The backend resolves the peer **once** (`messages.uploadMedia` binds the upload to the destination), uploads in markdown order, and passes `files=` only when non-empty. Four wire facts proven by `scripts/spike_rich_media.py` (2026-07-27, Saved Messages) that Telethon's stubs do **not** document, so they are recorded here rather than re-derived: `InputRichFilePhoto(id: str, photo)` / `InputRichFileDocument(id: str, document)` take a **caller-chosen** id string, not a Telegram file id; captions survive (`PageBlockPhoto.caption` is populated for both `![](tg://photo?id=x)` and `![alt](tg://photo?id=x "cap")`, and the read-back carries a real `PageBlockPhoto(photo_id=…)` — the article embeds the media, it does not link it); a video needs **no** thumbnail (`InputMediaUploadedDocument` + `DocumentAttributeVideo` is accepted), but an `.mp3` is only reachable through `tg://audio` when it carries a `DocumentAttributeAudio`, which Telethon's `utils.get_attributes()` omits entirely without `hachoir` — hence `_document_attributes()` appending `DocumentAttributeAudio(duration=0)` for `kind == "audio"`; and a `files=` list does **not** intercept http(s) URLs, so uploaded and remote media compose freely in one article. Media attributes are **not** left to Telethon or to the server, and video/audio/animation share one path. Without a metadata library `utils.get_attributes()` returns a stub `DocumentAttributeVideo(duration=0, w=1, h=1, supports_streaming=False)` for every mp4 and no `DocumentAttributeAudio` at all; Telegram repairs the metadata by re-parsing the upload, but only for smaller files — measured live 2026-07-29 (Saved Messages, msg 407429/407430), seven videos up to 6.30 MB came back with real duration, dimensions, `thumbs=2` and `supports_streaming=True`, while three from 12.72 MB up kept `duration=0, w=1, h=1, thumbs=None` and rendered as an **empty rectangle** in the clients. The threshold is undocumented, so nothing keys off file size: `messages/media_probe.py` (no Telethon imports, plain data out) runs `ffprobe` once per non-photo file and `_document_attributes()` builds the one video/audio/animation attribute from it — replacing Telethon's, never joining it, since two `DocumentAttributeVideo` in a document is a malformed request. Videos additionally get an `ffmpeg`-generated preview frame (10% in, frame 0 of a real recording is often black) as `thumb=`; the missing preview is what makes the empty rectangle, so it is generated for every video rather than for large ones only. A **cover-art** stream (`disposition.attached_pic`) is not a video stream — an `.mp3` with artwork would otherwise be shaped like a video and take the artwork's dimensions. `ffprobe`/`ffmpeg` are **optional external binaries, not pip dependencies**: a failed probe is never an error (the pre-probe stub goes out, with a `WARNING` naming the file), because a box with no ffmpeg must still be able to send. An **animated `.gif`** turns out to have the same undocumented size threshold as the empty-rectangle case above, proven live 2026-07-29 by `scripts/spike_rich_gif.py` (Saved Messages, msg 407434-407437): a 98 KB `image/gif` upload is transcoded server-side regardless of what we attach — the read-back document came back `mime=video/mp4`, 98 KB → 21 KB, carrying Telegram's *own* `DocumentAttributeVideo`/`DocumentAttributeAnimated`/`thumbs=2`, renamed `loop.gif.mp4`, and rendered as a real `PageBlockVideo` — but a 21.2 MB `image/gif` upload is not: it kept `mime=image/gif`, had `thumbs=None` and `DocumentAttributeAnimated` dropped, and the article send then failed outright with `RICH_MESSAGE_VIDEO_INVALID`, while the same spike's ffmpeg-converted mp4 attached correctly at both sizes (22.2 MB → 9.6 MB, real `DocumentAttributeVideo`/`DocumentAttributeAnimated`, `thumbs=2`, `PageBlockVideo`). Since the threshold is undocumented and the small-gif case the server already handles is not worth a second code path, `convert_gif_to_mp4()` unconditionally uploads a converted mp4 in its place (temp file removed in a `finally`, including the failure path) and the original `.gif` name with an `.mp4` suffix is written back as the filename attribute so the temp name never reaches the article. Since conversion is what makes a `.gif` of any size reliably attachable, a `.gif` with no `ffmpeg` on PATH is rejected by `_validate_rich_files` **before the operation row is opened**, leaving the idempotency key free for the retry after the install. Every probe/convert/thumbnail call is a blocking subprocess and runs through `asyncio.to_thread`, never on the event loop. The markdown body is untouched by all of this: a `.gif` is still referenced as `tg://video?id=…` and `media_kind()` is unchanged. `RichMediaForbidden` (a `ValueError`, so it lands on 400 / exit 2 with its message rather than an empty 500) wraps both the upload loop and the send RPC for `ChatSend*ForbiddenError` — an article's media is part of its body, so there is no media-less half to retry. That match is by class name **and** by raw RPC string (`CHAT_SEND_*_FORBIDDEN` via `_MEDIA_RIGHTS_RPC_RE`, read off `exc.message`): telethon 1.44 generates no class for `CHAT_SEND_DOCS_FORBIDDEN`/`CHAT_SEND_AUDIOS_FORBIDDEN`, which arrive as a bare `ForbiddenError`, and those two cover exactly the kinds local uploads added (a video and a `.gif` are documents, `tg://audio` is an audio document) — a name-only set would leave the newest sends on the unmapped exit 1 / 500. A named error's `message` is a bare `FORBIDDEN`, so the string path cannot steal `ChatSendPlainForbiddenError`, which is about text rather than media. `MediaCaptionTooLongError` is deliberately **not** in that set: it is a plain `ValueError` naming the caption, since "this chat forbids media" would send the operator to check admin rights over a fixable caption. Both rely on `messages send` catching `ValueError` → exit 2 (the same handler every other `messages` command has), so a domain rejection never reads as the internal-error exit 1. HTTP/MCP never call `scan_media`, so a remote caller cannot name a server-side path. diff --git a/README.md b/README.md index b5f36dc..cc3a6d6 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,8 @@ Member references in `members`/`admins` (group create) and in `members bulk-add` `messages send --rich-markdown ` sends the file's contents as a Telegram **rich message** — the server parses the markdown itself and delivers a single article, so a >4096-character post is *not* split. The same input is `rich_markdown` (a string, not a path) on `POST /telegram/messages` and on the `telegram_messages_send` MCP tool. - **Dialect** (parsed server-side): `#`…`######` headings, tables with alignment, task lists, `>` quotes, fenced code with a language, `---` dividers, `~~strike~~`, `==marked==`, `||spoiler||`, footnotes, math, `
`, and media as a standalone block — by public HTTPS URL (`![](https://…jpg "caption")`, fetched by the server) or, **on the CLI only**, by local file (uploaded and referenced as `tg://photo?id=…` / `tg://video?id=…` / `tg://audio?id=…`). Headings, lists, tables, quotes, code, dividers, URL media and uploaded local media are verified over MTProto; the remaining constructs are documented for the Bot API twin of the same server feature and are unverified here. -- **Paragraph spacing** (on by default): the server renders neighbouring paragraphs tight against each other, so the markdown is rewritten before sending — a U+00A0-only spacer paragraph is inserted between two consecutive paragraphs, before every heading, and after every medium (a photo/video/audio block, or the ``/`` a run was grouped into), but never *before* media, never after a heading, never inside code/tables/lists/quotes/HTML blocks, and never next to a spacer the author already wrote. Turn it off with `--no-spaced-paragraphs` (HTTP/MCP: `spaced_paragraphs: false`) to keep the author's own spacing; the default also comes from `telegram.defaults.rich_markdown_spaced_paragraphs`. Note that this switches off *only* the spacer pass — media grouping and local-media rewriting are independent, so a truly byte-for-byte send also needs `--no-line-breaks`, `telegram.defaults.rich_markdown_grouping: none` (or `--media-group =none` per run) and an article with no local media. Spacers count toward both limits below: if spacing would push the article past 500 blocks it is sent unspaced with the warning `spaced_paragraphs disabled: N blocks would exceed the 500-block limit`. +- **Wikilinks** (always on, no knob): Obsidian `[[target]]` / `[[target|alias]]` links are expanded to plain text before anything else runs — the alias wins when present, otherwise the target reads as Obsidian renders it (a leading `#`, a link into the current note, drops; every other `#` becomes ` > `; block references like `note#^blk` fall out of that same rule unchanged). Only the first `|` separates target from alias, so further pipes stay in the alias (`[[A|B|C]]` → `B|C`); an empty half falls back to the other (`[[Note|]]` → `Note`, `[[|Стас]]` → `Стас`); a link with both halves empty (`[[]]`, `[[|]]`) is not a link and ships verbatim rather than collapsing to nothing. `![[…]]` embeds are untouched (that syntax is media, resolved as described below) and so is anything inside inline code or a fenced code block. Unlike frontmatter stripping and local-media resolution, this is **not** CLI-only — a literal `[[…]]` reaching a Telegram reader is a defect whichever surface sent it, so it runs on the CLI, HTTP, and MCP alike. A link nested inside its own target/alias (`[[[[a]]]]`) expands too, up to a bounded nesting depth — an unrealistic depth beyond that ships the remainder verbatim rather than rescanning the document without limit. `--dry-run` reports the count as `rich_markdown_wikilinks`. +- **Paragraph spacing** (on by default): the server renders neighbouring paragraphs tight against each other, so the markdown is rewritten before sending — a U+00A0-only spacer paragraph is inserted between two consecutive paragraphs, before every heading, and after every medium (a photo/video/audio block, or the ``/`` a run was grouped into), but never *before* media, never after a heading, never inside code/tables/lists/quotes/HTML blocks, and never next to a spacer the author already wrote. Turn it off with `--no-spaced-paragraphs` (HTTP/MCP: `spaced_paragraphs: false`) to keep the author's own spacing; the default also comes from `telegram.defaults.rich_markdown_spaced_paragraphs`. Note that this switches off *only* the spacer pass — media grouping and local-media rewriting are independent, so a truly byte-for-byte send also needs `--no-line-breaks`, `telegram.defaults.rich_markdown_grouping: none` (or `--media-group =none` per run), an article with no local media, and no `[[wikilinks]]`, which are always expanded (no knob). Spacers count toward both limits below: if spacing would push the article past 500 blocks it is sent unspaced with the warning `spaced_paragraphs disabled: N blocks would exceed the 500-block limit`. - **Line breaks** (on by default): Telegram parses the markdown itself and, like CommonMark, folds a *single* newline inside a paragraph into a space — so an Obsidian note's ``` Фотоальбом - https://… @@ -147,7 +148,7 @@ Member references in `members`/`admins` (group create) and in `members bulk-add` - **Obsidian frontmatter** (**CLI-only**): a leading `---` … `---` YAML block is dropped when the file is read, next to the BOM strip and for the same reason — this dialect has no notion of frontmatter, so the article would otherwise open with a divider and a large heading reading `tags: [...] date: ...`. Only an exact `---` on the first line starts a block, only a matching `---` ends one, and the lines between them must read as YAML (a `key: value` entry first, then only entries, `- ` items, indented continuations or blank lines), so a note that merely begins with a horizontal rule keeps it — even when a later `---` divider would otherwise close the pair; a file that is *nothing but* frontmatter is reported as empty. HTTP/MCP take a markdown string an agent composed rather than a note file, so their input is passed through untouched. - **Limits**: 1..32 768 characters (validated locally after normalization, inclusive), ~500 blocks and 50 media attachments (counted locally and reported as warnings — Telegram is the authority); the server additionally caps nesting and table columns and reports its own errors (`RICH_MESSAGE_MARKDOWN_INVALID`, `RICH_MESSAGE_TEXT_TOO_LONG`, …). - **Exclusivity**: `--rich-markdown` is a targeted-send-only alternative to the message body — it cannot be combined with `--text`, `--file`, `--file-url` (HTTP/MCP: `text`, `file_urls`, `base64_files`) or with mass mode. `--spaced-paragraphs`/`--no-spaced-paragraphs`, `--line-breaks`/`--no-line-breaks`, `--rich-file`, `--vault-dir` and `--media-group` are errors without `--rich-markdown` (CLI exit 2; HTTP `spaced_paragraphs`/`line_breaks` without `rich_markdown` is a `422`). -- Everything else is unchanged: entity resolution, the WRITE gate, `--operation-id` idempotency, topic/reply targeting (`--topic-id`/`--reply-to`), and scheduling (`--schedule-at`/`--delay`) all work. `--dry-run` reports the article as markers (`rich_markdown`, post-normalization `rich_markdown_chars`, `rich_markdown_blocks`, `rich_markdown_media`, `rich_markdown_file`, `spaced_paragraphs`, `spaced`, `line_breaks`, `media_grouping`, `rich_markdown_groups`, `rich_files`) rather than echoing a 32k body — the listed files are never read. Normalization warnings go to stderr as `warning: ...` on a real send and ride the result JSON as `warnings`. +- Everything else is unchanged: entity resolution, the WRITE gate, `--operation-id` idempotency, topic/reply targeting (`--topic-id`/`--reply-to`), and scheduling (`--schedule-at`/`--delay`) all work. `--dry-run` reports the article as markers (`rich_markdown`, post-normalization `rich_markdown_chars`, `rich_markdown_blocks`, `rich_markdown_media`, `rich_markdown_wikilinks`, `rich_markdown_file`, `spaced_paragraphs`, `spaced`, `line_breaks`, `media_grouping`, `rich_markdown_groups`, `rich_files`) rather than echoing a 32k body — the listed files are never read. Normalization warnings go to stderr as `warning: ...` on a real send and ride the result JSON as `warnings`. - A chat that forbids media rejects the **whole** article: Telegram's `ChatSendMediaForbiddenError` (and its per-type siblings) becomes a `RichMediaForbidden` error naming the chat (HTTP `400`, CLI exit 2). There is no media-less fallback — an article's media is part of its body. - If Telegram accepts the request but the response carries no readable message id, the send is **not** marked failed — the operation goes to `needs_review` (the queue never auto-retries it), because the article may well have been delivered. Check the chat before `operations retry`; a blind re-send under a fresh key would duplicate it. - A failed rich send is **not** silently retried as plain text — it surfaces as the normal send error, and the caller decides. Requires `telethon >= 1.44` (layer 227), now the project's minimum pin; if an older Telethon is force-installed anyway, only the rich send fails, with an explicit version error (HTTP `500 {"error": "rich_message_unsupported"}`, CLI exit 1, MCP error message) — and the idempotency key is left free, so the same `--operation-id` sends normally once Telethon is upgraded. diff --git a/docs/superpowers/plans/2026-08-01-obsidian-wikilink-strip.md b/docs/superpowers/plans/2026-08-01-obsidian-wikilink-strip.md new file mode 100644 index 0000000..8ef50c5 --- /dev/null +++ b/docs/superpowers/plans/2026-08-01-obsidian-wikilink-strip.md @@ -0,0 +1,582 @@ +# Obsidian Wikilink Stripping Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Expand Obsidian `[[wikilinks]]` to plain text in every rich-markdown article, on all three surfaces, so a vault note sent to Telegram no longer shows raw brackets and canonical note names. + +**Architecture:** One new pure pass, `strip_wikilinks()`, in `messages/rich_markdown.py`, called as the first step of `normalize_rich_markdown()`. It reuses the module's existing `scan_blocks()` to find code blocks and `_CODE_SPAN_RE` to mask inline code, then rewrites each surviving `[[…]]` in place. The count it reports rides on `RichMarkdownNormalization` next to the existing per-pass flags and surfaces in the CLI dry run. + +**Tech Stack:** Python 3.12, `re`, pytest. No new dependencies. + +## Global Constraints + +- **The rule is one rule, no special cases:** a wikilink expands to its alias when one is present, otherwise to its target with `#` replaced by ` > ` and a leading `#` dropped. +- **Only the first `|` separates** target from alias; further pipes belong to the alias (`[[A|B|C]]` → `B|C`). +- **An empty half falls back to the other:** `[[Note|]]` → `Note`, `[[|Стас]]` → `Стас`. Both halves empty (`[[]]`, `[[|]]`) is not a link — ship it verbatim, never collapse it to an empty string. +- **`![[…]]` embeds are never touched.** They are media, owned by `scan_media`. The absent `!` is the discriminator. +- **Inline code spans and fenced code blocks are never touched.** Reuse the existing `_CODE_SPAN_RE`; do not write a second matcher and do not hand-roll backtick pairing. +- **Code-span masking tests containment, not overlap** — the same rule `iter_line_media_refs` uses (`span_start <= start and end <= span_end`). +- **Identity on no-op.** Every pass in this module returns its input string by identity when it changes nothing; this one must too, or CRLF and the trailing newline stop surviving byte-for-byte. +- **No knob.** No CLI flag, no config key. Literal `[[…]]` in a Telegram article is always a defect. +- **No Telegram traffic in tests.** Every test uses in-memory fakes. +- Lint: `ruff check src tests` (line-length 100, py312). + +--- + +### Task 1: The `strip_wikilinks()` pass + +**Files:** +- Modify: `src/telegram_assistant/messages/rich_markdown.py` (add the pattern near `_CODE_SPAN_RE` at line 168; add the functions after `strip_yaml_frontmatter`, which ends around line 417) +- Test: `tests/test_rich_markdown_wikilinks.py` (create) + +**Interfaces:** +- Consumes: `scan_blocks(markdown) -> tuple[Block, ...]`, `split_lines(markdown) -> list[str]`, `_CODE_SPAN_RE`, and `Block` (fields `kind: str`, `start: int`, `end: int`, `children: tuple[Block, ...]`; `start`/`end` are a half-open range into the *document's* normalised line list, children included). +- Produces: `strip_wikilinks(markdown: str) -> tuple[str, int]` — the rewritten article and the number of links expanded. Returns `(markdown, 0)` **by identity** when nothing changed. Task 2 calls this. + +- [ ] **Step 1: Write the failing tests** + +Create `tests/test_rich_markdown_wikilinks.py`: + +```python +"""The Obsidian wikilink pass: [[Target|Alias]] becomes plain text. + +Telegram has no wikilink syntax, so an unexpanded link reaches the reader as +literal brackets plus the vault's canonical note name instead of the word the +author wrote. +""" + +import pytest + +from telegram_assistant.messages.rich_markdown import strip_wikilinks + + +@pytest.mark.parametrize( + ("source", "expected"), + [ + ("[[Андрей Смирнов]]", "Андрей Смирнов"), + ("[[Станислав Попов|Стасу]]", "Стасу"), + ("[[#Спорные моменты]]", "Спорные моменты"), + ("[[tasks#Настроить statusline]]", "tasks > Настроить statusline"), + ("[[Note#Heading|Алиас]]", "Алиас"), + # Only the first pipe separates; the rest belong to the alias. + ("[[A|B|C]]", "B|C"), + # An empty half falls back to the other one. + ("[[Note|]]", "Note"), + ("[[|Стас]]", "Стас"), + # A block reference falls out of the same rule with no special case. + ("[[note#^blk]]", "note > ^blk"), + # Surrounding prose is untouched, and several links share one line. + ( + "Отдал [[Денис Баталин|Дэну]] и [[Ирина Шлыкова|Ирине]].", + "Отдал Дэну и Ирине.", + ), + ], +) +def test_expands_wikilinks(source: str, expected: str) -> None: + assert strip_wikilinks(source) == (expected, source.count("[[")) + + +@pytest.mark.parametrize("source", ["[[]]", "[[|]]", "[[#]]"]) +def test_degenerate_link_is_left_verbatim(source: str) -> None: + """No target and no alias is not a link. + + Collapsing it to an empty string would silently delete characters the + author typed — worse than leaving a curiosity in the text. + """ + assert strip_wikilinks(source) == (source, 0) + + +def test_media_embed_is_left_to_scan_media() -> None: + """``![[…]]`` is media; ``scan_media`` rewrites it into a ``tg://`` ref.""" + source = "![[Pasted image 1.png|Закат]]" + assert strip_wikilinks(source) == (source, 0) + + +def test_embed_and_link_on_one_line() -> None: + source = "![[shot.png]] обсудили с [[Ольга Цветцых]]" + assert strip_wikilinks(source) == ("![[shot.png]] обсудили с Ольга Цветцых", 1) + + +def test_inline_code_span_is_opaque() -> None: + """An article documenting this dialect writes `[[Note]]` and means the text.""" + source = "Пиши `[[Note]]`, получишь Note." + assert strip_wikilinks(source) == (source, 0) + + +def test_code_span_overlapping_a_link_does_not_shield_it() -> None: + """Containment, not overlap — the rule ``iter_line_media_refs`` uses. + + The code span here starts inside the link and ends outside it. Skipping on + overlap would ship the raw brackets. + """ + source = "[[Note|запусти `make]] сначала`" + text, count = strip_wikilinks(source) + assert count == 1 + assert "[[" not in text + + +def test_fenced_code_block_is_opaque() -> None: + source = "```\n[[Note]]\n```\n" + assert strip_wikilinks(source) == (source, 0) + + +def test_fenced_code_inside_a_quote_is_opaque() -> None: + source = "> ```\n> [[Note]]\n> ```\n" + assert strip_wikilinks(source) == (source, 0) + + +def test_link_inside_a_quote_is_expanded() -> None: + source = "> Сказал [[Андрей Смирнов|Андрей]]\n" + assert strip_wikilinks(source) == ("> Сказал Андрей\n", 1) + + +def test_table_cell_pipe_no_longer_breaks_the_row() -> None: + """``[[A|B]]`` in a table currently splits the cell on its own pipe.""" + source = "| [[Станислав Попов|Стас]] | да |\n" + assert strip_wikilinks(source) == ("| Стас | да |\n", 1) + + +def test_returns_input_by_identity_when_unchanged() -> None: + """Identity is what keeps CRLF and the trailing newline byte-for-byte.""" + source = "# Заголовок\r\n\r\nПростой текст.\r\n" + text, count = strip_wikilinks(source) + assert text is source + assert count == 0 + + +def test_trailing_newline_survives_a_rewrite() -> None: + source = "Спросил у [[Ольга Андрющенко|Оли]].\n" + assert strip_wikilinks(source) == ("Спросил у Оли.\n", 1) + + +def test_is_idempotent() -> None: + source = "Отдал [[Денис Баталин|Дэну]].\n" + once, first = strip_wikilinks(source) + twice, second = strip_wikilinks(once) + assert first == 1 + assert second == 0 + assert twice is once +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `.venv/bin/pytest tests/test_rich_markdown_wikilinks.py -v` +Expected: FAIL at collection — `ImportError: cannot import name 'strip_wikilinks'`. + +- [ ] **Step 3: Add the pattern** + +In `src/telegram_assistant/messages/rich_markdown.py`, directly below `_CODE_SPAN_RE` (line 168) and its comment block, add: + +```python +#: An Obsidian wikilink: ``[[target]]`` or ``[[target|alias]]``. The negative +#: lookbehind is what keeps ``![[file.png]]`` out — that is a media embed, owned +#: by :func:`scan_media`, and expanding it here would strip the file reference +#: down to prose before anything could upload it. +_WIKILINK_RE = re.compile(r"(?[^\[\]]*)\]\]") +``` + +- [ ] **Step 4: Write the implementation** + +Add after `strip_yaml_frontmatter` (which ends around line 417), before `split_lines`: + +```python +def _expand_wikilink(body: str) -> str | None: + """Return the text a wikilink's body should become, or ``None`` to keep it. + + One rule, no special cases: the alias wins when there is one, otherwise the + target reads as Obsidian renders it — ``#`` becomes ``>`` and a leading + ``#`` (a link into the current note) simply drops. Block references + (``note#^blk``) fall out of that unchanged, which is why they need no + branch of their own. + """ + + target, _, alias = body.partition("|") + alias = alias.strip() + if alias: + return alias + target = target.strip() + if target.startswith("#"): + target = target[1:] + if not target: + # Neither half carries text, so this is not a link. Returning None + # ships it verbatim: silently deleting characters the author typed is + # worse than leaving a curiosity in the article. + return None + return target.replace("#", " > ") + + +def _strip_line_wikilinks(line: str) -> tuple[str, int]: + """Expand every wikilink on one line, leaving inline code spans alone.""" + + code_spans = [match.span() for match in _CODE_SPAN_RE.finditer(line)] + out: list[str] = [] + cursor = 0 + count = 0 + for match in _WIKILINK_RE.finditer(line): + start, end = match.span() + # Containment, not overlap — the rule iter_line_media_refs uses. A span + # that merely overlaps the link (a backtick inside the alias) must not + # shield it, or the raw brackets ship. + if any(span_start <= start and end <= span_end for span_start, span_end in code_spans): + continue + text = _expand_wikilink(match.group("body")) + if text is None: + continue + out.append(line[cursor:start]) + out.append(text) + cursor = end + count += 1 + if not count: + return line, 0 + out.append(line[cursor:]) + return "".join(out), count + + +def _code_line_indices(blocks: tuple[Block, ...] | list[Block]) -> set[int]: + """Document line indices covered by a code block, nesting included.""" + + found: set[int] = set() + for block in blocks: + if block.kind == "code": + found.update(range(block.start, block.end)) + elif block.children: + found.update(_code_line_indices(block.children)) + return found + + +def strip_wikilinks(markdown: str) -> tuple[str, int]: + """Expand Obsidian ``[[wikilinks]]`` to plain text; report how many. + + Telegram has no wikilink syntax, so an unexpanded link reaches the reader + as literal brackets around the vault's canonical note name — not the word + the author wrote. Unlike :func:`strip_yaml_frontmatter` and + :func:`scan_media`, which answer "this is a file from a vault" and are + therefore CLI-only, a wikilink is meaningless in Telegram no matter which + surface submitted it, so this runs for all three. + + Code is opaque: fenced blocks via :func:`scan_blocks`, inline spans via + :data:`_CODE_SPAN_RE`. An article documenting this dialect writes + ``[[Note]]`` inside backticks and means the characters. + + Returns ``(markdown, 0)`` by identity when nothing changed, which is what + keeps CRLF and the trailing newline intact for a byte-for-byte send. + """ + + if "[[" not in markdown: + return markdown, 0 + + code_lines = _code_line_indices(scan_blocks(markdown)) + lines = split_lines(markdown) + out: list[str] = [] + total = 0 + for index, line in enumerate(lines): + if index in code_lines or "[[" not in line: + out.append(line) + continue + rewritten, count = _strip_line_wikilinks(line) + out.append(rewritten) + total += count + if not total: + return markdown, 0 + text = "\n".join(out) + return (text + "\n" if markdown.endswith(("\n", "\r")) else text), total +``` + +- [ ] **Step 5: Run the tests to verify they pass** + +Run: `.venv/bin/pytest tests/test_rich_markdown_wikilinks.py -v` +Expected: PASS, all tests, no warnings. + +- [ ] **Step 6: Run the full suite and lint** + +Run: `.venv/bin/pytest -q && .venv/bin/ruff check src tests` +Expected: the whole suite green (no existing test asserts that `[[…]]` survives; if one does, it predates this feature — report it rather than editing it silently), ruff clean. + +- [ ] **Step 7: Commit** + +```bash +git add src/telegram_assistant/messages/rich_markdown.py tests/test_rich_markdown_wikilinks.py +git commit -m "feat(rich-markdown): expand Obsidian wikilinks to plain text" +``` + +--- + +### Task 2: Wire the pass into normalization + +**Files:** +- Modify: `src/telegram_assistant/messages/rich_markdown.py` — `RichMarkdownNormalization` (line 254) and `normalize_rich_markdown` (line 650, body starts line 694) +- Test: `tests/test_rich_markdown_normalize.py` (extend), `tests/test_messages_rich_surfaces.py` (extend), `tests/test_mcp_tools.py` (extend) + +**Interfaces:** +- Consumes: `strip_wikilinks(markdown: str) -> tuple[str, int]` from Task 1. +- Produces: `RichMarkdownNormalization.wikilinks: int` — the number of links expanded, defaulting to `0`. Task 3 reads it. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/test_rich_markdown_normalize.py`: + +```python +def test_normalize_expands_wikilinks_and_counts_them() -> None: + result = normalize_rich_markdown("Отдал [[Денис Баталин|Дэну]].\n") + assert "[[" not in result.markdown + assert "Дэну" in result.markdown + assert result.wikilinks == 1 + + +def test_normalize_reports_zero_wikilinks_when_there_are_none() -> None: + assert normalize_rich_markdown("Просто текст.\n").wikilinks == 0 + + +def test_wikilinks_are_expanded_before_blocks_are_counted() -> None: + """The pass edits inside lines, so blocks must be counted after it. + + A wikilink's own pipe splits a table cell; expanding it first is what makes + the reported block structure the one Telegram actually receives. + """ + source = "| a | b |\n| --- | --- |\n| [[Станислав Попов|Стас]] | да |\n" + result = normalize_rich_markdown(source) + assert "| Стас | да |" in result.markdown +``` + +Append to `tests/test_messages_rich_surfaces.py`, beside `test_http_rich_send_passes_markdown_to_backend` (line 135), reusing that module's `RecordingMessageBackend`, `_client` and `AUTH`: + +```python +def test_http_rich_send_expands_wikilinks() -> None: + """The pass is not CLI-only: an agent relaying note text has the same defect.""" + backend = RecordingMessageBackend() + client = _client(backend) + resp = client.post( + "/telegram/messages", + json={ + "telegram_chat_id": -100, + "rich_markdown": "Спросил у [[Станислав Попов|Стаса]].\n", + "operation_id": "rich-http-wikilink", + }, + headers=AUTH, + ) + assert resp.status_code == 200, resp.text + sent = backend.sent[0]["rich_markdown"] + assert "Спросил у Стаса." in sent + assert "[[" not in sent +``` + +Append to `tests/test_mcp_tools.py`, beside `test_mcp_send_rich_markdown_reaches_backend` (line 692), reusing that module's `FakeRichMessageBackend`, `_client`, `_mint_token`, `_initialize` and `_call_tool`: + +```python +def test_mcp_send_rich_markdown_expands_wikilinks( + minimal_config_yaml: str, tmp_path: Path +) -> None: + """Same proof for the MCP tool surface — expansion is surface-independent.""" + backend = FakeRichMessageBackend() + with _client(minimal_config_yaml, tmp_path, message_backend=backend) as client: + token = _mint_token(client) + _initialize(client, token) + + result = _call_tool( + client, + token, + "telegram_messages_send", + { + "telegram_chat_id": -100123, + "rich_markdown": "Отдал [[Денис Баталин|Дэну]].\n", + "operation_id": "mcp-rich-wikilink", + }, + ) + + assert result["isError"] is False, result + sent = backend.sent[0]["rich_markdown"] + assert "Отдал Дэну." in sent + assert "[[" not in sent +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `.venv/bin/pytest tests/test_rich_markdown_normalize.py -k wikilink -v` +Expected: FAIL with `AttributeError: 'RichMarkdownNormalization' object has no attribute 'wikilinks'`. + +- [ ] **Step 3: Add the field** + +In `RichMarkdownNormalization` (line 254), after `lines_split: bool = False`: + +```python + #: How many Obsidian wikilinks :func:`strip_wikilinks` expanded. Unlike the + #: flags above this is a count, because it is what a surface reports to the + #: operator — there is no knob to explain, only a number. + wikilinks: int = 0 +``` + +- [ ] **Step 4: Call the pass first** + +In `normalize_rich_markdown`, replace the opening two lines of the body (line 694-695): + +```python + blocks = scan_blocks(markdown) + warnings: list[str] = [] +``` + +with: + +```python + # First, and before scan_blocks: this pass edits *inside* lines, so every + # later pass — and the block count the 500-block rollback weighs — must see + # the text that will actually be sent. It also settles a table cell whose + # wikilink pipe would otherwise split it. + markdown, wikilinks = strip_wikilinks(markdown) + blocks = scan_blocks(markdown) + warnings: list[str] = [] +``` + +Then in the `return RichMarkdownNormalization(...)` at the end of the function, add the argument: + +```python + wikilinks=wikilinks, +``` + +Note that `grouped is not markdown` and the other identity comparisons in the function keep working: they compare against the rebound local `markdown`, which is exactly what the later passes received. + +- [ ] **Step 5: Run the tests to verify they pass** + +Run: `.venv/bin/pytest tests/test_rich_markdown_normalize.py tests/test_messages_rich_surfaces.py tests/test_mcp_tools.py -v` +Expected: PASS. + +- [ ] **Step 6: Run the full suite and lint** + +Run: `.venv/bin/pytest -q && .venv/bin/ruff check src tests` +Expected: green, ruff clean. + +- [ ] **Step 7: Commit** + +```bash +git add src/telegram_assistant/messages/rich_markdown.py tests/ +git commit -m "feat(rich-markdown): expand wikilinks on every surface" +``` + +--- + +### Task 3: Dry-run marker and documentation + +**Files:** +- Modify: `src/telegram_assistant/cli/rich_send.py` — `rich_dry_run_markers` (line 249) +- Modify: `CLAUDE.md`, `README.md`, `skills/telegram-assistant/SKILL.md` +- Test: `tests/test_messages_rich_spacing_flag.py` — despite its name it is the de-facto owner of the CLI dry-run marker assertions (`rich_markdown_blocks`, `rich_markdown_media`, and the plain-send `None` shape), and it already has the `_cli_setup(tmp_path, monkeypatch, markdown=…)` harness. Reuse it rather than standing up a second CLI harness. + +**Interfaces:** +- Consumes: `RichMarkdownNormalization.wikilinks` from Task 2. +- Produces: the dry-run key `rich_markdown_wikilinks: int | None` (`None` for a plain send, matching every other `rich_*` marker). + +- [ ] **Step 1: Write the failing test** + +Append to `tests/test_messages_rich_spacing_flag.py`, after `test_cli_dry_run_plain_send_has_no_spacing_markers` (line 308): + +```python +def test_cli_dry_run_reports_expanded_wikilinks( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + config_file, md_file, backend = _cli_setup( + tmp_path, monkeypatch, markdown="Отдал [[Денис Баталин|Дэну]].\n" + ) + + result = _run_cli( + [ + "messages", + "send", + "--chat-id", + "-100", + "--rich-markdown", + str(md_file), + "--dry-run", + "--config", + str(config_file), + ] + ) + + assert result.exit_code == 0, _cli_output(result) + resolved = json.loads(result.stdout.strip().splitlines()[-1])["resolved"] + assert resolved["rich_markdown_wikilinks"] == 1 + assert backend.sent == [] + + +def test_cli_dry_run_plain_send_has_no_wikilink_marker( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The marker shape must never depend on the mode.""" + config_file, _md_file, _backend = _cli_setup(tmp_path, monkeypatch) + + result = _run_cli( + [ + "messages", + "send", + "--chat-id", + "-100", + "--text", + "hello", + "--dry-run", + "--config", + str(config_file), + ] + ) + + assert result.exit_code == 0, _cli_output(result) + resolved = json.loads(result.stdout.strip().splitlines()[-1])["resolved"] + assert resolved["rich_markdown_wikilinks"] is None +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `.venv/bin/pytest -k "wikilink and dry_run" -v` +Expected: FAIL with `KeyError: 'rich_markdown_wikilinks'`. + +- [ ] **Step 3: Add the marker** + +In `rich_dry_run_markers` (`src/telegram_assistant/cli/rich_send.py`), inside the `markers` dict, directly after the `"rich_markdown_media"` entry: + +```python + "rich_markdown_wikilinks": ( + normalization.wikilinks if normalization is not None else None + ), +``` + +- [ ] **Step 4: Run to verify it passes** + +Run: `.venv/bin/pytest -k "wikilink" -v` +Expected: PASS. + +- [ ] **Step 5: Pay the documentation debt in CLAUDE.md** + +`CLAUDE.md` currently opens the rich-markdown bullet with: + +> **`messages/rich_markdown.py` owns every rewrite of an article, and it only ever inserts or wraps lines.** + +That sentence is now false — this pass edits within a line. Rewrite the claim so it stays true and still explains *why* the module has no markdown parser, then document the pass itself: the rule and its table, that `![[…]]` and code (fenced and inline, containment not overlap) are excluded, that it runs first and before `scan_blocks` so the block count matches what is sent, that it is idempotent and identity-on-no-op, that it has no knob because a literal `[[…]]` is always a defect, and that — unlike `strip_yaml_frontmatter` and `scan_media` — it is **not** CLI-only, with the reason (a wikilink is meaningless in Telegram whoever submitted it). + +- [ ] **Step 6: Update README.md and the skill** + +In `README.md`, note the pass alongside the other normalization passes in the rich-markdown section. In `skills/telegram-assistant/SKILL.md`, add it to the `messages send` rich-markdown description and add `rich_markdown_wikilinks` to the listed dry-run markers. Then re-sync: + +```bash +cp skills/telegram-assistant/SKILL.md ~/.claude/skills/telegram-assistant/SKILL.md +``` + +- [ ] **Step 7: Run the full suite and lint** + +Run: `.venv/bin/pytest -q && .venv/bin/ruff check src tests` +Expected: green (including `tests/test_skill_inventory.py`, which fails when the CLI catalog drifts from the skill), ruff clean. + +- [ ] **Step 8: Commit** + +```bash +git add src/telegram_assistant/cli/rich_send.py tests/ CLAUDE.md README.md skills/telegram-assistant/SKILL.md +git commit -m "docs(rich-markdown): report and document wikilink expansion" +``` + +--- + +## Manual verification (optional, requires the human) + +The live check is one send of a real vault note to Saved Messages. It is a +mutating live call, so **ask first — every time**; never run it on your own +initiative. The note +`/home/popstas/projects/text/obsidian/home/Notes/2026/07/telegram-assistant/Планёрка 29.07.2026.md` +holds 19 wikilinks and is the article whose earlier send (message 408926) +motivated this work; a `--dry-run` against it needs no permission and should +report `rich_markdown_wikilinks: 19`. diff --git a/docs/superpowers/specs/2026-08-01-obsidian-wikilink-strip-design.md b/docs/superpowers/specs/2026-08-01-obsidian-wikilink-strip-design.md new file mode 100644 index 0000000..5e194c7 --- /dev/null +++ b/docs/superpowers/specs/2026-08-01-obsidian-wikilink-strip-design.md @@ -0,0 +1,130 @@ +# Obsidian wikilink stripping in rich markdown + +**Date:** 2026-08-01 +**Status:** approved, ready for planning + +## Problem + +An Obsidian note sent as a Telegram article carries its internal +`[[wikilinks]]` verbatim. Telegram has no such syntax, so the reader sees raw +brackets and, for aliased links, the vault's canonical note name instead of the +word the author wrote. + +This is not hypothetical. Message 408926 in Saved Messages — the note +`Notes/2026/07/telegram-assistant/Планёрка 29.07.2026.md`, sent 2026-08-01 — +went out carrying 19 wikilinks, among them `[[Станислав Попов|Стасу]]` and +`[[Денис Баталин|Дэн]]`. The vault holds 554 of them across its notes. + +## Rule + +A wikilink is `[[…]]` **not** preceded by `!`. It expands to: + +- its **alias** when one is present (the text after the first `|`), otherwise +- its **target**, with `#` replaced by ` > ` and a leading `#` dropped. + +| input | output | +|---|---| +| `[[Андрей Смирнов]]` | `Андрей Смирнов` | +| `[[Станислав Попов\|Стасу]]` | `Стасу` | +| `[[#Спорные моменты]]` | `Спорные моменты` | +| `[[tasks#Настроить statusline]]` | `tasks > Настроить statusline` | +| `[[Note#Heading\|Алиас]]` | `Алиас` | + +That single rule is the whole specification — there are no special cases. Block +references (`[[note#^blk]]` → `note > ^blk`) fall out of it for free; the vault +contains none, so no code is written for them specifically. + +Only the **first** `|` separates target from alias; any further pipes belong to +the alias, so `[[A|B|C]]` yields `B|C` — which is what Obsidian renders. + +An empty half falls back to the other one: `[[Note|]]` yields `Note` and +`[[|Стас]]` yields `Стас`. A link with **both** halves empty (`[[]]`, `[[|]]`) +is not a link at all and is shipped verbatim rather than collapsed to an empty +string — silently deleting a character run the author typed is worse than +leaving a curiosity in the text. + +## What the pass must not touch + +- **`![[…]]` embeds.** Those are media, owned by `scan_media`, which rewrites + them into `tg://` references. The leading `!` is what distinguishes them, so + the pattern must require its absence. +- **Inline code spans.** An article documenting this dialect writes + `` `[[Note]]` `` and means the characters. Mask them with the same + `_CODE_SPAN_RE` `scan_media` already uses — do not write a second matcher, and + do not hand-roll backtick pairing. (A naive `` `[^`]*\[\[…\]\][^`]*` `` probe + run against the vault during design produced only false positives: it matched + the span *between* two separate code spans.) +- **Fenced code blocks.** Opaque to the block scanner for the same reason. + +## Placement + +`strip_wikilinks()` in `messages/rich_markdown.py`, called as the **first** step +of `normalize_rich_markdown()`, before `scan_blocks()`. + +All three surfaces get it, because `send_message` normalises once for every +surface. This differs deliberately from `strip_yaml_frontmatter()` and +`scan_media()`, which are CLI-only: those answer "this is a file from a vault", a +question only the CLI's file-read boundary can ask. A wikilink, by contrast, is +meaningless in Telegram no matter who submitted it — an MCP client relaying note +text has the same defect as a CLI reading the note directly. + +Running before `scan_blocks()` is what makes the block count honest: this pass +edits *inside* lines, so blocks must be counted against the text that will +actually be sent. It also fixes a latent bug for free — `| [[A|B]] |` in a table +currently breaks the cell on the wikilink's own pipe, and after the pass it does +not. + +## Invariants + +- **Identity on no-op.** Returns the input string by identity when it changes + nothing, so CRLF and the trailing newline survive byte-for-byte, exactly as + the other passes promise. +- **Idempotent.** No `[[` survives the pass, so re-running it is a no-op — + unlike `_split_paragraph_lines`, which deliberately gives up idempotency. +- **Usually shrinks, but can grow.** Removing `[[`/`]]`/an alias pipe is -4 + or more, but each `#` in a bare target becomes `" > "` (+2 net per `#`), so + a target with three or more `#` grows the source overall. The + `MAX_RICH_MARKDOWN_CHARS` pre-check that runs *before* normalisation is + unaffected and stays where it is: it guards the event loop ahead of the + WRITE gate, and that reason is independent of which direction a pass moves + the length — it is deliberately conservative rather than exact. A + 33k-character source dense with wikilinks would be rejected before the pass + could bring it under the limit; that is accepted, not an oversight. The + post-normalisation over-limit check names `"wikilink expansion"` in + `grew_by` alongside the other passes, so growth caused by this pass is + never blamed on — or hidden behind — paragraph spacing/line splitting/media + grouping. + +## Reporting + +`rich_markdown_wikilinks: ` in the dry-run payload — how many links the +pass expanded. No CLI flag and no config knob: literal `[[…]]` in a Telegram +article is always a defect, so there is nothing to switch off. + +## Testing + +Unit tests in the existing rich-markdown test module: + +- each row of the rule table above +- `![[file.png]]` left untouched, and the interaction with `scan_media` (embeds + already rewritten to `![](tg://…)` by the time the CLI normalises) +- a wikilink inside an inline code span and inside a fenced block, both left + verbatim +- a wikilink whose caption contains a code span (overlap, not containment — the + case `scan_media` documents as the one silent drop it must never make) +- identity return on markdown with no wikilinks, including CRLF input +- idempotency: normalising the output again changes nothing +- `| [[A|B]] |` in a table yields a well-formed single cell +- HTTP and MCP sends strip too — the surface-level proof that this is not + CLI-only + +## Documentation debt + +`CLAUDE.md` currently states that `messages/rich_markdown.py` "owns every +rewrite of an article, and it only ever inserts or wraps lines." This pass is +the first exception — it edits within a line. That sentence must be rewritten in +the same change, or the most load-bearing paragraph in the file starts lying. + +`README.md` and `skills/telegram-assistant/SKILL.md` need the behaviour noted +alongside the other normalization passes, and the skill re-synced to +`~/.claude/skills/telegram-assistant/SKILL.md`. diff --git a/skills/telegram-assistant/SKILL.md b/skills/telegram-assistant/SKILL.md index 99b3fff..b9785ab 100644 --- a/skills/telegram-assistant/SKILL.md +++ b/skills/telegram-assistant/SKILL.md @@ -575,10 +575,11 @@ command rather than after `folder_cache_ttl` seconds. send time, and any reply target. For a rich send it echoes the article as markers only — `rich_markdown: true`, `rich_markdown_chars` (post-normalization), `rich_markdown_blocks`, `rich_markdown_media`, - `rich_markdown_file`, `spaced_paragraphs` (the effective decision), - `spaced` (what the pass actually did), `line_breaks`, `media_grouping`, - `rich_markdown_groups` and `rich_files` — never the body; show the - human those, plus the file path they can re-read. + `rich_markdown_wikilinks`, `rich_markdown_file`, `spaced_paragraphs` + (the effective decision), `spaced` (what the pass actually did), + `line_breaks`, `media_grouping`, `rich_markdown_groups` and + `rich_files` — never the body; show the human those, plus the file + path they can re-read. - Rich markdown (`--rich-markdown`): use it when the human asks for a post/article/статья with formatting Telegram's plain text cannot carry — headings, tables, quotes, long-form (>4096 chars, up to 32 768). The @@ -593,6 +594,19 @@ command rather than after `folder_cache_ttl` seconds. `--text`/`--file`/`--file-url`/`--mass`. If a rich send fails, do **not** silently retry it as a plain `--text` message — report the error and ask. +- Wikilinks (**always on, no flag**): Obsidian `[[target]]` / + `[[target|alias]]` links are expanded to plain text before any other + pass runs — the alias wins when present, otherwise the target reads + as Obsidian renders it (a leading `#` drops, every other `#` becomes + ` > `). Only the first `|` splits target from alias, so further pipes + stay in the alias; an empty half falls back to the other; `[[]]`/ + `[[|]]` are not links and ship verbatim. `![[…]]` embeds and anything + inside code (inline or fenced) are left alone. This runs on every + surface, not just the CLI — unlike frontmatter stripping and local + media, a wikilink is meaningless in Telegram whoever sent it. Nesting + (`[[[[a]]]]`) expands too, up to a bounded depth — an unrealistic note + past that ships the remainder verbatim. The + dry-run reports the count as `rich_markdown_wikilinks`. - Paragraph spacing (**on by default**): the server renders neighbouring paragraphs tight against each other, so the CLI/HTTP/MCP insert a U+00A0-only spacer paragraph between two paragraphs and before every @@ -601,7 +615,8 @@ command rather than after `folder_cache_ttl` seconds. e.g. because they hand-tuned the spacing. It switches off the spacer pass only — media grouping and local-media rewriting are independent, so mention `--media-group =none` if they want the source truly - byte-for-byte. The flag is an error + byte-for-byte, and note wikilinks are always expanded regardless (no + knob) — a `[[…]]`-bearing article can never go byte-for-byte. The flag is an error (exit 2 / 422) without `--rich-markdown`. The default also comes from `telegram.defaults.rich_markdown_spaced_paragraphs`. Spacers count toward both the character and the block limit; if spacing would push @@ -1494,11 +1509,12 @@ Request: «Опубликуй в чате Клиент / проект стать 4. Show the resolved chat id and the article markers from the dry-run JSON (`rich_markdown: true`, `rich_markdown_chars`, - `rich_markdown_blocks`, `rich_markdown_media`, `rich_markdown_file`, - `spaced_paragraphs`, plus `rich_files` when the article carries local - media) — the body is deliberately not echoed, so quote the file path - and, if the human wants to review the text, show the file contents - yourself. Relay any `warnings` verbatim. + `rich_markdown_blocks`, `rich_markdown_media`, `rich_markdown_wikilinks`, + `rich_markdown_file`, `spaced_paragraphs`, `spaced`, `line_breaks`, + `media_grouping`, `rich_markdown_groups`, plus `rich_files` when the + article carries local media) — the body is deliberately not echoed, so + quote the file path and, if the human wants to review the text, show + the file contents yourself. Relay any `warnings` verbatim. 5. If the dry-run reports a non-empty `rich_markdown_groups`, ask about the grouping **before** asking for the send confirmation. One `AskUserQuestion` call: «В статье N групп подряд идущих медиа, все diff --git a/src/telegram_assistant/cli/rich_send.py b/src/telegram_assistant/cli/rich_send.py index 7060d43..8b7be38 100644 --- a/src/telegram_assistant/cli/rich_send.py +++ b/src/telegram_assistant/cli/rich_send.py @@ -286,6 +286,9 @@ def rich_dry_run_markers( "rich_markdown_media": ( normalization.media if normalization is not None else None ), + "rich_markdown_wikilinks": ( + normalization.wikilinks if normalization is not None else None + ), "rich_markdown_file": (str(rich_markdown) if is_rich else None), # Local media the real send would upload. The files are listed, # never read — a dry run touches no bytes and no network. diff --git a/src/telegram_assistant/messages/rich_markdown.py b/src/telegram_assistant/messages/rich_markdown.py index 437f703..f61e58f 100644 --- a/src/telegram_assistant/messages/rich_markdown.py +++ b/src/telegram_assistant/messages/rich_markdown.py @@ -56,6 +56,23 @@ #: ``RecursionError`` on caller input. MAX_BLOCK_NESTING = 64 +#: How many times :func:`strip_wikilinks` re-scans a document looking for a +#: still-nested ``[[…]]`` (see :func:`_strip_once`). Real Obsidian notes nest +#: wikilinks one, maybe two, levels deep — a link inside an alias inside +#: another alias is already exotic. Each extra pass is a full document +#: rescan (``scan_blocks`` + a line-by-line regex sweep) on the event loop, +#: *ahead* of the WRITE gate, and unlike a single scan's cost — bounded by +#: :data:`~telegram_assistant.messages.service.MAX_RICH_MARKDOWN_CHARS` — +#: an uncapped loop's *pass count* is not: a pathological run of nested +#: brackets (``"["*16000 + "a" + "]"*16000``) resolves exactly one bracket +#: pair per pass, so an uncapped loop turns one 32k-character request into +#: ~16 000 rescans of it. This cap bounds total pre-auth work to a small +#: constant multiple of one scan instead, the same trade-off +#: :func:`_scan_nested` makes when it stops at :data:`MAX_BLOCK_NESTING`: a +#: leftover past the cap ships with literal ``[[``/``]]`` still in it, +#: exactly like a degenerate ``[[]]``. +MAX_WIKILINK_PASSES = 8 + #: HTML container tags the dialect defines; their contents are scanned as #: nested blocks so grouping can tell author-written groups from runs it may #: wrap itself. @@ -167,6 +184,12 @@ #: ``` `![](shot.png)` ``` and means the text, not a file to upload. _CODE_SPAN_RE = re.compile(r"(?P`+)(?P.+?)(?P=ticks)") +#: An Obsidian wikilink: ``[[target]]`` or ``[[target|alias]]``. The negative +#: lookbehind is what keeps ``![[file.png]]`` out — that is a media embed, owned +#: by :func:`scan_media`, and expanding it here would strip the file reference +#: down to prose before anything could upload it. +_WIKILINK_RE = re.compile(r"(?[^\[\]]*)\]\]") + #: A CommonMark backslash escape: a backslash before ASCII punctuation. A #: backslash before anything else is a literal backslash, which is what keeps a #: Windows-style ``C:\Users\me\a.png`` target intact. @@ -276,6 +299,10 @@ class RichMarkdownNormalization: grouped: bool = False spacers_added: bool = False lines_split: bool = False + #: How many Obsidian wikilinks :func:`strip_wikilinks` expanded. Unlike the + #: flags above this is a count, because it is what a surface reports to the + #: operator — there is no knob to explain, only a number. + wikilinks: int = 0 class MediaResolutionError(ValueError): @@ -444,6 +471,146 @@ def _is_frontmatter_body(lines: list[str]) -> bool: ) +def _expand_wikilink(body: str) -> str | None: + """Return the text a wikilink's body should become, or ``None`` to keep it. + + One rule, no special cases: the alias wins when there is one, otherwise the + target reads as Obsidian renders it — ``#`` becomes ``>`` and a leading + ``#`` (a link into the current note) simply drops. Block references + (``note#^blk``) fall out of that unchanged, which is why they need no + branch of their own. + """ + + target, _, alias = body.partition("|") + alias = alias.strip() + if alias: + return alias + target = target.strip() + if target.startswith("#"): + target = target[1:] + if not target: + # Neither half carries text, so this is not a link. Returning None + # ships it verbatim: silently deleting characters the author typed is + # worse than leaving a curiosity in the article. + return None + return target.replace("#", " > ") + + +def _strip_line_wikilinks(line: str) -> tuple[str, int]: + """Expand every wikilink on one line, leaving inline code spans alone.""" + + code_spans = [match.span() for match in _CODE_SPAN_RE.finditer(line)] + out: list[str] = [] + cursor = 0 + count = 0 + for match in _WIKILINK_RE.finditer(line): + start, end = match.span() + # Containment, not overlap — the rule iter_line_media_refs uses. A span + # that merely overlaps the link (a backtick inside the alias) must not + # shield it, or the raw brackets ship. + if any(span_start <= start and end <= span_end for span_start, span_end in code_spans): + continue + text = _expand_wikilink(match.group("body")) + if text is None: + continue + out.append(line[cursor:start]) + out.append(text) + cursor = end + count += 1 + if not count: + return line, 0 + out.append(line[cursor:]) + return "".join(out), count + + +def _code_line_indices(blocks: tuple[Block, ...] | list[Block]) -> set[int]: + """Document line indices covered by a code block, nesting included.""" + + found: set[int] = set() + for block in blocks: + if block.kind == "code": + found.update(range(block.start, block.end)) + elif block.children: + found.update(_code_line_indices(block.children)) + return found + + +def _strip_once(markdown: str) -> tuple[str, int]: + """Expand every top-level Obsidian ``[[wikilink]]`` once; report how many. + + A link whose target or alias itself contains ``[[…]]`` — the body pattern + excludes brackets, so it cannot match a link spanning another one — is + only resolved one level at a time: the innermost pair expands and the + outer pair's own ``[[``/``]]`` survive this single call. :func:`strip_wikilinks` + is the public entry point and loops this to a fixpoint. + + Code is opaque: fenced blocks via :func:`scan_blocks`, inline spans via + :data:`_CODE_SPAN_RE`. An article documenting this dialect writes + ``[[Note]]`` inside backticks and means the characters. + + Returns ``(markdown, 0)`` by identity when nothing changed, which is what + keeps CRLF and the trailing newline intact for a byte-for-byte send. + """ + + if "[[" not in markdown: + return markdown, 0 + + code_lines = _code_line_indices(scan_blocks(markdown)) + lines = split_lines(markdown) + out: list[str] = [] + total = 0 + for index, line in enumerate(lines): + if index in code_lines or "[[" not in line: + out.append(line) + continue + rewritten, count = _strip_line_wikilinks(line) + out.append(rewritten) + total += count + if not total: + return markdown, 0 + text = "\n".join(out) + return (text + "\n" if markdown.endswith(("\n", "\r")) else text), total + + +def strip_wikilinks(markdown: str) -> tuple[str, int]: + """Expand Obsidian ``[[wikilinks]]`` to plain text; report how many. + + Telegram has no wikilink syntax, so an unexpanded link reaches the reader + as literal brackets around the vault's canonical note name — not the word + the author wrote. Unlike :func:`strip_yaml_frontmatter` and + :func:`scan_media`, which answer "this is a file from a vault" and are + therefore CLI-only, a wikilink is meaningless in Telegram no matter which + surface submitted it, so this runs for all three. + + A link can nest inside its own target or alias (``[[[[a]]]]``, + ``[[a|[[b]]]]``) — Obsidian itself renders these as the innermost link's + text — and :func:`_strip_once`'s body pattern deliberately excludes + brackets, so one call only resolves the innermost pair. A literal ``[[`` + reaching a Telegram reader is always a defect, so this loops + :func:`_strip_once` up to :data:`MAX_WIKILINK_PASSES` times rather than + settling for "mostly expanded": each iteration strictly removes at least + one ``[[``/``]]`` pair. The reported count is the sum across every + iteration, matching the number of ``[[`` the source contained — up to + the cap. Nesting past :data:`MAX_WIKILINK_PASSES` levels deep (not a + real note; see the constant's own comment) stops there and ships the + remaining ``[[``/``]]`` verbatim, the same trade-off + :func:`_scan_nested` makes past :data:`MAX_BLOCK_NESTING`. + + Returns ``(markdown, 0)`` by identity when nothing changed, which is what + keeps CRLF and the trailing newline intact for a byte-for-byte send. + """ + + text, total = _strip_once(markdown) + passes = 1 + while total and "[[" in text and passes < MAX_WIKILINK_PASSES: + text, again = _strip_once(text) + passes += 1 + if not again: + break + total += again + return text, total + + def split_lines(markdown: str) -> list[str]: """Normalise newlines (CRLF/CR → LF) and split into lines. @@ -691,6 +858,11 @@ def normalize_rich_markdown( decides. """ + # First, and before scan_blocks: this pass edits *inside* lines, so every + # later pass — and the block count the 500-block rollback weighs — must see + # the text that will actually be sent. It also settles a table cell whose + # wikilink pipe would otherwise split it. + markdown, wikilinks = strip_wikilinks(markdown) blocks = scan_blocks(markdown) warnings: list[str] = [] @@ -757,6 +929,7 @@ def _split(text: str, text_blocks: tuple[Block, ...]) -> str: grouped=grouped is not markdown, spacers_added=spacers_added, lines_split=unspaced is not grouped, + wikilinks=wikilinks, ) diff --git a/src/telegram_assistant/messages/service.py b/src/telegram_assistant/messages/service.py index 52ba467..2af438a 100644 --- a/src/telegram_assistant/messages/service.py +++ b/src/telegram_assistant/messages/service.py @@ -733,13 +733,17 @@ async def send_message( ) if not request.rich_markdown.strip(): raise ValueError("rich_markdown must be non-empty") - # Bound the *source* before normalising it. Both passes only ever grow - # the text, so a source already over the limit can never come back - # under it — and normalisation is a full line-by-line scan of caller - # input that runs on the event loop, before the WRITE gate. Without - # this, any token holder could hand a multi-megabyte string to - # HTTP/MCP (neither bounds the field) and block the loop for seconds - # on a send it was never authorized to make. + # Bound the *source* before normalising it. This protects the event + # loop ahead of the WRITE gate, and it is deliberately conservative + # rather than exact: grouping, spacing and splitting only ever grow + # the text, but wikilink expansion can shrink it, so a source that is + # over the limit only because of `[[…]]` brackets stripping would + # later remove is still rejected — normalisation as a whole is a full + # line-by-line scan of caller input that runs on the event loop, + # before the WRITE gate. Without this, any token holder could hand a + # multi-megabyte string to HTTP/MCP (neither bounds the field) and + # block the loop for seconds on a send it was never authorized to + # make. if len(request.rich_markdown) > MAX_RICH_MARKDOWN_CHARS: raise ValueError( f"rich_markdown exceeds {MAX_RICH_MARKDOWN_CHARS} characters " @@ -767,6 +771,7 @@ async def send_message( grew_by = [ name for name, applied in ( + ("wikilink expansion", normalization.wikilinks > 0), ("paragraph spacing", normalization.spacers_added), ("line splitting", normalization.lines_split), ("media grouping", normalization.grouped), diff --git a/tests/test_mcp_tools.py b/tests/test_mcp_tools.py index 80d7c7a..f671b40 100644 --- a/tests/test_mcp_tools.py +++ b/tests/test_mcp_tools.py @@ -715,6 +715,32 @@ def test_mcp_send_rich_markdown_reaches_backend( assert backend.sent[0]["text"] == "" +def test_mcp_send_rich_markdown_expands_wikilinks( + minimal_config_yaml: str, tmp_path: Path +) -> None: + """Same proof for the MCP tool surface — expansion is surface-independent.""" + backend = FakeRichMessageBackend() + with _client(minimal_config_yaml, tmp_path, message_backend=backend) as client: + token = _mint_token(client) + _initialize(client, token) + + result = _call_tool( + client, + token, + "telegram_messages_send", + { + "telegram_chat_id": -100123, + "rich_markdown": "Отдал [[Денис Баталин|Дэну]].\n", + "operation_id": "mcp-rich-wikilink", + }, + ) + + assert result["isError"] is False, result + sent = backend.sent[0]["rich_markdown"] + assert "Отдал Дэну." in sent + assert "[[" not in sent + + def test_mcp_send_rich_markdown_never_resolves_a_local_media_path( minimal_config_yaml: str, tmp_path: Path ) -> None: diff --git a/tests/test_messages_rich_spacing.py b/tests/test_messages_rich_spacing.py index 07230da..6da52f4 100644 --- a/tests/test_messages_rich_spacing.py +++ b/tests/test_messages_rich_spacing.py @@ -198,6 +198,42 @@ async def test_send_message_oversize_without_spacing_omits_spacing_note( assert backend.sent == [] +async def test_send_message_length_check_names_wikilink_expansion( + store: OperationStore, +) -> None: + """Wikilink expansion can grow the source: three-or-more ``#`` in a target + turns each into ``" > "`` (+2 chars net) once the surrounding ``[[``/``]]`` + (-4 chars) is removed. A source under the limit whose expansion pushes it + over must name the pass that grew it — not the passes that never ran.""" + # Each unit is net +2 chars after expansion ("[[a#b#c#d]] " -> "a > b > c > d "). + unit = "[[a#b#c#d]] " + count = (MAX_RICH_MARKDOWN_CHARS // len(unit)) - 1 + markdown = unit * count + assert len(markdown) <= MAX_RICH_MARKDOWN_CHARS + backend = RecordingBackend() + + with pytest.raises(ValueError) as excinfo: + await send_message( + backend=backend, + store=store, + request=SendMessageRequest( + telegram_chat_id=-100, + text="", + rich_markdown=markdown, + spaced_paragraphs=False, + line_breaks=False, + media_grouping="none", + operation_id="wikilink-too-long", + ), + ) + + message = str(excinfo.value) + assert str(MAX_RICH_MARKDOWN_CHARS) in message + assert "after wikilink expansion" in message + assert "paragraph spacing" not in message + assert backend.sent == [] + + async def test_send_message_block_limit_fallback_warns_and_still_sends( store: OperationStore, ) -> None: diff --git a/tests/test_messages_rich_spacing_flag.py b/tests/test_messages_rich_spacing_flag.py index b823232..3fc893e 100644 --- a/tests/test_messages_rich_spacing_flag.py +++ b/tests/test_messages_rich_spacing_flag.py @@ -185,6 +185,43 @@ def test_cli_no_spaced_paragraphs_sends_byte_for_byte( assert backend.sent[0]["rich_markdown"] == TWO_PARAGRAPHS +def test_cli_no_spaced_paragraphs_still_expands_wikilinks( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Byte-for-byte via ``--no-spaced-paragraphs`` still expands wikilinks — + there is no knob for that pass. Same proof as the HTTP/MCP surfaces + (``test_http_rich_send_expands_wikilinks``, + ``test_mcp_send_rich_markdown_expands_wikilinks``), pinned here for the + CLI's own ``send_message`` wiring.""" + config_file, md_file, backend = _cli_setup( + tmp_path, + monkeypatch, + config_spaced=True, + markdown="Отдал [[Денис Баталин|Дэну]].\n", + ) + + result = _run_cli( + [ + "messages", + "send", + "--chat-id", + "-100", + "--rich-markdown", + str(md_file), + "--no-spaced-paragraphs", + "--operation-id", + "cli-flag-off-wikilink", + "--config", + str(config_file), + ] + ) + + assert result.exit_code == 0, _cli_output(result) + sent = backend.sent[0]["rich_markdown"] + assert sent == "Отдал Дэну.\n" + assert "[[" not in sent + + def test_cli_spaced_paragraphs_overrides_config_off( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -331,6 +368,58 @@ def test_cli_dry_run_plain_send_has_no_spacing_markers( assert resolved["rich_markdown_media"] is None +def test_cli_dry_run_reports_expanded_wikilinks( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + config_file, md_file, backend = _cli_setup( + tmp_path, monkeypatch, markdown="Отдал [[Денис Баталин|Дэну]].\n" + ) + + result = _run_cli( + [ + "messages", + "send", + "--chat-id", + "-100", + "--rich-markdown", + str(md_file), + "--dry-run", + "--config", + str(config_file), + ] + ) + + assert result.exit_code == 0, _cli_output(result) + resolved = json.loads(result.stdout.strip().splitlines()[-1])["resolved"] + assert resolved["rich_markdown_wikilinks"] == 1 + assert backend.sent == [] + + +def test_cli_dry_run_plain_send_has_no_wikilink_marker( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The marker shape must never depend on the mode.""" + config_file, _md_file, _backend = _cli_setup(tmp_path, monkeypatch) + + result = _run_cli( + [ + "messages", + "send", + "--chat-id", + "-100", + "--text", + "hello", + "--dry-run", + "--config", + str(config_file), + ] + ) + + assert result.exit_code == 0, _cli_output(result) + resolved = json.loads(result.stdout.strip().splitlines()[-1])["resolved"] + assert resolved["rich_markdown_wikilinks"] is None + + def test_cli_dry_run_reports_block_limit_warnings( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: diff --git a/tests/test_messages_rich_surfaces.py b/tests/test_messages_rich_surfaces.py index c457b7a..b71c52b 100644 --- a/tests/test_messages_rich_surfaces.py +++ b/tests/test_messages_rich_surfaces.py @@ -153,6 +153,25 @@ def test_http_rich_send_passes_markdown_to_backend() -> None: assert backend.sent[0]["text"] == "" +def test_http_rich_send_expands_wikilinks() -> None: + """The pass is not CLI-only: an agent relaying note text has the same defect.""" + backend = RecordingMessageBackend() + client = _client(backend) + resp = client.post( + "/telegram/messages", + json={ + "telegram_chat_id": -100, + "rich_markdown": "Спросил у [[Станислав Попов|Стаса]].\n", + "operation_id": "rich-http-wikilink", + }, + headers=AUTH, + ) + assert resp.status_code == 200, resp.text + sent = backend.sent[0]["rich_markdown"] + assert "Спросил у Стаса." in sent + assert "[[" not in sent + + def test_http_rich_send_allows_topic_reply_and_schedule() -> None: backend = RecordingMessageBackend() client = _client(backend) diff --git a/tests/test_rich_markdown_normalize.py b/tests/test_rich_markdown_normalize.py index c8ad4b1..7536faa 100644 --- a/tests/test_rich_markdown_normalize.py +++ b/tests/test_rich_markdown_normalize.py @@ -325,3 +325,25 @@ def test_media_over_the_limit_only_warns() -> None: def test_media_exactly_at_the_limit_does_not_warn() -> None: source = "\n\n".join(f"![](https://x/{n}.jpg)" for n in range(MAX_RICH_MEDIA)) assert normalize_rich_markdown(source).warnings == () + + +def test_normalize_expands_wikilinks_and_counts_them() -> None: + result = normalize_rich_markdown("Отдал [[Денис Баталин|Дэну]].\n") + assert "[[" not in result.markdown + assert "Дэну" in result.markdown + assert result.wikilinks == 1 + + +def test_normalize_reports_zero_wikilinks_when_there_are_none() -> None: + assert normalize_rich_markdown("Просто текст.\n").wikilinks == 0 + + +def test_wikilinks_are_expanded_before_blocks_are_counted() -> None: + """The pass edits inside lines, so blocks must be counted after it. + + A wikilink's own pipe splits a table cell; expanding it first is what makes + the reported block structure the one Telegram actually receives. + """ + source = "| a | b |\n| --- | --- |\n| [[Станислав Попов|Стас]] | да |\n" + result = normalize_rich_markdown(source) + assert "| Стас | да |" in result.markdown diff --git a/tests/test_rich_markdown_wikilinks.py b/tests/test_rich_markdown_wikilinks.py new file mode 100644 index 0000000..7e48ea4 --- /dev/null +++ b/tests/test_rich_markdown_wikilinks.py @@ -0,0 +1,166 @@ +"""The Obsidian wikilink pass: [[Target|Alias]] becomes plain text. + +Telegram has no wikilink syntax, so an unexpanded link reaches the reader as +literal brackets plus the vault's canonical note name instead of the word the +author wrote. +""" + +import pytest + +from telegram_assistant.messages.rich_markdown import MAX_WIKILINK_PASSES, strip_wikilinks + + +@pytest.mark.parametrize( + ("source", "expected"), + [ + ("[[Андрей Смирнов]]", "Андрей Смирнов"), + ("[[Станислав Попов|Стасу]]", "Стасу"), + ("[[#Спорные моменты]]", "Спорные моменты"), + ("[[tasks#Настроить statusline]]", "tasks > Настроить statusline"), + ("[[Note#Heading|Алиас]]", "Алиас"), + # Only the first pipe separates; the rest belong to the alias. + ("[[A|B|C]]", "B|C"), + # An empty half falls back to the other one. + ("[[Note|]]", "Note"), + ("[[|Стас]]", "Стас"), + # A block reference falls out of the same rule with no special case. + ("[[note#^blk]]", "note > ^blk"), + # Surrounding prose is untouched, and several links share one line. + ( + "Отдал [[Денис Баталин|Дэну]] и [[Ирина Шлыкова|Ирине]].", + "Отдал Дэну и Ирине.", + ), + ], +) +def test_expands_wikilinks(source: str, expected: str) -> None: + assert strip_wikilinks(source) == (expected, source.count("[[")) + + +@pytest.mark.parametrize("source", ["[[]]", "[[|]]", "[[#]]"]) +def test_degenerate_link_is_left_verbatim(source: str) -> None: + """No target and no alias is not a link. + + Collapsing it to an empty string would silently delete characters the + author typed — worse than leaving a curiosity in the text. + """ + assert strip_wikilinks(source) == (source, 0) + + +def test_media_embed_is_left_to_scan_media() -> None: + """``![[…]]`` is media; ``scan_media`` rewrites it into a ``tg://`` ref.""" + source = "![[Pasted image 1.png|Закат]]" + assert strip_wikilinks(source) == (source, 0) + + +def test_embed_and_link_on_one_line() -> None: + source = "![[shot.png]] обсудили с [[Ольга Цветцых]]" + assert strip_wikilinks(source) == ("![[shot.png]] обсудили с Ольга Цветцых", 1) + + +def test_inline_code_span_is_opaque() -> None: + """An article documenting this dialect writes `[[Note]]` and means the text.""" + source = "Пиши `[[Note]]`, получишь Note." + assert strip_wikilinks(source) == (source, 0) + + +def test_code_span_overlapping_a_link_does_not_shield_it() -> None: + """Containment, not overlap — the rule ``iter_line_media_refs`` uses. + + The code span here starts inside the link and ends outside it. Skipping on + overlap would ship the raw brackets. + """ + source = "[[Note|запусти `make]] сначала`" + text, count = strip_wikilinks(source) + assert count == 1 + assert "[[" not in text + + +def test_fenced_code_block_is_opaque() -> None: + source = "```\n[[Note]]\n```\n" + assert strip_wikilinks(source) == (source, 0) + + +def test_fenced_code_inside_a_quote_is_opaque() -> None: + source = "> ```\n> [[Note]]\n> ```\n" + assert strip_wikilinks(source) == (source, 0) + + +def test_link_inside_a_quote_is_expanded() -> None: + source = "> Сказал [[Андрей Смирнов|Андрей]]\n" + assert strip_wikilinks(source) == ("> Сказал Андрей\n", 1) + + +def test_table_cell_pipe_no_longer_breaks_the_row() -> None: + """``[[A|B]]`` in a table currently splits the cell on its own pipe.""" + source = "| [[Станислав Попов|Стас]] | да |\n" + assert strip_wikilinks(source) == ("| Стас | да |\n", 1) + + +def test_returns_input_by_identity_when_unchanged() -> None: + """Identity is what keeps CRLF and the trailing newline byte-for-byte.""" + source = "# Заголовок\r\n\r\nПростой текст.\r\n" + text, count = strip_wikilinks(source) + assert text is source + assert count == 0 + + +def test_trailing_newline_survives_a_rewrite() -> None: + source = "Спросил у [[Ольга Андрющенко|Оли]].\n" + assert strip_wikilinks(source) == ("Спросил у Оли.\n", 1) + + +def test_is_idempotent() -> None: + source = "Отдал [[Денис Баталин|Дэну]].\n" + once, first = strip_wikilinks(source) + twice, second = strip_wikilinks(once) + assert first == 1 + assert second == 0 + assert twice is once + + +@pytest.mark.parametrize( + ("source", "expected"), + [ + # A wikilink nested inside another link's target. + ("[[ [[a]] ]]", "a"), + # A wikilink nested inside another link's alias. + ("[[a|[[b]]]]", "b"), + # A wikilink whose target is itself bracketed twice over. + ("[[[[a]]]]", "a"), + ], +) +def test_nested_wikilinks_expand_to_a_fixpoint(source: str, expected: str) -> None: + """One pass only resolves the innermost link, leaving ``[[``/``]]`` + behind — literal double brackets in a Telegram article are exactly the + defect this pass exists to eliminate, so a single call must expand all + the way through.""" + text, count = strip_wikilinks(source) + assert text == expected + assert "[[" not in text + assert count == source.count("[[") + + +def test_nesting_past_the_cap_stops_and_ships_the_remainder_verbatim() -> None: + """Structural proof of the ``MAX_WIKILINK_PASSES`` bound (no wall clock): + an unrealistically deep chain of nested links resolves exactly one level + per pass (verified by the three-input fixpoint tests above), so a depth + well past the cap must stop after exactly ``MAX_WIKILINK_PASSES`` passes + and leave ``[[``/``]]`` behind — the same trade-off ``_scan_nested`` makes + past ``MAX_BLOCK_NESTING``. This is what keeps the call bounded: a real + fixpoint loop over this input would need one pass per nesting level. + """ + depth = MAX_WIKILINK_PASSES * 3 + source = "[[" * depth + "a" + "]]" * depth + + text, count = strip_wikilinks(source) + + # Each pass unwraps exactly one level (4 chars: the removed "[[" + "]]"), + # so the number of passes actually run is recoverable from the length + # delta — and it must be capped, not `depth`. + levels_resolved = (len(source) - len(text)) // 4 + assert levels_resolved == MAX_WIKILINK_PASSES + assert count == MAX_WIKILINK_PASSES + assert "[[" in text + assert text == "[[" * (depth - MAX_WIKILINK_PASSES) + "a" + "]]" * ( + depth - MAX_WIKILINK_PASSES + )