init mlmdx

dune-project

@@ -15,6 +15,29 @@
 
 (documentation https://url/to/documentation)
 
+; The mlx dialect: .mlx files are OCaml + JSX syntax, lowered to plain OCaml
+; parsetree (carrying [@JSX]) by mlx-pp, then [@JSX] is lowered per-library by
+; (preprocess (pps html_of_jsx.ppx)). merlin_reader points at ocamlmerlin-mlx.
+; mlmdx (pkg/mlmdx) reuses this chain to render .mlx components embedded in pages.
+(dialect
+ (name mlx)
+ (implementation
+  (extension mlx)
+  (merlin_reader mlx)
+  (preprocess
+   (run mlx-pp %{input-file}))))
+
+; The mlmdx dialect: .mlmdx files are prose-first Markdown pages. mlmdx-pp emits
+; a plain-OCaml parsetree (structural nodes as JSX.node calls; embedded {expr}
+; parsed with the real OCaml parser). ocamlmerlin-mlmdx shares the same codegen.
+(dialect
+ (name mlmdx)
+ (implementation
+  (extension mlmdx)
+  (merlin_reader mlmdx)
+  (preprocess
+   (run mlmdx-pp %{input-file}))))
+
 (package
  (name oystermark-core)
  (synopsis "Oystermark parser: CommonMark and various extensions")
@@ -44,6 +67,7 @@
    (>= 3.22))
   oystermark-core
   oymarkit
+  merlin-extend
   core
   core_unix
   tyxml

oystermark.opam

@@ -14,6 +14,7 @@ depends: [
   "dune" {>= "3.22"}
   "oystermark-core"
   "oymarkit"
+  "merlin-extend"
   "core"
   "core_unix"
   "tyxml"

pkg/mlmdx/README.md

@@ -0,0 +1,113 @@
+# mlmdx
+
+MDX for OCaml: prose-first `.mlmdx` pages with embedded, **type-checked** OCaml
+expressions and components, rendered to HTML at build time (static site
+generation).
+
+A `.mlmdx` file compiles to an ordinary OCaml module exposing a `make` function —
+the analog of MDX compiling Markdown to a JS component module. Because the target
+is OCaml, everything embedded in a page is type-checked as part of the normal
+build: a component used with the wrong prop type is a compile error, not a
+runtime surprise. (Not to be confused with `ocaml-mdx`, which *executes* code
+blocks in documentation — mlmdx is the opposite direction: markup embedded in a
+program, compiled and rendered.)
+
+## The pipeline
+
+```
+.mlmdx
+  │  oymarkit.cmarkit  (parser only; native Ext nodes for {expr}, inline JSX,
+  │                     and block JSX, storing raw source + Textloc)
+  ▼
+Cmarkit.Doc.t
+  │  mlmdx-pp  ── structural nodes ────▶ components table (components.h1 …;
+  │                                        default = plain JSX.node)
+  │            ── {expr} / prop values ─▶ compiler-libs Parse.expression
+  │                                        on a position-primed lexbuf
+  │            ── host JSX / fragments ─▶ JSX.node / JSX.list
+  │            ── component JSX ────────▶ Component.make calls
+  ▼
+Parsetree.structure   (plain OCaml, [@JSX]-attributed, ghost-located wrappers)
+  │  html_of_jsx.ppx   (per-library, lowers [@JSX])
+  ▼
+plain OCaml module  ──  JSX.render  ──▶  HTML string
+```
+
+Two thin binaries — `mlmdx-pp` (the dialect preprocessor) and `ocamlmerlin-mlmdx`
+(the merlin reader) — share one `Codegen` core; they differ only in the output
+wrapper.
+
+## Dependencies and scope
+
+mlmdx depends on the **parser** half of `oymarkit` (a cmarkit fork we own) plus
+`compiler-libs` and the `html_of_jsx` runtime — and explicitly **not** on
+`pkg/oystermark` (the vault: wikilinks, embeds, Tyxml renderer). Different
+product, same underlying Markdown parser.
+
+- **`oymarkit.cmarkit`** — parses `.mlmdx` prose to a `Cmarkit.Doc.t`. Native
+  extension nodes (`Inline.Ext_jsx_expr`, `Inline.Ext_jsx_element`,
+  `Block.Ext_jsx_block`, gated behind `?jsx_expr` / `?jsx_element` on
+  `Doc.of_string`) capture raw source plus `Textloc` and do no OCaml parsing — no
+  compiler-libs leaks into oymarkit.
+- **`compiler-libs`** — `mlmdx-pp` runs `Parse.expression` on the leaf OCaml that
+  authors write (`{expr}` bodies and component prop values). The load-bearing
+  discipline: prime the lexbuf's absolute position from the stored `Textloc`, so
+  type errors and hovers land on the right byte of the `.mlmdx`.
+- **`mlx`** — used *stock*, for hand-written `.mlx` component files only, via the
+  dune dialect declared in the root `dune-project`. `mlmdx-pp` does **not** link
+  or invoke mlx; it reparses the raw JSX tags oymarkit identified and builds the
+  `[@JSX]` parsetree directly. (mlx exposes no callable parser library — only its
+  `mlx-pp` / `ocamlmerlin-mlx` binaries.)
+- **`html_of_jsx`** — `html_of_jsx.ppx` lowers `[@JSX]`-attributed parsetree
+  (from both `mlx-pp` on `.mlx` and `mlmdx-pp` on `.mlmdx`) into
+  `JSX.node`/component-`make` calls; `JSX.render` produces the HTML string. This
+  is the whole SSG runtime — inert HTML, no client JS in the toolchain.
+- **`mlmdx` (runtime, `runtime/`)** — exposes `Mlmdx.Components`, the overridable
+  components table that generated `.mlmdx` modules route markdown-structural
+  elements through. Every generated `.mlmdx` module depends on it.
+- **`merlin-extend`** — `ocamlmerlin-mlmdx`, a reader wrapping the same codegen,
+  gives `.mlmdx` files hovers and jumps; dispatch is per-extension via dune.
+
+## Status
+
+The full `.mlmdx` → HTML chain is working end-to-end:
+
+- **`lib/codegen.ml`** (`mlmdx_codegen`): `Doc.t → Parsetree.structure` exposing
+  `let make ?components () = <element>`. Structural nodes → the components table
+  (`components.h1 ~children:[…]`); `{expr}` and JSX prop expressions →
+  `Parse.expression` on a position-primed lexbuf; host JSX → `JSX.node`,
+  fragments → `JSX.list`, component JSX → `Component.make`. Covered by inline
+  expect tests (`ppx_expect`) that pin the generated parsetree.
+- **Components table** (the overridable `_components` map — what makes this *MDX*,
+  not markdown-to-HTML): every markdown-structural element routes through the
+  `?components` parameter of `make` (`# Hi` → `components.h1 ~children:[…]`), so a
+  consumer can restyle or swap any element page-wide via a record-`with`. The
+  default table renders vanilla elements, so a page rendered without a custom
+  table is byte-for-byte identical to plain markdown-to-HTML — the table is
+  always present, its default is the identity. Literal JSX and `<Component/>`
+  calls in the page do *not* route through it (author intent, like raw JSX in
+  MDX). Type `Mlmdx.Components.t` lives in the `runtime/` library.
+- **Strict prelude**: top-of-file `open`/`let`/`module` blocks are parsed as OCaml
+  structure items before the generated `make`; the first Markdown block switches
+  permanently to Markdown.
+- **`pp/mlmdx_pp.ml`** (`mlmdx-pp`): the dialect preprocessor, emitting the binary
+  `-pp` AST protocol (magic number + filename + structure). Registered as the
+  `mlmdx` dialect in the root `dune-project`.
+- **`pp/ocamlmerlin_mlmdx.ml`** (`ocamlmerlin-mlmdx`): a merlin-extend reader over
+  the same codegen core, wired via `(merlin_reader mlmdx)`.
+- **`examples/hello/`**: the full-chain example (see its README), pinned by a cram
+  test on the rendered HTML.
+
+Key property proven: a type error inside `{expr}` (e.g. `{JSX.int "x"}`) reports
+against the `.mlmdx` file at the exact byte of the offending code, quoting the
+`.mlmdx` source line — the position-priming works.
+
+```
+dune exec pkg/mlmdx/examples/hello/render.exe
+# <h1>4</h1><p>Some <strong>bold</strong> prose and an inline value: 42.</p>...
+# <h1 class="title">4</h1>...   (second line: same page with an overridden h1)
+```
+
+### Not yet done
+
+JSX inside embedded `{expr}`; client interactivity / hydration; a `.mlmdx` formatter

pkg/mlmdx/examples/hello/README.md

@@ -0,0 +1,49 @@
+# hello
+
+The end-to-end mlmdx example: one prose-first `.mlmdx` page that embeds OCaml
+expressions and instantiates typed `.mlx` components, rendered to an HTML string
+at build time.
+
+## Files
+
+- `hello.mlmdx` — the page. A top-of-file `let panel_title = "Panel"` prelude,
+  then Markdown prose carrying every embedded construct: `{expr}` spans
+  (`{JSX.int (2 + 2)}`), self-closing and container component calls
+  (`<Greeting .../>`, `<Panel>...</Panel>`), and raw host JSX (`<b class="loud">`,
+  `<div className="box">`) with Markdown nested inside.
+- `greeting.mlx` — a typed component, `~name:string ~count:int`. Because it is
+  lowered to plain OCaml before type-checking, `<Greeting count={1 + 1} />` in the
+  page type-checks the prop at the call site; passing a string would be a compile
+  error pointing at the exact byte.
+- `panel.mlx` — a container component taking `~children`.
+- `render.ml` — plain OCaml driving the page. Renders it twice: once with the
+  default components table (`JSX.render (Hello.make ())`) and once with an
+  overridden `h1` (`~components:{ Mlmdx.Components.default with h1 = … }`), to show
+  that markdown headings route through the overridable table while literal JSX and
+  `<Component/>` calls do not.
+- `render.t` — a cram test pinning both rendered lines (the end-to-end regression
+  check).
+
+## Run
+
+```
+dune exec pkg/mlmdx/examples/hello/render.exe
+```
+
+```
+<h1>4</h1><p>Some <strong>bold</strong> prose and an inline value: 42.</p>...
+<h1 class="title">4</h1>...   (same page, h1 overridden via the components table)
+```
+
+## The chain
+
+```
+hello.mlmdx  ──mlmdx-pp (dialect)──▶  Parsetree ([@JSX])  ──html_of_jsx.ppx──▶  OCaml
+   ▲ Greeting/Panel (.mlx, via mlx-pp)                                    │
+   └──────────────────────────────────────────────  Hello.make ()  ──JSX.render──▶ HTML
+```
+
+`mlmdx-pp` lowers the `.mlmdx`; `mlx-pp` lowers the hand-written `.mlx`
+components; both emit `[@JSX]`-attributed parsetree that `html_of_jsx.ppx`
+turns into `JSX.node`/component-`make` calls. Everything collapses to one
+plain-OCaml module graph that type-checks together.

pkg/mlmdx/examples/hello/dune

@@ -0,0 +1,14 @@
+; Full-chain example: hello.mlmdx is preprocessed by the mlmdx dialect (mlmdx-pp)
+; into module Hello, then render.ml renders it to an HTML string.
+(executable
+ (name render)
+ (modules render hello greeting panel)
+ (libraries html_of_jsx mlmdx)
+ (preprocess
+  (pps html_of_jsx.ppx)))
+
+; Cram test pinning the end-to-end render output. This is the one path the
+; codegen inline expect tests don't cover: mlmdx-pp -> html_of_jsx.ppx ->
+; JSX.render -> HTML string.
+(cram
+ (deps %{exe:render.exe}))

pkg/mlmdx/examples/hello/greeting.mlx

@@ -0,0 +1,8 @@
+(* A typed component authored in .mlx, used from hello.mlmdx as
+   <Greeting name="World" count={...} />. Props are type-checked at the call
+   site in the .mlmdx. *)
+let make ~name ~count () =
+  <span class_="greeting">
+    (JSX.string ("Hello, " ^ name ^ "! x"))
+    (JSX.int count)
+  </span>

pkg/mlmdx/examples/hello/hello.mlmdx

@@ -0,0 +1,25 @@
+let panel_title = "Panel"
+
+# {JSX.int (2 + 2)}
+
+Some **bold** prose and an inline value: {JSX.int (6 * 7)}.
+
+A component inline: <Greeting name="inline" count={1 + 1} /> — and standalone:
+
+<Greeting name="World" count={3 * 7} />
+
+Host JSX inline: <b class="loud">rendered **bold**</b>.
+
+<div className="box">
+
+## Host block
+
+Markdown **inside** a host block.
+
+</div>
+
+<Panel title={panel_title}>
+
+Panel **children** with {JSX.string "expr"}.
+
+</Panel>

pkg/mlmdx/examples/hello/panel.mlx

@@ -0,0 +1,5 @@
+let make ~title ~children () =
+  <section class_="panel">
+    <h2>(JSX.string title)</h2>
+    (children)
+  </section>

pkg/mlmdx/examples/hello/render.ml

@@ -0,0 +1,17 @@
+(* Plain OCaml driving a .mlmdx page: proves the full chain
+   .mlmdx -> mlmdx-pp -> html_of_jsx.ppx -> JSX.render -> HTML string. *)
+
+(* Default: no components table supplied, so every markdown element renders as a
+   vanilla HTML node. *)
+let () = print_endline (JSX.render (Hello.make ()))
+
+(* Overridden: swap the h1 renderer page-wide via the components table. Only the
+   markdown heading (# ...) is affected; literal JSX and <Component/> calls in
+   the page are untouched. This is what makes mlmdx MDX, not markdown-to-HTML. *)
+let () =
+  let components =
+    { Mlmdx.Components.default with
+      h1 = (fun ~children -> JSX.node "h1" [ "class", `String "title" ] children)
+    }
+  in
+  print_endline (JSX.render (Hello.make ~components ()))

pkg/mlmdx/examples/hello/render.t

@@ -0,0 +1,6 @@
+Render hello.mlmdx through the full chain
+(.mlmdx -> mlmdx-pp -> html_of_jsx.ppx -> JSX.render -> HTML):
+
+  $ ./render.exe
+  <h1>4</h1><p>Some <strong>bold</strong> prose and an inline value: 42.</p><p>A component inline: <span class="greeting">Hello, inline! x2</span> — and standalone:</p><p><span class="greeting">Hello, World! x21</span></p><p>Host JSX inline: <b class="loud">rendered <strong>bold</strong></b>.</p><div class="box"><h2>Host block</h2><p>Markdown <strong>inside</strong> a host block.</p></div><section class="panel"><h2>Panel</h2><p>Panel <strong>children</strong> with expr.</p></section>
+  <h1 class="title">4</h1><p>Some <strong>bold</strong> prose and an inline value: 42.</p><p>A component inline: <span class="greeting">Hello, inline! x2</span> — and standalone:</p><p><span class="greeting">Hello, World! x21</span></p><p>Host JSX inline: <b class="loud">rendered <strong>bold</strong></b>.</p><div class="box"><h2>Host block</h2><p>Markdown <strong>inside</strong> a host block.</p></div><section class="panel"><h2>Panel</h2><p>Panel <strong>children</strong> with expr.</p></section>