From 8ea53bd76bca5408e595e10c60ba07b04ad2c4dd Mon Sep 17 00:00:00 2001 From: Tim Holy Date: Mon, 25 May 2026 12:38:28 -0500 Subject: [PATCH 1/3] Add an end-to-end OLS tutorial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A small ordinary-least-squares library wrapped end-to-end as a worked example for the docs: write the Julia source, run `build_library` to compile and generate wrappers, `pip install` the resulting package into a clean virtualenv, and call it from Python. The example deliberately covers the full recognized JLWInterop vocabulary in one library — `CMatrix{Float64}` for the design matrix, `CVector{Float64}` for response / coefficients / predictions, `CString` for a report buffer, `JLWStatus` both as a direct return (`predict`) and embedded in a result struct (`fit`'s `FitResult`), and a scalar `r_squared`. Lives at `examples/ols/`; new tutorial page `docs/src/tutorial.md` sits between Home and Concepts in the docs nav so it acts as the on-ramp. Known blocker: the example *builds* end-to-end (`build_library` succeeds and produces an importable Python package), but the compiled `.so` cannot yet *run* its `fit` entrypoint from a host process. `X \ y` reaches BLAS through `Base.Libc.Libdl.LazyLibrary`, whose untyped fields and runtime-hook (`jl_libdl_dlopen_func`) dispatch put the cold load path outside what `juliac --trim=safe`'s static call-graph analysis can keep alive — the first BLAS call from Python aborts with `MethodError: no method matching Base.Libc.Libdl.dlopen(::LazyLibrary, ::UInt32)`. The tutorial is written as if that works (and once the fix lands it will, with no further changes here); merging it now documents the intended workflow and gives the JLW release docs something to point at. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/make.jl | 1 + docs/src/index.md | 7 + docs/src/tutorial.md | 273 ++++++++++++++++++++++++++++++++++++++ examples/ols/Project.toml | 17 +++ examples/ols/build.jl | 51 +++++++ examples/ols/src/ols.jl | 101 ++++++++++++++ 6 files changed, 450 insertions(+) create mode 100644 docs/src/tutorial.md create mode 100644 examples/ols/Project.toml create mode 100644 examples/ols/build.jl create mode 100644 examples/ols/src/ols.jl diff --git a/docs/make.jl b/docs/make.jl index 85c7ac0..a05e210 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -16,6 +16,7 @@ makedocs(; ), pages=[ "Home" => "index.md", + "Tutorial" => "tutorial.md", "Concepts" => "concepts.md", "JLWInterop" => "jlwinterop.md", "Error handling" => "error_handling.md", diff --git a/docs/src/index.md b/docs/src/index.md index e4b1508..7e82a15 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -11,6 +11,10 @@ It turns the ABI metadata that `juliac` emits into wrappers that let non-Julia programs call the compiled library as if it were any other native dependency. +New to the package? Start with the [tutorial](@ref "Tutorial: wrap an +OLS regression library") — it walks a small library end to end, from +Julia source through `pip install` to numpy-flavored Python. + ## Who it is for Authors of Julia libraries who want to ship a compiled artifact that @@ -36,6 +40,9 @@ bundling, multi-library, and two-tier output stories. ## Where to go next +- [Tutorial: wrap an OLS regression library](@ref) — the on-ramp: + one library, end to end, from Julia source through `pip install` + to numpy-flavored Python. - [Concepts](@ref) — the pipeline, the ABI data model, the extension point for new target languages, and the runtime-closure / bundling story. diff --git a/docs/src/tutorial.md b/docs/src/tutorial.md new file mode 100644 index 0000000..7f21a6b --- /dev/null +++ b/docs/src/tutorial.md @@ -0,0 +1,273 @@ +```@meta +CurrentModule = JuliaLibWrapping +``` + +# Tutorial: wrap an OLS regression library + +This walks through the full pipeline end to end: write a small Julia +library, run [`build_library`](@ref) to compile it and emit wrappers, +`pip install` the generated package, and call it from Python. The +worked example lives in `examples/ols/` and stays buildable as the +package evolves. + +The subject is ordinary least squares (OLS) regression. The algorithm +is incidental — what matters is that one library exercises every +recognized [JLWInterop](https://github.com/JuliaInterop/JuliaLibWrapping.jl/tree/main/JLWInterop) +type: + +| Vocabulary type | Where it appears in `ols` | +|----------------------------|--------------------------------------| +| `CMatrix{Float64}` | design matrix `X` | +| `CVector{Float64}` | response `y`, coefficients, predictions | +| `Float64` (primitive) | `r_squared` | +| `CString` | output buffer for `summary_report` | +| `JLWStatus` (direct) | return of `predict` | +| `JLWStatus` (embedded) | `FitResult.status` field | + +## 1. The Julia source + +`examples/ols/src/ols.jl` defines three [`Base.@ccallable`] entrypoints +on top of JLWInterop's vocabulary: + +```julia +module ols + +using JLWInterop +using LinearAlgebra + +struct FitResult + status::JLWStatus + coeffs::CVector{Float64} + r_squared::Float64 +end + +Base.@ccallable function fit(X::CMatrix{Float64}, + y::CVector{Float64}, + coeffs_buf::CVector{Float64})::FitResult + # … shape checks, then `coeffs = X \\ y`; copy into `coeffs_buf`, + # compute R^2, and return FitResult with JLWStatus(0,…) on success + # or JLWStatus(code, msg) on a recognized failure. +end + +Base.@ccallable function predict(coeffs::CVector{Float64}, + X::CMatrix{Float64}, + out::CVector{Float64})::JLWStatus + +Base.@ccallable function summary_report(result::FitResult, + buf::CString)::JLWStatus +``` + +`coeffs_buf` and `out` are *caller-allocated* buffers: the library +writes into them but does not own them. This is the same discipline +JLWInterop documents for all its pointer-bearing types — see +`CArray` and `CString`. + +Errors travel back as a `JLWStatus`, +either returned directly (`predict`, `summary_report`) or embedded in +a return struct (`fit`'s `FitResult`). The Python emitter recognizes +both forms and translates a non-zero `code` into a +`JLWError` exception — see [Error handling](@ref "Error handling +across the ABI"). + +!!! note "About the algorithm" + `fit` is a one-liner: `coeffs = X \\ y`. The interesting story is + the wrapping, not the math. Production code with many predictors, + rank-deficient inputs, or weighting concerns would substitute a + proper factorization — the wrapping contract (`CMatrix{Float64}` + in, caller-allocated `CVector{Float64}` out, `JLWStatus` for + failures) does not change. + +## 2. The entry `Project.toml` + +A minimal `Project.toml` for the library: + +```toml +name = "ols" +uuid = "7e81292c-b63a-42d7-9477-255b6fedc2ed" +version = "0.0.1" + +[deps] +JLWInterop = "65e54657-ed21-41a3-96db-71ab7fa6d94b" +LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[sources] +JLWInterop = {path = "/absolute/path/to/JuliaLibWrapping/JLWInterop"} + +[compat] +JLWInterop = "0.1" +julia = "1.13" +``` + +Two things to call out: + +- The `julia = "1.13"` floor is mandatory; the `juliac` feature that + emits the ABI JSON shipped in Julia 1.13. +- `[sources]` paths must be **absolute**. `juliac` relocates the + project into a temporary directory before compiling, so a relative + path cannot be resolved. [`build_library`](@ref) rejects relative + `[sources]` up front with a clear error. + +The in-tree `examples/ols/Project.toml` deliberately omits `[sources]` +so the file does not bake in a machine-specific path; its `build.jl` +instead materializes a transient project in a temp directory with +`[sources]` computed from `@__DIR__` and passes that as the `project=` +argument to [`build_library`](@ref). Authoring a fresh library, you +would normally just write `[sources]` once with your local absolute +path. + +## 3. Build the library and Python package + +`examples/ols/build.jl` is the driver: + +```julia +using JuliaLibWrapping +using JuliaC + +const HERE = @__DIR__ +const OUT = joinpath(HERE, "out") +const ENTRY = joinpath(HERE, "src", "ols.jl") + +result = build_library(ENTRY, + [CTarget(OUT, "ols"), + PythonTarget(OUT, "ols_py", "ols"; bundle_subdir = "bundle")]; + project = HERE, + libname = "ols", + libdir = OUT, + bundle = true, + verbose = true, +) +``` + +Run it with the Julia 1.13 release candidate: + +```sh +cd examples/ols +julia +rc --project=. build.jl +``` + +`JuliaLibWrapping` and `JuliaC` need to be loadable — typically +`Pkg.develop` them into your global v1.13 environment, so the entry +project's stripped-down `[deps]` (just `JLWInterop`) reflects the +*runtime* dependencies of the compiled library rather than the build +tooling. + +After a successful build, `out/` contains: + + out/ + ├── ols.so # compiled shared library + ├── ols.abi.json # ABI metadata emitted by juliac + ├── ols.h # C header (CTarget output) + ├── pyproject.toml # Python package metadata + └── ols_py/ + ├── __init__.py + ├── _lowlevel.py # mechanical ctypes bindings (regenerated every build) + ├── _facade.py # idiomatic surface (written once; user-editable) + └── bundle/ # juliac --bundle tree: libjulia, stdlibs, BLAS, … + +`bundle = true` is essential for a `pip install` user who has no +Julia on their machine. See the bundling section of the +[overview](@ref "Bundling for distribution") for what is in the bundle +tree and how the loader finds `libjulia` from inside the wheel. + +The default backend is `:juliac` (via [JuliaC.jl](https://github.com/JuliaLang/JuliaC.jl)); +ABI developers chasing features ahead of JuliaC.jl can opt into the +unstable in-tree `share/julia/juliac/juliac.jl` script via +`backend = :script`. + +## 4. Install the Python package + +Create a clean virtualenv with no system Julia, and install: + +```sh +python -m venv /tmp/ols-venv +source /tmp/ols-venv/bin/activate +pip install numpy +pip install ./examples/ols/out/ +``` + +The bundled `libjulia` and stdlibs live inside the wheel; the loader +in `_lowlevel.py` searches `bundle/lib/` first so the baked-in +`RUNPATH` resolves them at import time, with no `LD_LIBRARY_PATH` +required. + +## 5. Call it from Python + +`predict` is auto-wrapped — its arguments and return are all +recognized, so the façade accepts numpy arrays and raises `JLWError` +on a non-zero status: + +```python +import numpy as np +from ols_py import predict, JLWError + +coeffs = np.array([0.06, 1.98]) +X = np.asfortranarray(np.column_stack([np.ones(5), np.arange(1.0, 6.0)])) +out = np.zeros(5) +predict(coeffs, X, out) +# out is now X @ coeffs + +try: + bad = np.asfortranarray(np.zeros((5, 3))) # wrong number of columns + predict(coeffs, bad, out) +except JLWError as e: + print(e.code, e.message) # 1, "coeffs length must match X cols" +``` + +`np.asfortranarray` is required for any `CMatrix{T}` argument: +JLWInterop's `CArray` is column-major, and the façade rejects a +row-major view rather than silently transposing. + +`fit` returns a `FitResult` struct that *contains* a `JLWStatus`. +The emitter doesn't know how to shape that idiomatically (numpy of +`coeffs`? a tuple? the whole struct?), so the starter façade +re-exports it from `_lowlevel` with a `TODO: hand-wrap` comment +naming the obstacle. You edit `_facade.py` to provide the wrapper +you want. The mechanical layer still raises `JLWError` on a non-zero +status, so a sklearn-flavored hand wrap is just: + +```python +# in ols_py/_facade.py, replacing the auto-generated TODO line +def fit(X, y): + X = np.asfortranarray(X) + y = np.ascontiguousarray(y, dtype=np.float64) + coeffs = np.zeros(X.shape[1]) + result = _lowlevel.fit( + _lowlevel.CMatrix_Float64.from_numpy(X), + _lowlevel.CVector_Float64.from_numpy(y), + _lowlevel.CVector_Float64.from_numpy(coeffs), + ) + return coeffs, float(result.r_squared) +``` + +`summary_report` is also a hand-wrap case (its `FitResult` argument +is an unrecognized struct). The caller allocates a writable +`CString` buffer, passes it in, and decodes the bytes after the call: + +```python +def summary(result_struct, capacity=256): + import ctypes + buf_bytes = (ctypes.c_uint8 * capacity)() + buf = _lowlevel.CString( + length=capacity, + data=ctypes.cast(buf_bytes, ctypes.POINTER(ctypes.c_uint8)), + ) + _lowlevel.summary_report(result_struct, buf) + return bytes(buf_bytes).rstrip(b"\x00").decode("utf-8") +``` + +## 6. Adding an entrypoint later + +When you add a new `Base.@ccallable` to `ols.jl`: + +- `_lowlevel.py` is **regenerated on every** `write_wrapper` / + `build_library` call — your new entrypoint shows up automatically. +- `_facade.py` is **written once** and then never touched. To pick + up new entrypoints in the starter façade, delete the file and + rebuild; JuliaLibWrapping will regenerate it (auto-wrapping where + it can, leaving `# TODO: hand-wrap` markers where it cannot). +- `__init__.py` is regenerated to re-export from `_facade`. + +A common pattern is to keep `_facade.py` checked into your own +repository alongside the build script, treating it as hand-written +glue that occasionally absorbs new auto-wrappers when you delete +and regenerate it. diff --git a/examples/ols/Project.toml b/examples/ols/Project.toml new file mode 100644 index 0000000..93f47b9 --- /dev/null +++ b/examples/ols/Project.toml @@ -0,0 +1,17 @@ +name = "ols" +uuid = "7e81292c-b63a-42d7-9477-255b6fedc2ed" +version = "0.0.1" + +[deps] +JLWInterop = "65e54657-ed21-41a3-96db-71ab7fa6d94b" +LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +# Note: `[sources]` is intentionally absent. `build.jl` materializes a +# transient project in a temp directory with `[sources]` pointing at an +# absolute path to the in-tree `JLWInterop/`, so this file stays free of +# any machine-specific path. Tutorial readers who hand-write their own +# entry project will add their own `[sources]` here. + +[compat] +JLWInterop = "0.1" +julia = "1.13" diff --git a/examples/ols/build.jl b/examples/ols/build.jl new file mode 100644 index 0000000..654809e --- /dev/null +++ b/examples/ols/build.jl @@ -0,0 +1,51 @@ +# Build the OLS tutorial library end-to-end with a bundled Python package. +# Run from this directory with the Julia 1.13 release candidate: +# +# julia +rc --project=. build.jl +# +# Output lands in `out/`: +# out/ols.so — the compiled shared library +# out/ols.h — the C header +# out/ols_py/ — the Python package (with `bundle/` runtime closure) +# +# The bundled tree is several hundred MiB; that is what lets a downstream +# `pip install ./out/ols_py` work in a clean venv with no system Julia. +# +# `juliac` requires `[sources]` entries in the entry project's `Project.toml` +# to be absolute paths, so this script materializes a transient project in a +# temp directory with `[sources]` pointing at the in-tree `JLWInterop/`. The +# committed `Project.toml` therefore stays free of any machine-specific path. + +using TOML: TOML +using JuliaLibWrapping +using JuliaC + +const HERE = @__DIR__ +const OUT = joinpath(HERE, "out") +const ENTRY = joinpath(HERE, "src", "ols.jl") +const JLW_INTEROP = abspath(joinpath(HERE, "..", "..", "JLWInterop")) + +# Materialize a temp project with absolute `[sources]` for this machine. +function prepare_project() + toml = TOML.parsefile(joinpath(HERE, "Project.toml")) + sources = get(toml, "sources", Dict{String, Any}()) + sources["JLWInterop"] = Dict("path" => JLW_INTEROP) + toml["sources"] = sources + tmp = mktempdir(; prefix = "ols-project-") + open(joinpath(tmp, "Project.toml"), "w") do io + TOML.print(io, toml; sorted = true) + end + return tmp +end + +result = build_library(ENTRY, + [CTarget(OUT, "ols"), + PythonTarget(OUT, "ols_py", "ols"; bundle_subdir = "bundle")]; + project = prepare_project(), + libname = "ols", + libdir = OUT, + bundle = true, + verbose = true, +) + +@info "Built ols" library=result.library bundle=result.bundle_dir diff --git a/examples/ols/src/ols.jl b/examples/ols/src/ols.jl new file mode 100644 index 0000000..f3024ad --- /dev/null +++ b/examples/ols/src/ols.jl @@ -0,0 +1,101 @@ +module ols + +# Tiny OLS regression library, wrapped end-to-end for the JuliaLibWrapping +# tutorial. Three entrypoints exercise the recognized JLWInterop vocabulary: +# `fit` returns a struct embedding a `JLWStatus` (so its façade wrapper is +# the hand-edited path), `predict` returns `JLWStatus` directly (the +# auto-wrapped path that raises `JLWError` on failure), and `summary_report` +# writes into a caller-allocated `CString` buffer. + +using JLWInterop +using LinearAlgebra + +struct FitResult + status::JLWStatus + coeffs::CVector{Float64} + r_squared::Float64 +end + +# Caller-owned storage discipline: `coeffs_buf` is allocated by the caller +# and outlives the call; the returned `FitResult.coeffs` aliases it. +Base.@ccallable function fit(X::CMatrix{Float64}, + y::CVector{Float64}, + coeffs_buf::CVector{Float64})::FitResult + n, p = size(X) + if length(y) != n + return FitResult(jlw_error(1, "y length must match X rows"), coeffs_buf, 0.0) + end + if length(coeffs_buf) != p + return FitResult(jlw_error(2, "coeffs_buf length must match X cols"), coeffs_buf, 0.0) + end + if n < p + return FitResult(jlw_error(3, "underdetermined system (rows < cols)"), coeffs_buf, 0.0) + end + + coeffs = X \ y + @inbounds for i in 1:p + coeffs_buf[i] = coeffs[i] + end + + ymean = 0.0 + @inbounds for i in 1:n + ymean += y[i] + end + ymean /= n + ss_tot = 0.0 + ss_res = 0.0 + @inbounds for i in 1:n + yhat = 0.0 + for j in 1:p + yhat += X[i, j] * coeffs_buf[j] + end + d_tot = y[i] - ymean + ss_tot += d_tot * d_tot + d_res = y[i] - yhat + ss_res += d_res * d_res + end + r_squared = ss_tot == 0 ? 0.0 : 1.0 - ss_res / ss_tot + return FitResult(jlw_ok(), coeffs_buf, r_squared) +end + +Base.@ccallable function predict(coeffs::CVector{Float64}, + X::CMatrix{Float64}, + out::CVector{Float64})::JLWStatus + n, p = size(X) + if length(coeffs) != p + return jlw_error(1, "coeffs length must match X cols") + end + if length(out) != n + return jlw_error(2, "out length must match X rows") + end + @inbounds for i in 1:n + s = 0.0 + for j in 1:p + s += X[i, j] * coeffs[j] + end + out[i] = s + end + return jlw_ok() +end + +Base.@ccallable function summary_report(result::FitResult, buf::CString)::JLWStatus + if result.status.code != 0 + return jlw_error(101, "result has non-zero status; nothing to report") + end + msg = "OLS fit: " * string(length(result.coeffs)) * + " coefficients, R^2 = " * string(result.r_squared) + bytes = codeunits(msg) + capacity = Int(buf.length) + if length(bytes) > capacity + return jlw_error(102, "report buffer too small") + end + @inbounds for i in 1:length(bytes) + unsafe_store!(buf.data, bytes[i], i) + end + @inbounds for i in (length(bytes) + 1):capacity + unsafe_store!(buf.data, 0x00, i) + end + return jlw_ok() +end + +end # module From 7f1fbc1dc3915d0ecda435a8f0a026c82c693966 Mon Sep 17 00:00:00 2001 From: Tim Holy Date: Tue, 26 May 2026 07:48:02 -0500 Subject: [PATCH 2/3] Tutorial fixes --- Project.toml | 2 +- docs/src/tutorial.md | 172 +++++++++++++++------------- examples/ols/build-env/Project.toml | 26 +++++ examples/ols/build.jl | 48 ++++---- src/JuliaLibWrapping.jl | 2 +- src/build.jl | 51 +++++++++ 6 files changed, 198 insertions(+), 103 deletions(-) create mode 100644 examples/ols/build-env/Project.toml diff --git a/Project.toml b/Project.toml index 96991d6..e7e3328 100644 --- a/Project.toml +++ b/Project.toml @@ -1,6 +1,6 @@ name = "JuliaLibWrapping" uuid = "d61f35a8-f6af-436f-bc10-cee6b101f7bd" -version = "1.0.0-DEV" +version = "0.1" authors = ["Tim Holy and contributors"] [deps] diff --git a/docs/src/tutorial.md b/docs/src/tutorial.md index 7f21a6b..c8f29c1 100644 --- a/docs/src/tutorial.md +++ b/docs/src/tutorial.md @@ -10,12 +10,11 @@ library, run [`build_library`](@ref) to compile it and emit wrappers, worked example lives in `examples/ols/` and stays buildable as the package evolves. -The subject is ordinary least squares (OLS) regression. The algorithm -is incidental — what matters is that one library exercises every -recognized [JLWInterop](https://github.com/JuliaInterop/JuliaLibWrapping.jl/tree/main/JLWInterop) -type: +The subject is ordinary least squares (OLS) regression, which exercises several +[JLWInterop](https://github.com/JuliaInterop/JuliaLibWrapping.jl/tree/main/JLWInterop) +types: -| Vocabulary type | Where it appears in `ols` | +| JLWInterop type | Where it appears in `ols` | |----------------------------|--------------------------------------| | `CMatrix{Float64}` | design matrix `X` | | `CVector{Float64}` | response `y`, coefficients, predictions | @@ -24,6 +23,12 @@ type: | `JLWStatus` (direct) | return of `predict` | | `JLWStatus` (embedded) | `FitResult.status` field | +`CMatrix` and `CVector` are specific cases of `CArray`, the multi-dimensional +array type in JLWInterop. + +The core of the algorithm is a single statement, `X \ y`, using Julia's own +LinearAlgebra for computation. + ## 1. The Julia source `examples/ols/src/ols.jl` defines three [`Base.@ccallable`] entrypoints @@ -59,7 +64,7 @@ Base.@ccallable function summary_report(result::FitResult, `coeffs_buf` and `out` are *caller-allocated* buffers: the library writes into them but does not own them. This is the same discipline -JLWInterop documents for all its pointer-bearing types — see +JLWInterop documents for all its pointer-bearing types, notably `CArray` and `CString`. Errors travel back as a `JLWStatus`, @@ -69,14 +74,6 @@ both forms and translates a non-zero `code` into a `JLWError` exception — see [Error handling](@ref "Error handling across the ABI"). -!!! note "About the algorithm" - `fit` is a one-liner: `coeffs = X \\ y`. The interesting story is - the wrapping, not the math. Production code with many predictors, - rank-deficient inputs, or weighting concerns would substitute a - proper factorization — the wrapping contract (`CMatrix{Float64}` - in, caller-allocated `CVector{Float64}` out, `JLWStatus` for - failures) does not change. - ## 2. The entry `Project.toml` A minimal `Project.toml` for the library: @@ -90,67 +87,88 @@ version = "0.0.1" JLWInterop = "65e54657-ed21-41a3-96db-71ab7fa6d94b" LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" -[sources] -JLWInterop = {path = "/absolute/path/to/JuliaLibWrapping/JLWInterop"} - [compat] JLWInterop = "0.1" julia = "1.13" ``` -Two things to call out: +There are two things to point out: -- The `julia = "1.13"` floor is mandatory; the `juliac` feature that - emits the ABI JSON shipped in Julia 1.13. -- `[sources]` paths must be **absolute**. `juliac` relocates the - project into a temporary directory before compiling, so a relative - path cannot be resolved. [`build_library`](@ref) rejects relative - `[sources]` up front with a clear error. +- The requirement for `julia = "1.13"` cannot be changed to an earlier Julia + release, as the features needed to build language bindings shipped in Julia + 1.13. The OLS example here calls into BLAS via `\`, and that path needs Julia + 1.13.0-rc2 or later (or any build of the `backports-release-1.13` branch from + 2026-05-20 onward). -The in-tree `examples/ols/Project.toml` deliberately omits `[sources]` -so the file does not bake in a machine-specific path; its `build.jl` -instead materializes a transient project in a temp directory with -`[sources]` computed from `@__DIR__` and passes that as the `project=` -argument to [`build_library`](@ref). Authoring a fresh library, you -would normally just write `[sources]` once with your local absolute -path. +- The `[deps]` here describe what must be baked into `ols.so`. Keep it minimal, + without build tooling or test dependencies. If you need a `[sources]` entry to + point at a local checkout, use an absolute path: `juliac` relocates the + project into a temporary directory before compiling, so relative `[sources]` + paths cannot be resolved, and [`build_library`](@ref) rejects them. ## 3. Build the library and Python package -`examples/ols/build.jl` is the driver: +Two distinct environments are in play during a build: + +- The **entry project** (`examples/ols/Project.toml`, activated by + `julia --project=.`) declares the runtime deps just described. +- A **build-env** (`examples/ols/build-env/Project.toml`) declares the + build tooling — [JuliaLibWrapping](https://github.com/JuliaInterop/JuliaLibWrapping.jl) + and [JuliaC](https://github.com/JuliaLang/JuliaC.jl). `build.jl` pushes + this directory onto `LOAD_PATH` so that `using JuliaLibWrapping` + resolves there. Keeping the build tooling out of the entry project's + `[deps]` is what lets your library remain reasonably minimal. + +The build-env's `Project.toml` is just: + +```toml +[deps] +JuliaC = "acedd4c2-ced6-4a15-accc-2607eb759ba2" +JuliaLibWrapping = "d61f35a8-f6af-436f-bc10-cee6b101f7bd" + +[compat] +JuliaC = "0.3" +JuliaLibWrapping = "0.1" +julia = "1.13" +``` + +Instantiate it once: + +```sh +julia --project=examples/ols/build-env -e 'using Pkg; Pkg.instantiate()' +``` + +To handle the two-environment split, we'll push `build-env` onto `LOAD_PATH`, +making it reachable, and then pop it again once the build concludes. +Here is the `build.jl` script: ```julia -using JuliaLibWrapping -using JuliaC - -const HERE = @__DIR__ -const OUT = joinpath(HERE, "out") -const ENTRY = joinpath(HERE, "src", "ols.jl") - -result = build_library(ENTRY, - [CTarget(OUT, "ols"), - PythonTarget(OUT, "ols_py", "ols"; bundle_subdir = "bundle")]; - project = HERE, - libname = "ols", - libdir = OUT, - bundle = true, - verbose = true, -) +push!(LOAD_PATH, joinpath(@__DIR__, "build-env")) +using JuliaLibWrapping, JuliaC + +standard_build(@__DIR__; libname = "ols", verbose = true) +pop!(LOAD_PATH) ``` -Run it with the Julia 1.13 release candidate: +`using JuliaC` is what activates JuliaLibWrapping's weak dependency on +[JuliaC.jl](https://github.com/JuliaLang/JuliaC.jl); without it +[`build_library`](@ref) errors with a hint pointing at this line. + +[`standard_build`](@ref) is a convenience wrapper around +[`build_library`](@ref) for the conventional layout — `src/.jl` +as the entry, `out/` as the artifact directory, both a C header and a +bundled Python `ctypes` package named `_py`. For layouts +outside that convention, or to drop one of the targets, call +[`build_library`](@ref) directly; `standard_build`'s docstring shows +the equivalent expansion. + +Then run the build: ```sh cd examples/ols -julia +rc --project=. build.jl +julia --project=. build.jl ``` -`JuliaLibWrapping` and `JuliaC` need to be loadable — typically -`Pkg.develop` them into your global v1.13 environment, so the entry -project's stripped-down `[deps]` (just `JLWInterop`) reflects the -*runtime* dependencies of the compiled library rather than the build -tooling. - After a successful build, `out/` contains: out/ @@ -169,11 +187,6 @@ Julia on their machine. See the bundling section of the [overview](@ref "Bundling for distribution") for what is in the bundle tree and how the loader finds `libjulia` from inside the wheel. -The default backend is `:juliac` (via [JuliaC.jl](https://github.com/JuliaLang/JuliaC.jl)); -ABI developers chasing features ahead of JuliaC.jl can opt into the -unstable in-tree `share/julia/juliac/juliac.jl` script via -`backend = :script`. - ## 4. Install the Python package Create a clean virtualenv with no system Julia, and install: @@ -192,9 +205,10 @@ required. ## 5. Call it from Python -`predict` is auto-wrapped — its arguments and return are all -recognized, so the façade accepts numpy arrays and raises `JLWError` -on a non-zero status: +To verify that things work, first we'll try calling `predict`, which is +auto-wrapped because its arguments and return are all recognized as known types +to JuliaLibWrapping's emitter. We'll call `predict` twice, once with correct +inputs and once with incorrect ones to verify that errors work as expected: ```python import numpy as np @@ -213,17 +227,17 @@ except JLWError as e: print(e.code, e.message) # 1, "coeffs length must match X cols" ``` -`np.asfortranarray` is required for any `CMatrix{T}` argument: -JLWInterop's `CArray` is column-major, and the façade rejects a -row-major view rather than silently transposing. +`np.asfortranarray` is required for any `CMatrix{T}` argument: JLWInterop's +`CArray` is column-major, and the automatically created façade rejects a +row-major view rather than silently transposing. In a moment you'll +see how to edit the wrapper, so you can choose any interface you wish. -`fit` returns a `FitResult` struct that *contains* a `JLWStatus`. -The emitter doesn't know how to shape that idiomatically (numpy of -`coeffs`? a tuple? the whole struct?), so the starter façade -re-exports it from `_lowlevel` with a `TODO: hand-wrap` comment -naming the obstacle. You edit `_facade.py` to provide the wrapper -you want. The mechanical layer still raises `JLWError` on a non-zero -status, so a sklearn-flavored hand wrap is just: +In contrast with `predict`, `fit` is not automatically wrapped: it returns a +`FitResult`, and JuliaLibWrapping declines to make choices about what that +should look like from the Python perspective. The starter façade re-exports it +from `_lowlevel` with a `TODO: hand-wrap` comment naming the obstacle. You edit +`_facade.py` to provide the wrapper you want. The mechanical layer still raises +`JLWError` on a non-zero status, so a sklearn-flavored hand wrap is just: ```python # in ols_py/_facade.py, replacing the auto-generated TODO line @@ -267,7 +281,9 @@ When you add a new `Base.@ccallable` to `ols.jl`: it can, leaving `# TODO: hand-wrap` markers where it cannot). - `__init__.py` is regenerated to re-export from `_facade`. -A common pattern is to keep `_facade.py` checked into your own -repository alongside the build script, treating it as hand-written -glue that occasionally absorbs new auto-wrappers when you delete -and regenerate it. +A common pattern is to keep `_facade.py` checked into your own repository +alongside the build script, treating it as hand-written glue. If you need to +automatically wrap new functions, currently the best option is to create a +branch in which you delete it, regenerate it with a fresh build, and then copy +the new pieces you want to keep into your existing hand-edited `_facade.py` and +make any additional edits needed for the new code. diff --git a/examples/ols/build-env/Project.toml b/examples/ols/build-env/Project.toml new file mode 100644 index 0000000..ba52bee --- /dev/null +++ b/examples/ols/build-env/Project.toml @@ -0,0 +1,26 @@ +# Build-tooling environment for the OLS tutorial. +# +# `build.jl` pushes this directory onto `LOAD_PATH` so that +# `using JuliaLibWrapping` resolves here. This is intentionally +# separate from the entry project (`../Project.toml`), which describes +# the *runtime* dependency closure baked into `ols.so` by `juliac`. +# +# Instantiate once: +# +# julia --project=examples/ols/build-env -e 'using Pkg; Pkg.instantiate()' +# +# The `[sources]` line below pins JuliaLibWrapping to the in-tree +# checkout; tutorial readers writing their own library will drop it +# once JuliaLibWrapping is registered. + +[deps] +JuliaC = "acedd4c2-ced6-4a15-accc-2607eb759ba2" +JuliaLibWrapping = "d61f35a8-f6af-436f-bc10-cee6b101f7bd" + +[sources] +JuliaLibWrapping = {path = "../../.."} + +[compat] +JuliaC = "0.3" +JuliaLibWrapping = "0.1" +julia = "1.13" diff --git a/examples/ols/build.jl b/examples/ols/build.jl index 654809e..171f3a4 100644 --- a/examples/ols/build.jl +++ b/examples/ols/build.jl @@ -1,31 +1,34 @@ # Build the OLS tutorial library end-to-end with a bundled Python package. -# Run from this directory with the Julia 1.13 release candidate: +# Run from this directory with a recent enough Julia 1.13: # -# julia +rc --project=. build.jl +# julia --project=. build.jl # -# Output lands in `out/`: -# out/ols.so — the compiled shared library -# out/ols.h — the C header -# out/ols_py/ — the Python package (with `bundle/` runtime closure) +# Output lands in `out/`. See `../../docs/src/tutorial.md` for the full +# walkthrough. The build-env at `./build-env/` must be instantiated once: # -# The bundled tree is several hundred MiB; that is what lets a downstream -# `pip install ./out/ols_py` work in a clean venv with no system Julia. +# julia --project=build-env -e 'using Pkg; Pkg.instantiate()' # -# `juliac` requires `[sources]` entries in the entry project's `Project.toml` -# to be absolute paths, so this script materializes a transient project in a -# temp directory with `[sources]` pointing at the in-tree `JLWInterop/`. The -# committed `Project.toml` therefore stays free of any machine-specific path. +# In a tutorial-shaped library (post-registration of JuliaLibWrapping and +# JLWInterop) this script collapses to: +# +# push!(LOAD_PATH, joinpath(@__DIR__, "build-env")) +# using JuliaLibWrapping, JuliaC +# standard_build(@__DIR__; libname = "ols", verbose = true) +# +# `using JuliaC` is what activates JuliaLibWrapping's weak dependency +# on JuliaC.jl — without it, `build_library` errors with a hint. +# +# The extra machinery below exists only because we are dogfooding against +# the in-tree `JLWInterop/` checkout: `juliac` requires `[sources]` paths +# in the entry project to be absolute, so we materialize a transient copy +# of `Project.toml` with `[sources]` injected. Once `JLWInterop` is +# registered, the `prepare_project` step disappears too. using TOML: TOML -using JuliaLibWrapping -using JuliaC const HERE = @__DIR__ -const OUT = joinpath(HERE, "out") -const ENTRY = joinpath(HERE, "src", "ols.jl") const JLW_INTEROP = abspath(joinpath(HERE, "..", "..", "JLWInterop")) -# Materialize a temp project with absolute `[sources]` for this machine. function prepare_project() toml = TOML.parsefile(joinpath(HERE, "Project.toml")) sources = get(toml, "sources", Dict{String, Any}()) @@ -38,13 +41,12 @@ function prepare_project() return tmp end -result = build_library(ENTRY, - [CTarget(OUT, "ols"), - PythonTarget(OUT, "ols_py", "ols"; bundle_subdir = "bundle")]; - project = prepare_project(), +push!(LOAD_PATH, joinpath(HERE, "build-env")) +using JuliaLibWrapping, JuliaC + +result = standard_build(HERE; libname = "ols", - libdir = OUT, - bundle = true, + project = prepare_project(), verbose = true, ) diff --git a/src/JuliaLibWrapping.jl b/src/JuliaLibWrapping.jl index c6c907b..aa593b9 100644 --- a/src/JuliaLibWrapping.jl +++ b/src/JuliaLibWrapping.jl @@ -4,7 +4,7 @@ using OrderedCollections: OrderedDict using Graphs: SimpleDiGraph, add_edge!, strongly_connected_components, topological_sort using JSON: JSON -export parse_abi_info, read_abi_info, write_wrapper, build_library +export parse_abi_info, read_abi_info, write_wrapper, build_library, standard_build export AbstractTarget, CTarget, PythonTarget, ABIInfo include("abi_import.jl") diff --git a/src/build.jl b/src/build.jl index 30a9525..ea8645d 100644 --- a/src/build.jl +++ b/src/build.jl @@ -156,6 +156,57 @@ function _copy_bundle_into_python_package(t::PythonTarget, bundle_dir::AbstractS return dest end +""" + standard_build(dir = pwd(); libname, kwargs...) + +Convenience wrapper around [`build_library`](@ref) for the conventional +single-library layout: + + dir/ + ├── Project.toml # entry project (runtime deps only) + ├── src/ + │ └── .jl # @ccallable entrypoints + └── out/ # generated artifacts + +Emits both a C header and a Python `ctypes` package (`_py`), +bundled for distribution. Equivalent to: + +```julia +build_library(joinpath(dir, "src", libname*".jl"), + [CTarget(joinpath(dir, "out"), libname), + PythonTarget(joinpath(dir, "out"), libname*"_py", libname; + bundle_subdir = "bundle")]; + project = dir, libname, libdir = joinpath(dir, "out"), + bundle = true, kwargs...) +``` + +The kwargs `out`, `entry`, `python_package`, `project`, and `bundle` +override the defaults above; anything else is forwarded to +`build_library` (e.g. `verbose`, `trim`, `privatize`). `project` +defaults to `dir`, but can be pointed at a separate location when the +on-disk source layout and the entry `Project.toml` live in different +directories (e.g. a transient project materialized with absolute +`[sources]` paths for `juliac`). For layouts outside this convention, +call `build_library` directly. +""" +function standard_build(dir::AbstractString = pwd(); + libname::AbstractString, + project::AbstractString = dir, + out::AbstractString = joinpath(dir, "out"), + entry::AbstractString = joinpath(dir, "src", libname * ".jl"), + python_package::AbstractString = libname * "_py", + bundle::Bool = true, + kwargs...) + targets = AbstractTarget[ + CTarget(out, libname), + PythonTarget(out, python_package, libname; + bundle_subdir = bundle ? "bundle" : nothing), + ] + return build_library(entry, targets; + project, libname, libdir = out, bundle, + kwargs...) +end + function _validate_sources_absolute(project::AbstractString) pf = joinpath(project, "Project.toml") isfile(pf) || return # nothing to validate From acabec84faefe30d6d53a58f8473543002d87f5e Mon Sep 17 00:00:00 2001 From: Tim Holy Date: Fri, 29 May 2026 08:17:20 -0500 Subject: [PATCH 3/3] More tutorial updates --- docs/src/index.md | 22 ++++----- docs/src/tutorial.md | 111 +++++++++++++++++++++++++++++++++++++------ 2 files changed, 108 insertions(+), 25 deletions(-) diff --git a/docs/src/index.md b/docs/src/index.md index 7e82a15..92b78be 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -17,8 +17,8 @@ Julia source through `pip install` to numpy-flavored Python. ## Who it is for -Authors of Julia libraries who want to ship a compiled artifact that -downstream users — typically C or Python programmers — can consume +Authors of Julia libraries who want to ship compiled code that +downstream users — currently C or Python programmers — can use without installing a Julia runtime themselves. The compiled library is produced by `juliac`; this package produces the binding code that makes it usable from the target language. @@ -40,16 +40,16 @@ bundling, multi-library, and two-tier output stories. ## Where to go next -- [Tutorial: wrap an OLS regression library](@ref) — the on-ramp: - one library, end to end, from Julia source through `pip install` - to numpy-flavored Python. -- [Concepts](@ref) — the pipeline, the ABI data model, the extension +- [Tutorial: wrap an OLS regression library](@ref): build a small library with a + Python wrapper using numpy. +- [Concepts](@ref): the pipeline, the ABI data model, the extension point for new target languages, and the runtime-closure / bundling story. -- [JLWInterop](@ref) — the dependency-free vocabulary package - (`CVector`, `CMatrix`, `CString`, `JLWStatus`) that compiled - libraries and the Python emitter both speak. -- [Error handling across the ABI](@ref) — the `JLWStatus` convention +- [JLWInterop](@ref): a small package needed by almost any wrapped Julia module. + Defines a few interoperability types (`CArray`, `CString`, and `JLWStatus`) + that your Julia wrapper-code should use to ensure interopability with the + language binding. +- [Error handling across the ABI](@ref): the `JLWStatus` convention that lets wrapped libraries surface errors as native exceptions in the target language. -- [API reference](@ref) — generated docstrings for the public API. +- [API reference](@ref): the public API, in full detail. diff --git a/docs/src/tutorial.md b/docs/src/tutorial.md index c8f29c1..cd06251 100644 --- a/docs/src/tutorial.md +++ b/docs/src/tutorial.md @@ -31,8 +31,8 @@ LinearAlgebra for computation. ## 1. The Julia source -`examples/ols/src/ols.jl` defines three [`Base.@ccallable`] entrypoints -on top of JLWInterop's vocabulary: +`examples/ols/src/ols.jl` defines three `Base.@ccallable` entrypoints. +Note that all three of these take arguments with types defined in JLWInterop: ```julia module ols @@ -62,10 +62,12 @@ Base.@ccallable function summary_report(result::FitResult, buf::CString)::JLWStatus ``` +!!! tip + To see the full source code, check the `examples/ols/src` directory. + `coeffs_buf` and `out` are *caller-allocated* buffers: the library -writes into them but does not own them. This is the same discipline -JLWInterop documents for all its pointer-bearing types, notably -`CArray` and `CString`. +writes into them but does not own them. This is generally true for +JLWInterop types that hold pointers, `CArray` and `CString`. Errors travel back as a `JLWStatus`, either returned directly (`predict`, `summary_report`) or embedded in @@ -81,7 +83,7 @@ A minimal `Project.toml` for the library: ```toml name = "ols" uuid = "7e81292c-b63a-42d7-9477-255b6fedc2ed" -version = "0.0.1" +version = "0.1.0" [deps] JLWInterop = "65e54657-ed21-41a3-96db-71ab7fa6d94b" @@ -132,10 +134,14 @@ JuliaLibWrapping = "0.1" julia = "1.13" ``` -Instantiate it once: +Henceforth it will be assumed that you are in the directory containing both +environments (e.g., `examples/ols`): + +Instantiate each environment from your shell: ```sh -julia --project=examples/ols/build-env -e 'using Pkg; Pkg.instantiate()' +julia --project=build-env -e 'using Pkg; Pkg.instantiate()' +julia --project -e 'using Pkg; Pkg.instantiate()' ``` To handle the two-environment split, we'll push `build-env` onto `LOAD_PATH`, @@ -162,10 +168,9 @@ outside that convention, or to drop one of the targets, call [`build_library`](@ref) directly; `standard_build`'s docstring shows the equivalent expansion. -Then run the build: +Then run the build from your shell: ```sh -cd examples/ols julia --project=. build.jl ``` @@ -189,15 +194,23 @@ tree and how the loader finds `libjulia` from inside the wheel. ## 4. Install the Python package -Create a clean virtualenv with no system Julia, and install: +Create a clean virtualenv with no system Julia, and install. Put the +virtualenv **outside** the entry-project directory (here, `/tmp/ols-venv`): +`juliac` copies the whole project tree on every build, and a venv nested +inside it has symlinks that break that copy. ```sh python -m venv /tmp/ols-venv source /tmp/ols-venv/bin/activate pip install numpy -pip install ./examples/ols/out/ +pip install -e ./out ``` +The leading `./` matters: `pip install out` would treat `out` as a +package name and fetch an unrelated project from PyPI. The `-e` +(editable) flag installs the package in place, so the `_facade.py` +edits you make in the next section take effect without reinstalling. + The bundled `libjulia` and stdlibs live inside the wheel; the loader in `_lowlevel.py` searches `bundle/lib/` first so the baked-in `RUNPATH` resolves them at import time, with no `LD_LIBRARY_PATH` @@ -205,6 +218,8 @@ required. ## 5. Call it from Python +### Verifying that the basics work + To verify that things work, first we'll try calling `predict`, which is auto-wrapped because its arguments and return are all recognized as known types to JuliaLibWrapping's emitter. We'll call `predict` twice, once with correct @@ -227,11 +242,17 @@ except JLWError as e: print(e.code, e.message) # 1, "coeffs length must match X cols" ``` +This should be run (or copy/pasted line-by-line) in a python shell running +in the activated environment. If `out` is `array([2.04, 4.02, 6. , 7.98, 9.96])`, +and you see the expected error message, all is working as expected. + `np.asfortranarray` is required for any `CMatrix{T}` argument: JLWInterop's `CArray` is column-major, and the automatically created façade rejects a row-major view rather than silently transposing. In a moment you'll see how to edit the wrapper, so you can choose any interface you wish. +### Making edits to the wrapper + In contrast with `predict`, `fit` is not automatically wrapped: it returns a `FitResult`, and JuliaLibWrapping declines to make choices about what that should look like from the Python perspective. The starter façade re-exports it @@ -250,15 +271,23 @@ def fit(X, y): _lowlevel.CVector_Float64.from_numpy(y), _lowlevel.CVector_Float64.from_numpy(coeffs), ) - return coeffs, float(result.r_squared) + return coeffs, result ``` +Note that `fit` accepts a plain (row-major) `X` and converts it with +`np.asfortranarray` internally, so callers are spared the manual step +that `predict` required. Alongside the coefficients we return the raw +`result` struct, which carries `r_squared` and is what `summary_report` +(next) consumes. A production wrapper might instead bundle these into a +small result object exposing `coeffs`, `r_squared`, and a `.summary()` +method; this returns the pieces directly to keep the example short. + `summary_report` is also a hand-wrap case (its `FitResult` argument is an unrecognized struct). The caller allocates a writable `CString` buffer, passes it in, and decodes the bytes after the call: ```python -def summary(result_struct, capacity=256): +def summary_report(result_struct, capacity=256): import ctypes buf_bytes = (ctypes.c_uint8 * capacity)() buf = _lowlevel.CString( @@ -269,6 +298,35 @@ def summary(result_struct, capacity=256): return bytes(buf_bytes).rstrip(b"\x00").decode("utf-8") ``` +### Trying out `fit` and `summary_report` + +With both wrappers in place, the two compose directly. `fit` is the +inverse direction of `predict`: hand it a design matrix and observed +responses, get back the coefficients. To make the result easy to check, +we feed it the predictions from the `predict` call above as the +observations — a perfectly linear `y`, so `fit` should recover the same +`[0.06, 1.98]` with an `R²` of `1.0`. Restart Python first so the +`_facade.py` edits are picked up (the editable install means no +reinstall is needed): + +```python +import numpy as np +from ols_py import fit, summary_report + +X = np.column_stack([np.ones(5), np.arange(1.0, 6.0)]) # row-major is fine here +y = np.array([2.04, 4.02, 6.0, 7.98, 9.96]) # the predictions from §5, now treated as data + +coeffs, result = fit(X, y) +print(coeffs) # ≈ [0.06, 1.98] — the coefficients predict() used +print(result.r_squared) # 1.0 (the points lie exactly on a line) +print(summary_report(result)) # "OLS fit: 2 coefficients, R^2 = 1.0" +``` + +The `result` returned by `fit` is exactly the `FitResult` struct +`summary_report` expects, so the two chain without any glue. Try perturbing one +entry of `y` and re-running: the coefficients shift slightly and +`result.r_squared` drops below `1.0`, which the summary string reflects. + ## 6. Adding an entrypoint later When you add a new `Base.@ccallable` to `ols.jl`: @@ -281,6 +339,31 @@ When you add a new `Base.@ccallable` to `ols.jl`: it can, leaving `# TODO: hand-wrap` markers where it cannot). - `__init__.py` is regenerated to re-export from `_facade`. +You trigger all of this by re-running the same build: + +```sh +julia --project=. build.jl +``` + +with one current caveat: the build is **not yet idempotent** over an +existing `out/`. The bundle step copies files without overwriting, so a +second build into a populated `out/` fails with something like +`'…/ols-bundle/share/julia/cert.pem' exists`. Until +[JuliaLibWrapping#46](https://github.com/JuliaInterop/JuliaLibWrapping.jl/issues/46) +is resolved, clear the bundle tree first: + +```sh +rm -rf out/ols-bundle +julia --project=. build.jl +``` + +Clear only the bundle tree, **not** the whole `out/`: a blanket +`rm -rf out` also deletes your hand-edited `_facade.py`, which would +then be regenerated as a fresh starter (losing your wrappers). +`_lowlevel.py`, `pyproject.toml`, and `__init__.py` are rewritten in +place, and because you installed with `pip install -e`, a restarted +Python session picks them up with no reinstall. + A common pattern is to keep `_facade.py` checked into your own repository alongside the build script, treating it as hand-written glue. If you need to automatically wrap new functions, currently the best option is to create a