From 5a50f6c5eaead803296f31ce3dfec3339d06628f Mon Sep 17 00:00:00 2001 From: Nye Liu Date: Sun, 11 Oct 2026 19:35:06 -0700 Subject: [PATCH] ui: make -C publish goes to me7-logger not here publish is an me7-logger end to end process. It does not belong here. xdfkit does not manage the corpus, nor does it generate defs. It is purely a conversion tool --- .gitignore | 1 - DEVELOPER.md | 23 +---- Makefile | 4 +- api/api.go | 9 +- cmd/xdfkit/main.go | 7 +- docs/autocorrect.md | 2 +- docs/corpus.md | 90 +++-------------- docs/design.md | 10 +- docs/format-matrix.md | 2 +- docs/integrations.md | 10 +- docs/kp-format.md | 4 +- docs/model.md | 9 +- docs/naming.md | 40 +------- internal/corpus/corpus.go | 13 ++- model/categories.go | 3 +- model/model.go | 13 ++- model/schema.go | 4 +- model/schema.json | 4 +- publish/Makefile | 71 -------------- publish/README.md | 36 ------- publish/import-incoming.sh | 135 -------------------------- publish/index.php | 143 ---------------------------- testdata/archive/ecuxplot/README.md | 11 +-- xdf/xdf_test.go | 2 +- 24 files changed, 71 insertions(+), 575 deletions(-) delete mode 100644 publish/Makefile delete mode 100644 publish/README.md delete mode 100755 publish/import-incoming.sh delete mode 100644 publish/index.php diff --git a/.gitignore b/.gitignore index d9f2d27..460ac47 100644 --- a/.gitignore +++ b/.gitignore @@ -5,4 +5,3 @@ /plans/ /build/ /dist/ -/publish/local.mk diff --git a/DEVELOPER.md b/DEVELOPER.md index 629c1bc..07e81ee 100644 --- a/DEVELOPER.md +++ b/DEVELOPER.md @@ -1,6 +1,6 @@ # xdfkit developer notes -The history of this repo is rewritten often: expect to `git reset --hard origin/master`. Nothing is a contract yet: the CLI, the `api` package, the model and its schema, file formats and the corpus layout can all change. +The history of this repo is rewritten often: expect to `git reset --hard origin/master`. Nothing is a contract yet: the CLI, the `api` package, the model and its schema, and file formats can all change. ## Build and test @@ -54,29 +54,16 @@ Core packages work on bytes, without file or console I/O, and use no cgo; the CL me7-logger imports `model`, `xdf` and `canon` (`docs/integrations.md`): after changing them, run `make check-pinned` in me7-logger before pushing it. -## Published definitions - -The corpus definitions are model JSON in ecu-corpus `defs/`. `publish/` writes one `IMAGE-DATE.zip` from each and imports corrections made in WinOLS or TunerPro, and me7info's models (`publish/README.md`). - -```mermaid -flowchart LR - corpus[("ecu-corpus")] --> publish["make -C publish upload"] - publish --> users["WinOLS / TunerPro users"] - users -- "corrected KP or XDF" --> incoming["make -C publish incoming"] - me7info["me7-logger me7info"] -- "located model JSON" --> incoming - incoming --> corpus -``` - ## Rules - Commit messages start with a prefix that `cliff.toml` groups: `feat`/`add`, `fix`, `docs`, `refactor`, `chore`, `ci`, `build`, `test`, `style`, `chore(deps)`. -- Never edit or regenerate `testdata/archive/ecuxplot/` (see its README). `images.tsv` names each archived pack's corpus image for tests; it is not the publish list. +- Never edit or regenerate `testdata/archive/ecuxplot/` (see its README). `images.tsv` names each archived pack's corpus image for tests. - Private definition files (DAMOS, A2L, OLS) go only in the gitignored `testdata/local/`; never commit them. - `docs/` describes how things are, including spec not implemented yet; plans are kept outside the repo. ## Releases -`make package` writes `dist/`. A tag `vX.Y.Z` publishes those archives as a GitHub release, with notes generated by git-cliff (`cliff.toml`); `vX.Y.Z-rcN` is a prerelease. Archives are `.tar.gz` for macOS and Linux and `.zip` for Windows, each with `xdfkit`, its aliases (`ALIASES` in the Makefile: symlinks, or `.cmd` shims on Windows), `README.md`, `COPYING` and `COPYING.LESSER`. `xdfkit version` prints the tag (`git describe`). +`make package` writes `dist/`. A tag `vX.Y.Z` is a GitHub release of those archives, with notes generated by git-cliff (`cliff.toml`); `vX.Y.Z-rcN` is a prerelease. Archives are `.tar.gz` for macOS and Linux and `.zip` for Windows, each with `xdfkit`, its aliases (`ALIASES` in the Makefile: symlinks, or `.cmd` shims on Windows), `README.md`, `COPYING` and `COPYING.LESSER`. `xdfkit version` prints the tag (`git describe`). ## Documentation @@ -89,6 +76,6 @@ flowchart LR | `docs/autocorrect.md` | `lint` and `fix` | | `docs/stamp-and-metadata.md` | Edit stamp and XDF metadata file | | `docs/winols-script.md` | WinOLS script import format | -| `docs/corpus.md` | The shared image and definition corpus | -| `docs/naming.md` | File naming | +| `docs/corpus.md` | Test images from ecu-corpus | +| `docs/naming.md` | Output file names | | `docs/integrations.md` | ecuxplot, ME7Sum, me7-logger | diff --git a/Makefile b/Makefile index a14cd83..300ca15 100644 --- a/Makefile +++ b/Makefile @@ -50,8 +50,8 @@ test: schema: go test ./model -run "TestSchemaFile|TestKPDefaultsFile" -update -# build replaces build/xdfkit only when it changed, so publish/ rebuilds -# outputs only after a real change. +# build replaces build/xdfkit only when it changed, so a caller that +# checks the binary's mtime rebuilds only after a real change. build: mkdir -p build go build $(LDFLAGS) -o build/xdfkit.new ./cmd/xdfkit diff --git a/api/api.go b/api/api.go index e75c393..d159fa9 100644 --- a/api/api.go +++ b/api/api.go @@ -48,8 +48,8 @@ const ( // (docs/stamp-and-metadata.md). Origin, for KP or XDF input, is where its // definitions came from (model.Origins), recorded in the provenance; it // defaults to hand, or for an XDF to its metadata file's. Tuner is a category -// table (the corpus categories.json): only its maps and the objects their -// axes point at are kept, filed under its categories. +// table: only its maps and the objects their axes point at are kept, filed +// under its categories. type ConvertRequest struct { To string `json:"to"` Name string `json:"name,omitempty"` @@ -332,11 +332,10 @@ func loadModel(in []byte, name string, meta []byte) (*model.Model, []string, err return m, subsetWarning(m, warnings), m.Check() } -// subsetWarning adds a warning when m is a subset (provenance.subset), which -// must not replace the full definition in the corpus. +// subsetWarning adds a warning when m is a subset (provenance.subset). func subsetWarning(m *model.Model, warnings []string) []string { if m.Provenance != nil && m.Provenance.Subset != nil { - warnings = append(warnings, m.Provenance.Subset.Kind+" subset of the original definition, not a corpus source") + warnings = append(warnings, m.Provenance.Subset.Kind+" subset of the original definition") } return warnings } diff --git a/cmd/xdfkit/main.go b/cmd/xdfkit/main.go index 2e2fa42..7b8065f 100644 --- a/cmd/xdfkit/main.go +++ b/cmd/xdfkit/main.go @@ -57,10 +57,9 @@ since it was written converts back to the definition exactly, and edits made to it are reported. --origin records where a KP or XDF input's definitions came from in the model's provenance: damos or a2l (exported from Bosch data), hand (made by hand; the default, unless an XDF's metadata file has -another) or located (found in the image by a program such as me7info; never -a corpus source). --tuner keeps only the maps named in a category table (the -corpus categories.json) and the objects their axes point at, filed under the -table's categories. +another) or located (found in the image by a program such as me7info). +--tuner keeps only the maps named in a category table and the objects their +axes point at, filed under the table's categories. xdf2kp, kp2xdf and kp2json convert each FILE to the same name with the .kp, .xdf or .json extension, next to it, and reject input of another format. They diff --git a/docs/autocorrect.md b/docs/autocorrect.md index 084e2fe..c25f363 100644 --- a/docs/autocorrect.md +++ b/docs/autocorrect.md @@ -30,4 +30,4 @@ Fixing KP files mechanically, instead of hand-editing each axis in WinOLS. Evide - Family: from the Bosch identification string in the image (`40/1/ME7.1/5/...` is ME7, `5655/1/M3.82/...` and `9655/1/M5.92/...` are delta). Images without one need `--family`. - R7 is the merge report (`xdfkit merge --update` of the definition with the located model). It is not a `lint` rule. A caller that must reject a moved name does that before merge; merge does not decide that gate. - R1 changes what WinOLS displays: WinOLS shows a "subtract" axis as 256 minus the sum of the remaining raw values (confirmed in WinOLS 2.24; see the axis datasource section of `kp-format.md`), so a mislabelled absolute axis shows wrong values until fixed. -- R2 fixes display as intended in WinOLS 2.24 (confirmed on KFFA X in 8D0907558E: 1.0, 1.6, 2.2, ..., 10.5, 11.0 ms/rev). The corpus JSON of 8D0907558E and 8D0907558M has every R2 fix applied, so their published KPs have no R2 findings. +- R2 fixes display as intended in WinOLS 2.24 (confirmed on KFFA X in 8D0907558E: 1.0, 1.6, 2.2, ..., 10.5, 11.0 ms/rev). diff --git a/docs/corpus.md b/docs/corpus.md index 055af65..f7b685d 100644 --- a/docs/corpus.md +++ b/docs/corpus.md @@ -1,84 +1,24 @@ -# Shared test corpus +# Test corpus -`ecu-corpus` (`github.com/nyetlabs/ecu-corpus`, private): OEM flash images and their map definitions as canonical JSON, shared by xdfkit, ME7Sum and me7-logger as a git submodule (`corpus/`). Its tools (`tools/`, Go module `go.nyet.org/ecu-corpus/tools`): `corpus-manifest` builds, checks and classifies; `corpus-links` makes the root symlinks. +`ecu-corpus` (`github.com/nyetlabs/ecu-corpus`, private) supplies the OEM flash images and `categories.json` that xdfkit's tests read. It is the `corpus/` submodule (shallow, `update = none`). Tests do not read `defs/`. -Contents: 100 OEM images (99 seeded from the source repos), the manifest, CI check, the ecuxplot definitions as JSON and `categories.json`. Consumers fetch it in CI and run the weekly bump workflow (`.github/workflows/corpus-bump.yml`; each repo must allow Actions to create pull requests). Test access: xdfkit `internal/corpus`, me7-logger `internal/ecucorpus`, ME7Sum `scripts/test.sh`; none commits corpus images any more (ME7Sum keeps its non-OEM ones in `testdata/`). ecuxplot has no images and no submodule. +## What tests read -## Specification +- `images/NAME.bin`, named from `testdata/archive/ecuxplot/images.tsv` (`internal/corpus.ArchiveImage`). `8N0906018CB` uses the bin in that archive: the corpus image of that part is different software. +- `categories.json`, checked by `model.TestCorpusCategoryTable` against `model/categories.schema.json`. +- `corpus.tsv`. `internal/corpus.TestImages` checks every row's size and SHA-256 against `images/`. Lookups are by name or SHA-256. -### Rules +Without a manifest those tests skip. `XDFKIT_REQUIRE_CORPUS=1` makes that a failure. CI sets it when it has the credential. Lint also has synthetic fixtures that need no corpus (`testdata/lint/cases.json`). -- Only images (`.bin`), canonical JSON and the manifest. Never KP, XDF, DAMOS, A2L, OLS, hex or any other original definition file (DAMOS and A2L provenance is often questionable); JSON converted from them does go in, with their names, descriptions and conversions. Private files are never uploaded to third-party services (XDF Porter or similar). -- Contributed images are welcome (the corpus should go public eventually) under the OEM rules below, with `release` = `unknown`. -- OEM (factory, unmodified) images only; modified, tuned, re-signed or identification-wiped ones stay in their source repos or xdfkit `testdata/`. `tools/corpus-manifest -me7sum PATH -overrides corpus-overrides.tsv` runs ME7Sum and writes the `oem`, `release`, `checksums` and `rsa` columns. `rsa`, `checksums` and `hand` qualify. `release` stays `unknown` (all 99 at seeding) until set by hand, and must be `production` before the corpus could go public; a non-production image found later is removed, but stays in the private history, so going public needs a fresh history. +## Where the checkout is -| `oem` | Meaning | -| --- | --- | -| `rsa` | ME7Sum finds an RSA signature, the modulus is Bosch's, and the signed MD5 matches the computed one. Strong evidence. | -| `checksums` | No RSA signature (most ME7.1 and some ME7.5); identification intact and every ME7Sum checksum and CRC passes. Weaker: a tuned image with corrected checksums also passes, and so does a genuine Bosch calibration that never went into production. | -| `no` | Labelled (modified), foreign modulus (re-signed), Bosch modulus with a failing MD5 (signed regions changed), checksum errors, or left out by hand. | -| `hand` | ME7Sum can't check the family; judged OEM by the user (`corpus-overrides.tsv`). | -| `review` | Family ME7Sum doesn't check, ME7Sum fails outright, or the name needs a hand decision; resolved by adding a row to `corpus-overrides.tsv`. | +`XDFKIT_CORPUS` if set, else `../ecu-corpus` when that directory has `corpus.tsv`, else `corpus/` at the module root. -- Bosch modulus: SHA-256 `3565ec2768952ec00ce0a4ffaa9ce851408e29bb2adf4731017aa31cfb735d27` of the 128-byte modulus (exponent 3). Any other modulus means the image was re-signed. -- Strengthening `checksums`: two `checksums` images with the same part and version but different bytes would mean one is modified. A second source of the same image (another dump of the same part and version) also counts. -- Part number and version come from ME7Sum's reading of the ECUID table when `-me7sum` is given (fixed-width fields), and from a text match otherwise. -- `release` (`production`, `non-production`, `unknown`) is set by hand: no check tells a production calibration from a genuine unreleased one (development, pre-series). +```sh +make corpus # submodule at its pinned commit +make corpus-bump # submodule at the ecu-corpus head +``` -### What goes in: images and canonical JSON +## CI -The schema is meant to represent everything any of the projects needs. Other formats are generated from the JSON by xdfkit, and other projects pull in xdfkit rather than keeping format code of their own. - -- One JSON per definition, converted by xdfkit from the original (KP, hand-made XDF, DAMOS, A2L). An image can have several definitions (a KP, a dated KP, me7-logger's hand XDF, a DAMOS file). 4Z7907551R's definition was made on a lightly modified copy of the OEM image and is filed on `4Z7907551R-0003`. 8N0906018CB's pack fits a non-OEM image and has no corpus definition. -- Each JSON records its provenance: source format, original file name and SHA-256, and xdfkit version. So the original stays identifiable even though it isn't in the corpus. -- And the definitions' origin (`provenance.origin`): `damos` or `a2l` (from Bosch data, the reference for map addresses) or `hand` (as WinOLS map packs usually are); me7-logger's parity check relies on it. The DAMOS and A2L readers will set it; KP and XDF input take `xdfkit --origin` (default `hand`, or an XDF metadata file's). The publish import keeps the corpus JSON's origin unless given `ORIGIN=`, but makes a `hand` definition with 3000 or more maps `damos` (hand packs have at most about 540, most DAMOS exports over 4000); smaller exports (8D0907558E about 1800, 8D0907558M about 400) need `ORIGIN=damos`. Origin `located` (maps me7info found in the image, format `image`) is written by `me7parity publish --write` when confidence is not mostly low. With no corpus definition, or when the JSON is already located, the parity grade must be D or better. A `hand` definition is replaced before that grade when the located model is a strict superset (every hand object at the same address, and at least one more name), and also when the names match and a description or comment would change. The shared objects keep the hand conversion, units, and layout. The description becomes the name-table sentence and the comment the locating needle. It does not replace a `damos` or `a2l` definition with a located model. A key `obj-N` with a blank id is a placeholder, not a name. Publish lists every such definition, including one that is not a parity image, as `keys` with delta `key N`. `--write` clears the key and leaves the origin, and sets that row's `def` cell to `defs/NAME.json`. A row that is not eligible for another reason keeps that status with `, key N` on the end. Additions and wrong addresses in `damos` and `a2l` files are still edited into that JSON (`publish/README.md`). `import-incoming.sh` remains the importer for KP, XDF, and DAMOS, and still will not write `located` over `hand`. -- The edit stamp (`stamp-and-metadata.md`): clean means converted, regenerate at will; edited means a person changed it and it is the source of truth, never overwritten by regeneration. -- One category table, `categories.json` (schema `model/categories.schema.json`, `model.CategoryTable`): `{"schema": 1, "categories": {"KFZW": "Timing", ...}}`. Names: the tuner list (me7-logger `testdata/parity/names/tuner.yaml`, from the S4wiki tuning page); categories drafted from 8D0907551M's folders, then edited by hand. The 1.8T block list (about 4,100 more names) is left out: it covers most of any ME7 DAMOS (tuner XDFs kept 75-97% of maps). `xdfkit --tuner` and `publish/` (ME7 packs only) write subsets from it, marked `provenance.subset` (`model.md`). me7-logger ships a copy as `config/categories.json`, so a rename takes a corpus commit plus a bump in each consumer. -- Per-image tool data that isn't a map definition (ME7Sum's `.ini` checksum layout) goes into the schema as its own section if a consumer needs it from the corpus. Otherwise it stays with the tool. -- No tool outputs: goldens (mapdump CSV and XDF, ME7Sum `.bin.txt`, me7info XDF and `.ecu`) stay with the tool that produces them, keyed by corpus id (mapdump's CSVs for ecuxplot's packs are archived in xdfkit). - -### Where the original files live - -Readers need originals for regression tests (DAMOS and A2L have no writers; exact KP bytes need the file). - -- Private originals (DAMOS, A2L, OLS and their hex files) live only in a checkout's gitignored `testdata/local/`, never committed or pushed anywhere. Tests that need them skip when they are absent. -- Public originals are read from the repos that publish them or copied into xdfkit `testdata/`. ecuxplot's packs and mapdump CSVs are in `testdata/archive/ecuxplot/`: never edited, the KP reader's oracle; `images.tsv` names each archived pack's corpus image for tests. Their definitions are maintained as corpus JSON. -- Each original is checked against the provenance hash in its corpus JSON, where one exists. - -### Layout and ids - -- `images/PART-VERSION[-label].bin`: the images, and `defs/PART-VERSION.json`: the definitions (first, ecuxplot's packs), named as specified in `naming.md`. Part number and software version come from the image itself, so names are deterministic. Tests and tools use these paths only. -- Root symlinks into `images/` for browsing (`tools/corpus-links`, checked by `corpus-manifest -check`): `PART.bin` while a part has one image, else `PART-VERSION[-label].bin`. Names change when a part gains an image, so tools don't use them. -- `corpus.tsv` at the root, generated by `tools/corpus-manifest -corpus images` and committed. Columns: name, sha256, size, family, ident, oem, release, checksums, rsa, def (the image's definition, `defs/NAME.json`, or `-`). `corpus-manifest -check` (CI) also fails on a `def` that doesn't match `defs/` and on files in `defs/` not named after an image. Tests look files up by id or hash through the manifest, never by guessing paths. -- `categories.json` at the root, outside `defs/`, so `corpus-manifest -check` ignores it. - -### Policy - -- DAMOS and A2L definitions (origin `damos`, `a2l`) are maintained in the corpus JSON: fixes are made there, and a WinOLS project wanting them imports a KP generated from it (`xdfkit -f kp`) rather than being re-exported. Hand definitions are maintained in their WinOLS projects and imported with `publish/import-incoming.sh`. `me7parity publish` is what replaces a hand file with a located model, and only when that model strictly covers it or a description or comment would change (`publish/README.md`). The same command clears a placeholder key `obj-N` (blank id) in any definition, including `damos` and `a2l`, and leaves that origin. -- WinOLS's DAMOS import loses reciprocal conversions: a DAMOS conversion `/REP 0, B, C, 0` (phys = (B / C) / raw; the `zk*` time constants and `rfak*` factors) comes out as factor 1, so the map reads the raw value (confirmed against 06A906032HF's `.dam`; 4B0906018CM and 06A906032S kept theirs). A definition exported from such a project needs its reciprocal conversions restored in the corpus JSON, which keeps an edited stamp. `import-incoming.sh` warns when a DAMOS definition has no reciprocal conversions, or has fewer than the corpus JSON it replaces. Planned: the DAMOS reader sets `reciprocal` with factor B/C and offset 0 for `/REP 0, B, C, 0`, refuses a rational with A or D nonzero rather than writing factor 1, and is checked against 06A906032HF's `.dam`. -- Images are immutable: a changed image gets a new id. JSON files do change, through ordinary commits: hand fixes, and bulk regeneration when the schema or a reader improves. Bulk regeneration skips JSON with an edited stamp. -- Schema versions: every JSON names its schema version. xdfkit reads older versions, or the corpus is regenerated in one commit per schema bump. Consumers pin a corpus commit and an xdfkit version that agree. -- Canonical form (`jq -S .`, one value per line) keeps JSON diffs small and reviewable in git. -- Plain git, not LFS (about 100 MB; LFS adds a tool and bandwidth quotas). Consumers use `shallow = true` in `.gitmodules`. -- Edits go in a full clone, `ecu-corpus` beside the consumer repos (`../ecu-corpus`): commit and push there, then `make corpus-bump` in each consumer. The `corpus/` submodules are read-only: shallow, so `git log` may not reach a JSON file's last commit and dates its zip wrongly, and `corpus-bump` detaches them, so work left there is lost. xdfkit's `publish/` and `import-incoming.sh` use `../ecu-corpus` when it exists, else the submodule (`CORPUS=` overrides). - -### Access: private repo - -Private for now; consumer repos can be public (only the submodule URL and commit show). - -- Without access `corpus/` is empty and corpus tests skip, naming this document. Go modules never include submodule contents. -- xdfkit tests (`internal/corpus`) use `XDFKIT_CORPUS`, else `../ecu-corpus` if it has a `corpus.tsv` (so local edits are tested before a bump), else `corpus/`; no `corpus.tsv` means not available. Images are looked up by name or SHA-256. -- `XDFKIT_REQUIRE_CORPUS=1` makes not available a failure; CI sets it whenever it has the credential, so a dead credential fails the build. Fork PRs get no secrets and skip; don't use `pull_request_target` to get around that unless a maintainer gates the run. -- `.gitmodules`: absolute URL `https://github.com/nyetlabs/ecu-corpus.git` (a relative one breaks forks; SSH users map it with `url.git@github.com:.insteadOf`), and `update = none`, so `git submodule update --init` doesn't fail without access; with access, `git submodule update --init --checkout corpus` (or `make corpus`). Unconfirmed: `git clone --recurse-submodules` with this setting. -- Developers: collaborator read access, own GitHub login. CI credentials are automated: no long-lived personal tokens or hand-copied keys (`GITHUB_TOKEN` can't read another private repo). - -### CI credential: GitHub App - -Templates: ecu-corpus `tools/consumer-ci/`. - -- App `nyetlabs-corpus-reader`, owned by the `nyetlabs` organization: Contents read only, no webhooks, installed only on `ecu-corpus`. -- Organization Actions variable `CORPUS_APP_CLIENT_ID` (client ID; `actions/create-github-app-token` v3 deprecates the numeric App ID) and secret `CORPUS_APP_KEY` (private key), visible to all its repos. A consumer outside the organization gets repo-level copies under the same names. -- Each run mints a token (`create-github-app-token` v3, `client-id`, owner `nyetlabs`, repository `ecu-corpus`; about an hour), points git at it for the `ecu-corpus` URL only (`url..insteadOf`), then runs `git submodule update --init --checkout corpus`. A workflow reusing the build through `workflow_call` (release) must pass `secrets: inherit`. -- `tools/grant-ci-access` sets the variable and secret (`gh variable set`, `gh secret set`): no arguments for the organization, repo names for outside consumers. -- Rotation: new key in the app settings (web only, as far as known), re-run `grant-ci-access`, delete the old key. Revocation: delete the key or uninstall the app. -- Bumps: a scheduled workflow per consumer moves the submodule to the corpus head and opens a pull request. Not Dependabot: its secrets are static, so it can't mint the app token. +`.github/workflows/build.yml` checks out `corpus/` with a short-lived token from the `nyetlabs-corpus-reader` GitHub App (organization variable `CORPUS_APP_CLIENT_ID`, secret `CORPUS_APP_KEY`), for the `ecu-corpus` URL only. Fork PRs have no secrets and skip. `.github/workflows/corpus-bump.yml` moves the submodule to the corpus head and opens a pull request. `release.yml` reuses the build and must pass `secrets: inherit`. diff --git a/docs/design.md b/docs/design.md index d694c30..0664c96 100644 --- a/docs/design.md +++ b/docs/design.md @@ -13,14 +13,14 @@ Converts map definitions between WinOLS KP, A2L, DAMOS, TunerPro XDF, WinOLS scr - KP orientation is kept: TunerPro shows tables wider than 256 columns (MLHFM, 1 by 512). The KP axis "mirror" setting is not applied (no XDF equivalent known). - `author` only when the source has one; KP has none. TunerPro loads XDFs without it. - Targets TunerPro 5.00.9813 (2023) or later: older builds misload category numbers above 255 (8D0907551G has 323), and builds before 5.00.9503 crash on element text over 0x400 bytes in XDFs TunerPro didn't write. TunerPro 5.0.10305 crashes on load on any attribute value over 255 bytes (it copies each into a 256-byte buffer with `strcpy_s`); tests check every pack's XDF for it. - - Matches mapdump's XDF, compared as parsed XML, on the 13 publishable packs, except: no "Written" date comment, LF line ends in text, no units on axes the model lacks, empty text elements left out, empty elements written as start and end tags, a category repeated by name keeps its objects, float cells flagged 0x10000 (unconfirmed; no pack has float cells), conversion offsets below 0.5e-6 left out of the equation (junk such as -7.31383E-307 in some DAMOS exports is 325 characters in plain notation), and linked axes: an image axis whose breakpoints are another curve (a 1D object at the axis address with as many cells, the same storage and the same conversion up to KP's rounding) gets `embedinfo type="3"` with that object's uniqueid and keeps its own `EMBEDDEDDATA`, which the reader uses. TunerPro 5.0.10305 shows the links and carries an edit made in either the map's axis or the linked curve to the other (8D0907551M, 06A906032HS). An XDF it saves (format 1.80) keeps the links and each axis's `EMBEDDEDDATA`, and reads back to the same model, exactly with the metadata file. Six-digit precision with mapdump's arithmetic, including its Java int shift that measures 32-bit unsigned cells at raw 0. TunerPro displays 8D0907551M and 8D0907558E (with "subtract" axis labels). - - CSV map list (output only), in ecuxplot mapdump's format: one row per object in address order, value ranges read from the flash image, and a column per reference definition with the names of its maps whose id matches. It replaces mapdump for the published definitions (`mapcsv/`, `xdfkit -f csv -i image -r ref`). Differences from mapdump: axis columns follow the organisation (`kp-format.md`), objects without cells get "-" in the value columns, the raw range of 32-bit cells is masked to 32 bits instead of printed as 0x0, and float cells are read as floats. + - Matches mapdump's XDF, compared as parsed XML, on the 13 archived packs, except: no "Written" date comment, LF line ends in text, no units on axes the model lacks, empty text elements left out, empty elements written as start and end tags, a category repeated by name keeps its objects, float cells flagged 0x10000 (unconfirmed; no pack has float cells), conversion offsets below 0.5e-6 left out of the equation (junk such as -7.31383E-307 in some DAMOS exports is 325 characters in plain notation), and linked axes: an image axis whose breakpoints are another curve (a 1D object at the axis address with as many cells, the same storage and the same conversion up to KP's rounding) gets `embedinfo type="3"` with that object's uniqueid and keeps its own `EMBEDDEDDATA`, which the reader uses. TunerPro 5.0.10305 shows the links and carries an edit made in either the map's axis or the linked curve to the other (8D0907551M, 06A906032HS). An XDF it saves (format 1.80) keeps the links and each axis's `EMBEDDEDDATA`, and reads back to the same model, exactly with the metadata file. Six-digit precision with mapdump's arithmetic, including its Java int shift that measures 32-bit unsigned cells at raw 0. TunerPro displays 8D0907551M and 8D0907558E (with "subtract" axis labels). + - CSV map list (output only), in ecuxplot mapdump's format: one row per object in address order, value ranges read from the flash image, and a column per reference definition with the names of its maps whose id matches. It replaces mapdump (`mapcsv/`, `xdfkit -f csv -i image -r ref`). Differences from mapdump: axis columns follow the organisation (`kp-format.md`), objects without cells get "-" in the value columns, the raw range of 32-bit cells is masked to 32 bits instead of printed as 0x0, and float cells are read as floats. - Edit stamp: every JSON output (model and metadata file) and every XDF carries SHA-256 digests of its canonical text (RFC 8785 and `jq -S .`; for XDF, of the JSON view its elements parse to), so readers can tell whether the data was hand edited after xdfkit wrote it. See `stamp-and-metadata.md`. - Inputs: KP v1 (header codes 0x71/0x74) and KP v2 (0x124/0x149), exactly as ecuxplot's mapdump reads them (`org.nyet.mappack.Parser` in [ecuxplot](https://github.com/nyetlabs/ecuxplot)); A2L; DAMOS `.dam`; canonical JSON or YAML; XDF, with its metadata file if there is one (implemented), or an embedded model block (not yet). - Users are assumed to have no licensed WinOLS options. Two ways into WinOLS remain: - KP files (lossless, but the format is undocumented). - WinOLS scripts: a documented text format (help topic "Importing with scripts", `HelpEn.chm`) that creates maps through `set_map_property`. Scripts are a core feature; the help mentions licences only for checksums and DAMOS. A script can only insert maps, not modify existing ones, so autocorrect still needs the KP writer. For converting A2L/DAMOS/XDF into WinOLS, the script writer replaces synthesized KP (stage 2) as the primary route. -- Reader order: A2L before DAMOS. A2L follows a published spec (no `?` in `format-matrix.md`), so it proves the model as a pivot end to end soonest; it needs the model's `image` identity and the A2L parts (shared axes, conversions, record layouts) first. DAMOS has no spec and the most unknowns (19 `?`), so it follows once sample files and a field mapping against matching images exist. +- Reader order: A2L before DAMOS. A2L follows a public spec (no `?` in `format-matrix.md`), so it proves the model as a pivot end to end soonest; it needs the model's `image` identity and the A2L parts (shared axes, conversions, record layouts) first. DAMOS has no spec and the most unknowns (19 `?`), so it follows once sample files and a field mapping against matching images exist. - No A2L or DAMOS writers. In every WinOLS version checked (1.222 and 1.505 manuals, current product pages), DAMOS and A2L can only be imported, and only through the licensed Damos plugin (now OLS521, €785). The import is one-way, mostly limited to 1D/2D maps, and in EVC's words uses heuristics, so "the result isn't 100% safe". ## Implementation @@ -34,7 +34,7 @@ Converts map definitions between WinOLS KP, A2L, DAMOS, TunerPro XDF, WinOLS scr - C ABI: one generic call, `xdfkit_call(method, request JSON, input bytes)`, returning output bytes and response JSON, plus a free function and an ABI version call. New methods never change the ABI. UTF-8 strings and byte buffers with explicit lengths; output allocated by the library and released with its free function; no Go pointers kept by the caller and no callbacks; the JSON documents carry their schema version. - The project ships the C header and the shared libraries only. Language wrappers (.NET P/Invoke, Python ctypes or cffi, Java JNA or the FFM API) live with the apps that use them. - Known limits of Go shared libraries: each library carries its own Go runtime (several MB), loading two Go-built shared libraries into one process is unreliable, and building for each OS needs that OS's C toolchain (native CI runners). -- Licence: LGPL-3.0-or-later for the whole repo (`COPYING.LESSER`, with the GPL text it builds on in `COPYING`), so any program may link the shared library while changes to xdfkit itself stay copyleft. `kp/` is ported from ecuxplot's GPL-3.0 `org.nyet.mappack`, whose author relicenses the port. The corpus has its own licence (`corpus.md`). +- Licence: LGPL-3.0-or-later for the whole repo (`COPYING.LESSER`, with the GPL text it builds on in `COPYING`), so any program may link the shared library while changes to xdfkit itself stay copyleft. `kp/` is ported from ecuxplot's GPL-3.0 `org.nyet.mappack`, whose author relicenses the port. - CLI naming: conversions are named `XXX2YYY` (input format, `2`, output format), for example `kp2json`, `json2kp`, `xdf2ols`. One binary serves them all: `xdfkit kp2json FILE...` works on every platform, and the binary also dispatches on its own name (ignoring a `.exe` suffix), so a link named `kp2json` behaves like `xdfkit kp2json`. Each FILE becomes the same name with the output extension, next to it; an existing file is never replaced, XDF output writes its metadata file only with `-m`, and XDF input reads `NAME.meta.json` beside it. XDF output takes the flash image with `-i`. Options follow CLI options below. `make build` links them in `build/`; the Unix release archives hold the symlinks, and the Windows zip ships one-line `.cmd` shims (`@"%~dp0xdfkit.exe" kp2json %*`) instead, since Windows symlinks need extra privileges and zip files can't hold links. An `XXX2YYY` command rejects input that isn't format `XXX`. The plain form (`xdfkit [-f FMT] input [output]`, input format detected from the contents) stays. The commands are `xdf2kp`, `kp2xdf` and `kp2json` (`cmd/xdfkit/alias.go`, list in `ALIASES` in the Makefile; `make check-links` runs the links); other pairs are not implemented yet. On Unix, `xdfkit install-links DIR` creates the symlinks (not implemented yet). - CLI options: a single hyphen introduces single-letter options only (`-f xdf`, `-fxdf`, `-mi image`); long options take two hyphens (`--force`, `--template`, `--format`). A long option spelled with one hyphen (`-force`) is an error that names the two-hyphen spelling. Every single-letter option also has a long name; options may come before or after the files. The CLI parses options with `github.com/spf13/pflag`. Flags given to `go test` (`-run`, `-update`) follow `go test`'s convention. - Project name: `xdfkit` (Go module `go.nyet.org/xdfkit`, repo `github.com/nyetlabs/xdfkit`). @@ -106,7 +106,7 @@ flowchart LR ## Verification -- KP to XDF matches `mapdump -x -i bin`, compared as parsed XML, for every publishable archived ecuxplot KP and its image, except the deliberate differences above (`go test ./xdf` with `XDFKIT_MAPDUMP_DIR` set to mapdump's output, as written by `make -C publish` before it switched to xdfkit). +- KP to XDF matches `mapdump -x -i bin`, compared as parsed XML, for every archived ecuxplot KP and its image, except the deliberate differences above (`go test ./xdf` with `XDFKIT_MAPDUMP_DIR` set to mapdump's output). - KP to JSON to KP: byte-identical for v1; identical inflated block and container fields for v2 (the stage 1 gate). - Round trip: for every test input, input to JSON to model to JSON is identical, and JSON to YAML to JSON is identical. So is input to XDF (with embedded block) to model to JSON, including test strings containing `--`, `-->`, `