Get LinkML-derived Dandiset JSON Schema working with Meditor / dandi-archive frontend

[candleindark]

Goal

Get the Dandiset JSON Schema generated from the auto-translated LinkML schema (dandischema/models_linkml/dandiset.json on the linkml-auto-converted branch) working with the Meditor metadata editor in dandi-archive — i.e. behaving the same way the current Pydantic-generated schema does in production at https://dandiarchive.org/.

When the LinkML-derived schema is served to the frontend in place of the Pydantic-generated one, several gaps are visible (extra/missing tabs and fields, mismatched titles, broken ADD ITEM flows on the discriminated-union tabs, etc.). The remaining gaps need fixes either in pydantic2linkml (which auto-translates dandischema.models → models.yaml), in upstream linkml's JSON Schema generator, or in the dandi-archive frontend itself. This issue tracks those.

Tasks

dandi-archive frontend

• Adjust the dandi-archive frontend (Meditor) to render the LinkML-derived Dandiset JSON Schema. (in progress)
  Even with all the schema-side fixes below in place, a few LinkML-vs-Pydantic shape differences need accommodation in the consumer: nullable fields are emitted as type: [X, "null"] unions instead of a single-string type (Meditor's basic/complex/array/object classification has to recognize these); some $defs are recursive (PropertyValue.valueReference → PropertyValue) and break a JSON-stringify-based deep copy and Vue's reactivity proxy if dereferenced into JS cycles; the LinkML schema declares $schema: ".../draft/2019-09/schema" (Ajv 2020 trips on the unfamiliar dialect) and a top-level $id (Ajv complains about duplicate id when the basic and complex slices both retain it); regex pattern values use Python-flavored syntax (\Z, {,N}) that JS RegExp can't compile; and the pydantic2linkml "Unions as ranges" encoding produces {$ref: "#/$defs/Any", anyOf: [...]} which after dereference becomes an over-specified hybrid that breaks json-layout's fresh-form compile (e.g. ADD ITEM). Lives on a working branch of dandi/dandi-archive; will be tracked separately by a PR in that repo and surfaced here.

Titles

• Improve pydantic2linkml to emit missing/correct titles for Dandiset slots.
  After regenerating with the title-emitting gen-json-schema --title-from title, only 6 of the 25 top-level Dandiset properties have the right title; 17 have no title and 2 (name, contributor) inherit a generic title from CommonModel.slot_usage (Title, Contributors) instead of the Dandiset-specific one (Dandiset title, Dandiset contributors). The two mismatches are flagged directly in dandischema/models.yaml by the auto-generated note pydantic2linkml: Impossible to generate slot usage entry for the {contributor,name} slot. The slot representation … has changes in value in constraint meta slots: ['required'] . — pydantic2linkml should emit a Dandiset-level slot_usage override even when the only differing constraint is required, so the per-class title carries across.

Read-only fields

• Submit upstream PR Translate Pydantic json_schema_extra={"readOnly": True} to LinkML readonly pydantic2linkml#69 to dandi/pydantic2linkml so that it translates the readOnly JSON Schema keyword expressed in the json_schema_extra param of a Pydantic field to the readonly meta slot of a LinkML slot definition.

• Submit upstream PR to linkml/jsonschemagen.py to emit readOnly. (Filed an issue JsonSchemaGenerator: emit readOnly: true for slots with a readonly meta slot value linkml/linkml#3528 and submitted a PR feat(jsonschemagen): emit readOnly: true for slots with readonly set linkml/linkml#3537 for the proposed change)
  LinkML's metamodel has a readonly slot field, but linkml/generators/jsonschemagen.py doesn't translate it into the JSON Schema readOnly keyword (only summarygen.py reads it today). Two-line fix analogous to the existing prop.add_keyword("title", slot.title) block. Required so that fields the Pydantic model marks as read-only (id, schemaVersion, access, url, repository, identifier, dateCreated, dateModified, citation, assetsSummary, manifestLocation, version) can be hidden from Meditor — Meditor uses VJSF's readOnlyPropertiesMode: 'hide' to suppress them. Without this, Meditor's General tab gets cluttered with read-only metadata and the access / AssetsSummary sub-tabs appear when they shouldn't.

  Verified with LinkML 1.11.0 that the readonly slot is silently dropped by gen-json-schema — it does not translate to JSON Schema's readOnly keyword (nor to any annotation, comment, or other marker). The property for bar (with readonly) is indistinguishable from baz (without).

  test_readonly.yaml

  id: https://example.org/test
  name: test
  prefixes:
    linkml: https://w3id.org/linkml/
  default_prefix: test
  imports:
    - linkml:types
  classes:
    Foo:
      attributes:
        bar:
          range: string
          readonly: "this field is read-only"
        baz:
          range: string

  gen-json-schema --title-from title test_readonly.yaml

  {
      "$defs": {
          "Foo": {
              "additionalProperties": false,
              "description": "",
              "properties": {
                  "bar": {
                      "type": [
                          "string",
                          "null"
                      ]
                  },
                  "baz": {
                      "type": [
                          "string",
                          "null"
                      ]
                  }
              },
              "title": "Foo",
              "type": "object"
          }
      },
      "$id": "https://example.org/test",
      "$schema": "https://json-schema.org/draft/2019-09/schema",
      "additionalProperties": true,
      "metamodel_version": "1.11.0",
      "title": "test",
      "type": "object",
      "version": null
  }

Discriminated unions

• Emit discriminator + oneOf for Pydantic discriminated unions in pydantic2linkml (and linkml/jsonschemagen).
  Pydantic discriminated unions (Annotated[Union[Person, Organization], Field(discriminator="schemaKey")]) emit JSON Schema oneOf + discriminator{propertyName: "schemaKey", mapping: {...}}, which Meditor uses both to render a "pick the variant" control on ADD ITEM and to auto-populate schemaKey so the new item validates. The LinkML translation currently emits any_of slot ranges that round-trip to {$ref: "#/$defs/Any", anyOf: [...]} — no discriminator. Result: clicking ADD ITEM on the Dandiset contributors and Subject matter of the dataset tabs creates an empty record that fails validation with must match a schema in "anyOf". Both pydantic2linkml (preserve the discriminator info) and linkml/jsonschemagen (emit discriminator + oneOf from that info) need changes.

• Emit JSON Schema const for fixed-value fields like schemaKey.
  Pydantic emits Literal["Person"] fields as {"const": "Person", "default": "Person", ...}. The LinkML translation emits single-value enums ({"enum": ["Person"]}) instead. This affects two places in Meditor: (1) the top-level Dandiset.schemaKey is auto-hidden by VJSF when emitted as const but renders as a one-option dropdown when emitted as a single-element enum; (2) on discriminated-union variants (Person.schemaKey, Organization.schemaKey, etc.), const + default lets VJSF auto-fill the discriminator value when ADD ITEM creates a new item — without it, ADD ITEM + SAVE ITEM fails validation because the new record has no schemaKey.

• Honor Pydantic slot_usage range narrowing (e.g. Dandiset.wasGeneratedBy → Project).
  In dandischema.models, Dandiset.wasGeneratedBy is narrowed to List[Project]; the parent CommonModel exposes the broader Activity-family. Pydantic's JSON Schema reflects the narrow form (items = $ref: Project); the LinkML translation keeps the parent's polymorphic union (anyOf: [Activity, Project, PublishActivity, Session]). The auto-generated note pydantic2linkml: Impossible to generate slot usage entry for the wasGeneratedBy slot. … changes in value in constraint meta slots: ['range'] . confirms pydantic2linkml bailed on emitting the override. It should produce the per-class slot_usage entry with the narrowed range even when the diff is in a "constraint meta slot." Without this, Associated projects → ADD ITEM fails the same way contributor/about do.

Property ordering

• Investigate Dandiset JSON Schema property ordering.
  LinkML's gen-json-schema emits top-level Dandiset properties alphabetically, while Pydantic preserves Python declaration order. Visible in Meditor as: tabs ordered Subject matter of the dataset → Dandiset contributors instead of production's Dandiset contributors → Subject matter of the dataset, and the General tab fields ordered Acknowledgement, Description, … instead of Dandiset title, Description, Study Target, License, Protocol, Keywords, Acknowledgement. Cosmetic only — but worth resolving so the LinkML-driven UI matches production. The fix likely lives upstream (have linkml honor the slot list order declared on the class).