Handle LinkML migration issue of `pydantic2linkml: Impossible to generate slot usage entry for the`

[candleindark]

pydantic2link has the complains about not able to generate some slot usage entries in generation of some class definitions

such as in the translation of the Dandiset class

Stats of such complains

Total: 54 count
Ones about the schemaKey usage slots at different class defintions: 41 counts

Cause

pydantic2link unable to generate the usage slot cause by the fact that LinkML schema are monotonic. Restrictions from a slot usage definition are only additive and can not override the restrictions in the corresponding slot definition.

TODOs:

• Come up with a solution to handle each of these complains in the generated LinkML schema
  • The ones about schemaKey should be able to be handled the same way (do these first)
    • make the schemaKey slot be a type designator (using the _merge.yaml)
    • remove the ifabsent value in the schemaKey slot (this value is not relevant in a type designator; using the he _merge.yaml))
    • remove all the existing notes of Impossible to generate slot usage entry for the SchemaKey appended by pydantic2linkml
    • make schemaKey required (using the _merge.yaml). Doing this ensure new data coming in have the schemaKey field.
    • note: if _merge.yaml was merged "first" while processing subclasses, and if default value differs from the class name - we wanted to add notes: about that divergence
      • workaround: add manually to _merge.yaml file those notes for all those classes where we overload schemaKey with the value that is not the same as class name
    • Figure out why or what for we had schemaKey 'Dandiset' for PublishedDandiset and so on
      • check what class loading of a published dandiset manifest record results in (I guess should be PublishedDandiset)
      • check with @dandi/dandi-archive people how schemaKey is used in frontend (e.g. meditor) and whether it must remain e.g. "Dandiset" for "PublishedDandiset"

[candleindark]

Note: Inline definition of a slot via the attributes meta slot does
not change its behavior under inheritance. All slots are inherited the
same way regardless of whether they are defined inline within a class
(via attributes) or at the schema level (via top-level slots
referenced by the class's slots list).

Concretely, in a child class declared with is_a: Parent:

• Inherited slots are required if the parent declared them required.
• Inherited slots carry their parent-declared range and other
  constraints.
• The child can add new slots, and can extend inherited ones via
  slot_usage under the same rules that apply to schema-level slots.

Verified with linkml-validate against a minimal schema and three data
instances.

schema.yaml

id: https://example.org/inherit-demo
name: inherit_demo
prefixes:
  linkml: https://w3id.org/linkml/
  ex: https://example.org/
default_prefix: ex
default_range: string
imports:
  - linkml:types

classes:
  Person:
    attributes:
      name:
        required: true
      age:
        range: integer

  Employee:
    is_a: Person
    attributes:
      employer:
        required: true

employee_ok.yaml — valid Employee

name: Alice
age: 30
employer: Acme

$ linkml-validate -s schema.yaml -C Employee employee_ok.yaml
No issues found

employee_missing_inherited.yaml — missing inherited required slot name

employer: Acme
age: 30

$ linkml-validate -s schema.yaml -C Employee employee_missing_inherited.yaml
[ERROR] [.../employee_missing_inherited.yaml/0] 'name' is a required property in /

employee_bad_inherited_type.yaml — violates inherited range on age

name: Bob
age: not-a-number
employer: Acme

$ linkml-validate -s schema.yaml -C Employee employee_bad_inherited_type.yaml
[ERROR] [.../employee_bad_inherited_type.yaml/0] 'not-a-number' is not of type 'integer', 'null' in /age

[yarikoptic]

try to just make it more restrictive (to go from required: False into required: True to be ok). If it works for json schema and pydantic -- let's proceed that way and just add a test which checks that it remains so with current version of linkml.

If it all of a sudden stops working, which we detect via CI like we do in dandi-cli running against dev

    - name: Install dev versions of select dependencies
      if: matrix.mode == 'dev-deps'
      run: |
        pip install git+https://github.com/jaraco/keyring
        pip install git+https://github.com/NeurodataWithoutBorders/nwbinspector
        pip install git+https://github.com/NeurodataWithoutBorders/pynwb \
            git+https://github.com/hdmf-dev/hdmf \
            git+https://github.com/hdmf-dev/hdmf-zarr

then we will look into a "solution" (like screaming at upstream for breaking behavior) or come with a workaround. Let's just establish version ceiling for linkml dependency right away within e.g. major version.

[candleindark]

try to just make it more restrictive (to go from required: False into required: True to be ok). If it works for json schema and pydantic -- let's proceed that way and just add a test which checks that it remains so with current version of linkml.

If it all of a sudden stops working, which we detect via CI like we do in dandi-cli running against dev

    - name: Install dev versions of select dependencies
      if: matrix.mode == 'dev-deps'
      run: |
        pip install git+https://github.com/jaraco/keyring
        pip install git+https://github.com/NeurodataWithoutBorders/nwbinspector
        pip install git+https://github.com/NeurodataWithoutBorders/pynwb \
            git+https://github.com/hdmf-dev/hdmf \
            git+https://github.com/hdmf-dev/hdmf-zarr

then we will look into a "solution" (like screaming at upstream for breaking behavior) or come with a workaround. Let's just establish version ceiling for linkml dependency right away within e.g. major version.

This issue is codified as #405. The proposed translation to LinkML is implemented in pydantic2linkml via dandi/pydantic2linkml#66 and the continuous verification of the LinkML behavior against regression has been implemented via #408.