Skip to content

fix: coerce uuid.UUID and os.PathLike to str for VARCHAR/JSON fields - #3757

Open
pangwangshu wants to merge 3 commits into
milvus-io:masterfrom
pangwangshu:wapang/issue-2917
Open

fix: coerce uuid.UUID and os.PathLike to str for VARCHAR/JSON fields#3757
pangwangshu wants to merge 3 commits into
milvus-io:masterfrom
pangwangshu:wapang/issue-2917

Conversation

@pangwangshu

Copy link
Copy Markdown

Summary

Fixes #2917. Inserting a uuid.UUID or pathlib.Path value into a VARCHAR field — or nesting one inside a JSON/dynamic field — failed instead of auto-converting to a string, as the issue requested.

#3408 attempted this by patching dtype inference only. @XuanYang-cn correctly flagged that as incomplete (comment): convert_to_str_array/entity_to_str_arr reject non-str values independently of dtype inference. This PR fixes all three points on that path:

  • orm/types.pyinfer_dtype_by_scalar_data/infer_dtype_bydata recognize UUID/PathLike as VARCHAR (previously UNKNOWN; PathLike could also crash infer_dtype_bydata via an uncaught TypeError).
  • client/entity_helper.pyconvert_to_str_array coerces UUIDstr / PathLikeos.fspath(), for both the row-insert (scalar) and column-insert (list) call paths.
  • client/entity_helper.pyconvert_to_json now handles PathLike nested in JSON/dynamic fields (previously a raw orjson TypeError).

Test plan

  • Unit tests for dtype inference, convert_to_str_array (both call paths), and convert_to_json
  • End-to-end tests via pack_field_value_to_field_data / entity_to_field_data — the real insert path, not just the helpers in isolation (the gap that made fix: infer uuid.UUID and os.PathLike as VARCHAR in dtype inference #3408 look sufficient)
  • make coverage (full unit suite) — passing
  • make integration-lite — passing
  • make lint — clean

Insert of a UUID or Path value into a VARCHAR field, or nested inside a
JSON/dynamic field, previously failed instead of auto-converting to a
string as requested in milvus-io#2917. A prior attempt (milvus-io#3408) only patched dtype
inference, which left the actual VARCHAR serialization boundary
(convert_to_str_array/entity_to_str_arr) rejecting these values with
ParamError; this covers dtype inference, the VARCHAR/TEXT/GEOMETRY string
boundary (both scalar row-based and list column-based insert paths), and
JSON/dynamic-field serialization, with end-to-end regression tests for
each.

Signed-off-by: Wangshu Pang <pangwangshu@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pangwangshu
To complete the pull request process, please assign czs007 after the PR has been reviewed.
You can assign the PR to them by writing /assign @czs007 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot

Copy link
Copy Markdown

Welcome @pangwangshu! It looks like this is your first PR to milvus-io/pymilvus 🎉

@mergify

mergify Bot commented Aug 12, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Comment thread pymilvus/client/entity_helper.py Outdated
Comment thread pymilvus/client/entity_helper.py Outdated
…cion

Addresses review from @yhmo on milvus-io#3757:
- preprocess_numpy_types (convert_to_json) only descended into dict/list,
  so a PathLike nested in a tuple was left unconverted even though orjson
  accepts tuples as JSON arrays. Tuples are now traversed the same way,
  staged as a list since JSON has no tuple/list distinction.
- os.PathLike.__fspath__() may return bytes, not just str. Both coercion
  points (convert_to_json's PathLike branch and entity_helper's
  _coerce_str_like) used os.fspath(), which left bytes untouched and
  still unusable; switched to os.fsdecode() to normalize to text.

Signed-off-by: Wangshu Pang <pangwangshu@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@shashvat-singham shashvat-singham left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran the branch. The VARCHAR side does what it says, and the tuple fix in the walker is a nice catch — (PurePosixPath("/a/b"),) really did stay unconverted before.

I went looking for a uuid.UUID branch in preprocess_numpy_types to match the os.PathLike one and couldn't find it, assumed that was an oversight, then tested it and found it works anyway — orjson serialises UUID natively, so the walker doesn't need to touch it. Worth noting in case a future reader has the same reaction I did.

That does leave one real gap, though: it works because orjson handles it, and orjson isn't always the serialiser. convert_to_json falls back to stdlib json past orjson's ~500-level recursion limit, and stdlib json has no native UUID support:

def nest(leaf, depth):
    o = leaf
    for _ in range(depth):
        o = {"n": o}
    return o

convert_to_json(nest(uuid.UUID(int=0), 10))    # b'{"n":...{"n":"00000000-..."}}'
convert_to_json(nest(uuid.UUID(int=0), 600))   # TypeError: Object of type UUID is not JSON serializable
convert_to_json(nest(PurePosixPath("/a/b"), 600))  # fine

PathLike survives the deep case precisely because this PR normalises it to str in the walker, before either serialiser sees it. UUID doesn't, so a JSON field nested past ~500 levels still raises the raw TypeError that GH-2917 is about.

To be clear this fails the same way on master, so it's not a regression from this PR — but since the PR's stated goal is making UUID and PathLike work in VARCHAR and JSON fields, and it fixes the deep case for one and not the other, it seems worth closing here. Adding

elif isinstance(current, uuid.UUID):
    assign_to_parent(str(current))

next to the PathLike branch would make both paths serialiser-independent and match the reasoning already in that comment block.

Admittedly 500-deep JSON is an unusual payload, so if you'd rather keep this PR tight I'm happy to send that as a follow-up — just flagging it while the context is fresh.

@mergify mergify Bot added needs-dco and removed dco-passed labels Aug 17, 2026
@pangwangshu

Copy link
Copy Markdown
Author

@shashvat-singham Good catch, thanks for digging into the fallback path. Added an explicit uuid.UUID branch next to the PathLike one in preprocess_numpy_types so it's normalized to str before either serializer sees it, instead of relying on orjson's native support: 07cfff7

Verified with your exact repro — convert_to_json(nest(uuid.UUID(int=0), 600)) now succeeds instead of raising TypeError, and added a regression test covering the >500-level stdlib json fallback case.

…enarios

Signed-off-by: Wangshu Pang <pangwangshu@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: type support issue : unrecognized dtype for key (WindwosPath、UUID)

4 participants