Skip to content

Wildcard lookup operator performance #149

Description

@theneosloth

I'm using xee for a CLI that deals with fairly large arrays of nested maps (100+ elements).

One thing I noticed is that wildcard lookups suffer a huge performance degradation when nested. For example, using a map structured like the following (full file generated.json, loaded in through fn:parse-json)

[
  {
    "foo": 0,
    "nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}]
  },
  {
    "foo": 1,
    "nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}]
  },
...
  {
    "foo": 99,
    "nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}]
  }
]

A naive wildcard query ?*nested?*?foo takes ~30 seconds on my machine, while something like a call to map:find(?*?nested, "foo") takes 2 milliseconds. With larger arrays similar lookups take tens of minutes. I tested this out in xquery and basex as well to compare and they don't have a noticeable performance hit

I ran a query through a profiler and as far as I could tell this is due to a large number of allocations made in wildcard_helper and lookup_helper . I am working on a MR to improve the performance and I was wondering if you had an idea on how to approach this

Image

Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions