I'm using xee for a CLI that deals with fairly large arrays of nested maps (100+ elements).
One thing I noticed is that wildcard lookups suffer a huge performance degradation when nested. For example, using a map structured like the following (full file generated.json, loaded in through fn:parse-json)
[
{
"foo": 0,
"nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}]
},
{
"foo": 1,
"nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}]
},
...
{
"foo": 99,
"nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}]
}
]
A naive wildcard query ?*nested?*?foo takes ~30 seconds on my machine, while something like a call to map:find(?*?nested, "foo") takes 2 milliseconds. With larger arrays similar lookups take tens of minutes. I tested this out in xquery and basex as well to compare and they don't have a noticeable performance hit
I ran a query through a profiler and as far as I could tell this is due to a large number of allocations made in wildcard_helper and lookup_helper . I am working on a MR to improve the performance and I was wondering if you had an idea on how to approach this
Thank you!
I'm using xee for a CLI that deals with fairly large arrays of nested maps (100+ elements).
One thing I noticed is that wildcard lookups suffer a huge performance degradation when nested. For example, using a map structured like the following (full file generated.json, loaded in through fn:parse-json)
[ { "foo": 0, "nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}] }, { "foo": 1, "nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}] }, ... { "foo": 99, "nested": [{"foo": 0}, {"foo": 1}, {"foo": 2}] } ]A naive wildcard query
?*nested?*?footakes ~30 seconds on my machine, while something like a call tomap:find(?*?nested, "foo")takes 2 milliseconds. With larger arrays similar lookups take tens of minutes. I tested this out in xquery and basex as well to compare and they don't have a noticeable performance hitI ran a query through a profiler and as far as I could tell this is due to a large number of allocations made in wildcard_helper and lookup_helper . I am working on a MR to improve the performance and I was wondering if you had an idea on how to approach this
Thank you!