Full Text Search - Documentation of Limits

[erichiller]

I haven't seen this in the documentation, but following the tutorial and then inserting nodes with a large-ish string as an attribute intended for Full Text Search (query) - is there a max string length for attribute values? I was trying to to implement a hybrid of graph and full text search, but insertion alone was taking seconds per page (HTML)?

[krotik]

Hi Eric,

that is an interesting use-case which I am quite interested in myself. There is in general no maximum string length for attribute values.

The index (full text search index) is word based so an input string is split into words (split by whitespace). The words are then stored in two ways:

node attribute name + word -> node keys + word positions
node attribute name + md5 hash of value -> node keys

First one for word and phrase search - second one for value lookup.
The code for this can be found under /eliasdb/graph/util/indexmanager.go
Have a look at the unit test to see how this component works...

Since a lot of HTML doesn't have spaces between tags I would imagine that the words get quite long. It might help to chop them up a bit...

Best way forward to narrow down what exactly goes wrong would be to write a unit/benchmark for the IndexManager with some suitable test data.