Thoughts on parser rewrite

[priv-kweihmann]

The library now reached a point where the most basic functionality works well enough but corner cases start to pile up, performance is improvable and the API feels clunky here and there.

So it's time to collect ideas and thoughts on a major rewrite of this lib

Ideas:

• rewrite in rust: should improve performance (esp as a lot of objects are created)
• switch to in-memory DB instead of lists: should improve performance, as filtering, assuming the choice of a fitting DB provider, should eliminate the need to run over lists of lists of list, like hundreds of times (e.g. GetItemsFor)
• use proper lexer to get around issues like
  • line continuation
  • in function brackets and quotes (see Prepare Lines Subparser does not detect function correctly #340)
• better tokenizer, get rid of InFileLine attribute and just create a linked list of tokens found (see Inconsistent stash.GetItemsFor(filename=_file, classifier=Inherit.CLASSIFIER) #367 for details)
  • remove InFileLine
  • remove Raw
  • remove RealRaw
  • better multiline statement handling
  • use children or links to embedded items - alternative add a Parent or EmbeddedIn attribute to the "child" tokens
• way of assembling tokens (even from the same line) back together (see Inconsistent stash.GetItemsFor(filename=_file, classifier=Inherit.CLASSIFIER) #367 for an example)

Goals:

• heavily improved performance (test case parsing oe-core's recipes)
• lighter API (just expose what is really needed), likely the low level things like File, Line should be r/o, while all other "higher" level classes should provide r/w attributes

Non-Goals:

• support for arbitrary python inline functions or arbitrary helper function inside inline functions,
  as this will likely need to compute code - which shouldn't be done for security reasons
• support for layer assembly, like layer order etc etc - this needs to be done in full build using bitbake

Feel free to comment for ideas or other things to improve