Convenient variable-granularity chunk iterator

[KnorpelSenf]

I think this crate is amazing for its focus on performance. However, it isn't particularly convenient to use yet. One reason is that only line diffs are supported out of the box. Technically, one could implement a custom tokenizer and perform more granular diffs, or even run imara repeatedly on varying chunk sizes, but that's not exactly obvious and still requires a fair bit of complexity for library consumers.

I suggest the addition of a new convenience wrapper. It should be a very simple function (maybe called diff) for the main use case of this crate: provide a diff of two human-readable text strings. It should return the diff in a format that is trivial to understand even for people who have never thought about string diffing before (i.e. it shouldn't even require a knowledge of tokens).

It could take two strings and return an iterator of the following enum:

enum Chunk<'p, 'n> {
    Insert(&'n str),
    Delete(&'p str),
    Replace(&'p str, &'n str),
    Equal(&'n str),
}

We can add a method .index() to the iterator to drag along the two byte offsets in the old and the new string.

The implementation should implement a new heuristic to balance performance and character accuracy. It could:

• perform a line diff and iterate over its hunks
• if the current hunk is a pure insertion, yield Insert
• if the current hunk is a pure deletion, yield Delete
• if the current hunk is small in size (less than, say, 100 tokens), perform a word diff (feat: word diffs #33) and iterate over its hunks
  • if the current hunk is a pure insertion, yield Insert
  • if the current hunk is a pure deletion, yield Delete
  • if the current hunk is small in size (less than, say, 100 tokens), perform a character diff (feat: character diffs #38) and yield its hunks via Insert, Delete, and Replace
  • else, yield Replace
• else, yield Replace

After every item, Equal should be interspersed.

We might want to add a more sophisticated heuristic than a simple size check, such as guessing the complexity of the diff by looking at the number of added/removed tokens and/or factoring in token size.

[pascalkuthe]

I am not convinced that character level diff and that sort of sophisticated algorith is warranted. The usecase for character diff seems niche (even very thorough diffs like delta use word-level diffs) and if you want you can implement this on top of imara-diff. Imara-diff is definitely a more low-level library than maybe others.

I could be convinced to expand the work diff feature you build to have an iterator on top that yields changed byte ranges (but I don't think string slices are a good idea as you can trivially go from byte ranges to string slices but not the other way around. That would also require the input to be actual strings which would not work with something like a Rope which was one of the motivating exapmles for this crate)

[KnorpelSenf]

@pascalkuthe alright, I will wrap this crate then instead of contributing to it. Does #38 still have a chance of landing, given that it also exposes a more low-level API for word-diffing based on custom tokenizers?