hi, i've stumbled on your posts on reddit, and was looking tokenzier for a similar reason, (to create an eclipse plugin), but is there a way to get the actuall list of tokens and their positions and length in the original text?
like the SimpleTokenizer in openNLP tools the outputs Span
hi, i've stumbled on your posts on reddit, and was looking tokenzier for a similar reason, (to create an eclipse plugin), but is there a way to get the actuall list of tokens and their positions and length in the original text?
like the SimpleTokenizer in openNLP tools the outputs Span