Replace regex-based tokenizer

[blackwinter]

Replace current tokenizer with a more robust and more flexible solution. Flexibility in terms of implementation, not necessarily customization. Based on Racc/Ragel/ANTLR/...?