pip install -r requirements.txt
For Hungarian tokenizers please follow the installation guides for emMorphPy (https://github.com/dlt-rilmta/emmorphpy) and PurePOSPy (https://github.com/dlt-rilmta/purepospy). The HFST binaries are included under data folder so there is no need to install it seperately.
Note: If on Mac OS makefile for Purepospy will not work so following the steps below should do the job;
cd purepospy/
make __extra-deps
pip install -r requirements.txt
The structure is intented to be flexible so that other tokenizers can be added without many change. The tokenizer class under tokenizer_modules is an abstract class that has already implemented most of the methods. To add your custom tokenizer
- Extending the tokenizer class and implement necessary methods
- Add your configuration file under configs
- The custom tokenizer can then be used as Tokenizer.create('configs/my-custom-tokenizer-config.json')
If you find this code useful in your research, please consider citing:
@article{10.1007/s10579-021-09568-y,
year = {2022},
title = {{Abstractive text summarization and new large-scale datasets for agglutinative languages Turkish and Hungarian}},
author = {Baykara, Batuhan and Güngör, Tunga},
journal = {Language Resources and Evaluation},
issn = {1574-020X},
doi = {10.1007/s10579-021-09568-y},
pages = {1--35}}