You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Pinned issue on data ingestion/normalization pipeline.
As we want to keep the system as open as possible, we focus on open data as far as possible.
Investigate using wtpsplit for sentence segmentation for datasets with ambiguous sentence/paragraph boundaries
Investigate topic modeling extraction to be potentially used to show the topic breakdown of a collocation
Integrate Huggingface datasets support. Initial implementation should load the Japanese Wikipedia from wikimedia/wikipedia, cf. https://huggingface.co/datasets/wikimedia/wikipedia?library=datasets --when implementing, use the streaming=True option of load_dataset to only load a sample for testing.
Pinned issue on data ingestion/normalization pipeline.
As we want to keep the system as open as possible, we focus on open data as far as possible.
wikimedia/wikipedia, cf. https://huggingface.co/datasets/wikimedia/wikipedia?library=datasets --when implementing, use thestreaming=Trueoption ofload_datasetto only load a sample for testing.