Skip to content

Data loading/extraction pipeline #4

Description

@borh

Pinned issue on data ingestion/normalization pipeline.
As we want to keep the system as open as possible, we focus on open data as far as possible.

  • Investigate using wtpsplit for sentence segmentation for datasets with ambiguous sentence/paragraph boundaries
  • Investigate topic modeling extraction to be potentially used to show the topic breakdown of a collocation
  • Integrate Huggingface datasets support. Initial implementation should load the Japanese Wikipedia from wikimedia/wikipedia, cf. https://huggingface.co/datasets/wikimedia/wikipedia?library=datasets --when implementing, use the streaming=True option of load_dataset to only load a sample for testing.
  • Fix extraction bugs #8
  • Expand extraction of current case relation triplets
  • Expand extraction to 2-5 gram relations

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions