A comprehensive suite of high-level NLP tasks for Persian language. The dataset consists of the following tasks: Text entailment, Query paraphrasing, Reading comprehension, Multiple-choice QA, Machine translation and Sentiment analysis. They've been also fine-tuned mt5 models on these datasets which result in various Persian models.
ParsBench provides toolkits for benchmarking LLMs based on the Persian language tasks.
- ParsiNLU all tasks
- Persian NER
- Persian Math
- ConjNLI Entailment
- Persian MMLU (khayyam Chanllenge)
Benchmarking ChatGPT for Persian: A Preliminary Study
- Elemntry school
- Mathematical problems dataset
10k pairs with entailment label.
Utilizes the FarsTail dataset for fine-tuning its ParsBERT model, while also incorporating two other entailment datasets: Wiki Triplet and Wiki D/Similar.
Persian NLP team trained various mt5 and BERT models on their entailment dataset.
2.7k Relation of entities with translation and relation type.
It is a large Multilingual Dataset for Entity Linking containing data in 53 languages including Persian. DaMuEL consists of two components: a knowledge base that contains language-agnostic information about entities, including their claims from Wikidata and named entity types (PER, ORG, LOC, EVENT, BRAND, WORK_OF_ART, MANUFACTURED); and Wikipedia texts with entity mentions linked to the knowledge base, along with language-specific text from Wikidata such as labels, aliases, and descriptions, stored separately for each language. Paper. For this project UDPipe has been used.
It is a knowledge graph platform designed for extracting information from Wikipedia, tables, and unstructured texts. A portion of its data is also available for download.
Open information extraction from Persian web.
The Persian Simple Question Answering Dataset and System over Knowledge Graph. It consists of 36k records.
It is a dataset for Persian fact extraction and verification, developed in accordance with FEVER guidelines.
It includes detailed product information generated based on the title of each product, aiming to create a structured catalog in JSON format. The dataset encompasses a variety of product categories such as food, home and kitchen, clothing, handicrafts, tools, automotive equipment, and more.
Pre-trained word vectors of 157 languages including Persian, trained on CommonCrawl and Wikipedia using CBOW.
A tutorial on how to use 3 word embeddings; a) Downloading and using fasttext Persian word embeddings. b) How to get word embeddings of ParsBERT base model itself. c) How to get word embeddings of ParsGPT model.
A Persian Word2Vec Model trained by Wikipedia articles
Three similar models based on fine-tuning ParsBERT base model on 3 different entailment datasets. Each of these models can be used for Semantic Search, Clustering, Summerization, Information retrieval and Topic Modeling tasks.
Family of ParsBERT models including BERT, DistilBERT, ALBERT and ROBERTA. All of which are transformer based models with encoder-decoder design.
Multilingual BERT model consists of 104 languages including Persian.
Is a BERT based model trained on Divan dataset (proprietary). This model has 46.6M parameters. Its evaluation on NER and Sentiment Analysis is repoted.
Is a BERT based model trained on Divan dataset (proprietary). This model has 124M parameters. Its evaluation on NER and Sentiment Analysis is repoted.
Is a Persian BERT model trained on various Persian texts.
Is a Persian BERT model trained on various Persian texts.
Is a Persian BERT model trained on various Persian texts with 123M parameters. There is also a large version of this model with 353M parameters.