Skip to content

Commit 65f5c2b

Browse files
authored
Add Wikipedia Persian Dataset (#3629)
Currently, the Open-assistant model doesn't support Farsi. This is a text-only dataset to learn Farsi (Persian). One of my friends fine-tuned LLaMa on this dataset and It could understand Farsi grammar and word usage very well. If the Open-assistant team wants to add support to Farsi, this should be the first step. I have transformed the dataset into the standard that has been mentioned [here](https://projects.laion.ai/Open-Assistant/docs/data/datasets) and uploaded it to [my huggingface account](https://huggingface.co/datasets/pourmand1376/fa-wikipedia). - #2974
1 parent aa96100 commit 65f5c2b

2 files changed

Lines changed: 7 additions & 0 deletions

File tree

data/datasets/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
"tv_dialogue": "sedthh/tv_dialogue", # TV and Movie dialogues and transcripts
55
"fd_dialogue": "sedthh/fd_dialogue", # TV and Movie dialogues and transcripts from ForeverDreaming
66
"tlcv2.0_oa": "pythainlp/tlcv2.0_oa", # Thai classical literature texts
7+
"fa-wikipedia": "pourmand1376/fa-wikipedia", # Farsi Wikipedia texts
78
}
89

910
INSTRUCTION_DATASETS = {
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
This dataset is crawled from
2+
[farsi wikipedia](https://fa.wikipedia.org/wiki/%D8%B5%D9%81%D8%AD%D9%87%D9%94_%D8%A7%D8%B5%D9%84%DB%8C).
3+
This is valuable clean text data in persian (Farsi). It contains information
4+
about all subjects.
5+
6+
It has 2.53M Articles.

0 commit comments

Comments
 (0)