Skip to content

Commit e0d51d1

Browse files
gagan3012gagan3012
committed
Bump version to v0.16.0 for release
1 parent 9ff6b15 commit e0d51d1

4 files changed

Lines changed: 115 additions & 113 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## Unreleased
99

10+
## [v0.16.0](https://github.com/gagan3012/PolyDeDupe/releases/tag/v0.16.0) - 2023-12-03
11+
1012
## [v0.15.0](https://github.com/gagan3012/PolyDeDupe/releases/tag/v0.15.0) - 2023-11-28
1113

1214
## [v0.14.0](https://github.com/gagan3012/PolyDeDupe/releases/tag/v0.14.0) - 2023-11-28

‎PolyDeDupe/polydedupe.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -332,7 +332,7 @@ def deduplicate_dataset(
332332
duplicate_indices = set(x["base_index"] for cluster in duplicate_clusters for x in cluster)
333333
extreme_dict = {}
334334
extremes_clusters = find_extremes(duplicate_clusters, dataset, jaccard_threshold)
335-
extremes_clusters = find_extremes_semantic(extremes_clusters, dataset, jaccard_threshold)
335+
# extremes_clusters = find_extremes_semantic(extremes_clusters, dataset, jaccard_threshold)
336336
for extremes in extremes_clusters:
337337
for element in extremes:
338338
extreme_dict[element["base_index"]] = element

‎PolyDeDupe/version.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
_MAJOR = "0"
2-
_MINOR = "15"
2+
_MINOR = "16"
33
# On main and in a nightly release the patch should be one ahead of the last
44
# released build.
55
_PATCH = "0"

‎README.md‎

Lines changed: 111 additions & 111 deletions
Original file line numberDiff line numberDiff line change
@@ -1,111 +1,111 @@
1-
# PolyDeDupe: Multi-Lingual Data Deduplication
2-
3-
![](docs/source/polyde_preview.jpg)
4-
5-
PolyDeDupe is a Python package designed for efficient and effective data deduplication across multiple languages. With support for over 100 languages, this tool stands out in its ability to perform both syntactic and semantic deduplication, ensuring high-quality data preprocessing for various NLP tasks.
6-
7-
## Features
8-
9-
- **Multi-Lingual Support**: PolyDeDupe supports over 100 languages, including English, French, German, Spanish, Portuguese, Italian, Dutch, Czech, Polish, Slovak, Hungarian, Croatian, Slovenian, Latvian, Lithuanian, Russian, Bulgarian, Serbian, Ukrainian, Belarusian, Macedonian, Greek, Arabic, Persian (Farsi), Urdu, Pashto, Kurdish (Sorani), Hindi, Marathi, Sanskrit, Nepali, Konkani, Bodo, Amharic, Tigrinya, and many more.
10-
11-
- **Syntactic and Semantic Deduplication**: PolyDeDupe performs both syntactic and semantic deduplication, ensuring high-quality data preprocessing for various NLP tasks.
12-
13-
- **Customizable Jaccard Threshold**: PolyDeDupe allows users to customize the Jaccard similarity threshold for deduplication.
14-
15-
- **2x faster than other tools**: PolyDeDupe is 2x faster than other tools, such as [SlimPajama](https://github.com/Cerebras/modelzoo/tree/main/modelzoo/transformers/data_processing/slimpajama)
16-
17-
- **Support for Instruction tuning data**: PolyDeDupe supports deduplication of instruction tuning data, which is a common use case in NLP.
18-
19-
## Installation
20-
21-
PolyDeDupe can be installed using pip:
22-
23-
```bash
24-
pip install polydedupe
25-
```
26-
27-
## Usage
28-
29-
```python
30-
from PolyDeDupe import deduplicate_dataset, display_dataset_entries
31-
from datasets import load_dataset
32-
33-
dataset = load_dataset("tatsu-lab/alpaca",split="train")
34-
ds_dedup, duplicate_clusters = deduplicate_dataset(newdataset, jaccard_threshold=0.90)
35-
display_dataset_entries(newdataset, duplicate_clusters)
36-
```
37-
38-
Output:
39-
40-
```bash
41-
Original dataset size: 52002
42-
Number of duplicate clusters: 40
43-
Files in duplicate cluster: 82
44-
Unique files in duplicate cluster: 52
45-
Filtered dataset size: 51972
46-
Cluster:
47-
Base Index: 1482, Data: {'instruction': 'Find the five largest cities in France.', 'input': '', 'output': 'The five largest cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.', 'text': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nFind the five largest cities in France.\n\n### Response:\nThe five largest cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.'}
48-
Base Index: 1820, Data: {'instruction': 'Name five cities in France.', 'input': '', 'output': 'The five cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.', 'text': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName five cities in France.\n\n### Response:\nThe five cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.'}
49-
```
50-
51-
## Supported Langauges:
52-
53-
- **Western European Languages**:
54-
- French, German, Spanish, Portuguese, Italian, Dutch, etc.
55-
56-
- **Central European and Baltic Languages**:
57-
- Czech, Polish, Slovak, Hungarian, Croatian, Slovenian, Latvian, Lithuanian, etc.
58-
59-
- **Additional European Languages**:
60-
- Additional European languages with special characters.
61-
62-
- **Vietnamese and Some African Languages**:
63-
- Vietnamese and various African languages using extended Latin characters.
64-
65-
- **Slavic Languages Using Cyrillic Script**:
66-
- Russian, Bulgarian, Serbian, Ukrainian, Belarusian, Macedonian, etc.
67-
68-
- **Greek Language**:
69-
- Modern Greek.
70-
71-
- **Arabic Language and its Variants**:
72-
- Standard Arabic, Persian (Farsi), Urdu, Pashto, Kurdish (Sorani), etc.
73-
74-
- **Languages Using the Devanagari Script**:
75-
- Hindi, Marathi, Sanskrit, Nepali, Konkani, Bodo, etc.
76-
77-
- **Ethiopic Script Languages**:
78-
- Amharic, Tigrinya, and other languages in Ethiopia and Eritrea.
79-
80-
- **Tifinagh Script for Berber Languages**:
81-
- Berber languages in North Africa.
82-
83-
- **Vai Script**:
84-
- Used for the Vai language in West Africa.
85-
86-
- **East Asian Languages**:
87-
- Chinese, Japanese, Korean.
88-
89-
- **Dravidian Languages**:
90-
- Tamil, Telugu, Kannada, Malayalam.
91-
92-
- **Indian Languages**:
93-
- Bengali, Punjabi, Gujarati, Oriya.
94-
95-
- **General Latin, Numerals, and Underscore**:
96-
- Basic Latin characters, numbers, and underscore used globally.
97-
98-
99-
# Citation:
100-
101-
```
102-
@software{Gagan_PolyDeDupe_2023,
103-
author = {Gagan, Bhatia},
104-
doi = {10.5281/zenodo.1234},
105-
month = nov,
106-
title = {{PolyDeDupe}},
107-
url = {https://github.com/gagan3012/PolyDeDupe},
108-
version = {1.0.0},
109-
year = {2023}
110-
}
111-
```
1+
# PolyDeDupe: Multi-Lingual Data Deduplication
2+
3+
![](docs/source/polyde_preview.jpg)
4+
5+
PolyDeDupe is a Python package designed for efficient and effective data deduplication across multiple languages. With support for over 100 languages, this tool stands out in its ability to perform both syntactic and semantic deduplication, ensuring high-quality data preprocessing for various NLP tasks.
6+
7+
## Features
8+
9+
- **Multi-Lingual Support**: PolyDeDupe supports over 100 languages, including English, French, German, Spanish, Portuguese, Italian, Dutch, Czech, Polish, Slovak, Hungarian, Croatian, Slovenian, Latvian, Lithuanian, Russian, Bulgarian, Serbian, Ukrainian, Belarusian, Macedonian, Greek, Arabic, Persian (Farsi), Urdu, Pashto, Kurdish (Sorani), Hindi, Marathi, Sanskrit, Nepali, Konkani, Bodo, Amharic, Tigrinya, and many more.
10+
11+
- **Syntactic and Semantic Deduplication**: PolyDeDupe performs both syntactic and semantic deduplication, ensuring high-quality data preprocessing for various NLP tasks.
12+
13+
- **Customizable Jaccard Threshold**: PolyDeDupe allows users to customize the Jaccard similarity threshold for deduplication.
14+
15+
- **2x faster than other tools**: PolyDeDupe is 2x faster than other tools, such as [SlimPajama](https://github.com/Cerebras/modelzoo/tree/main/modelzoo/transformers/data_processing/slimpajama)
16+
17+
- **Support for Instruction tuning data**: PolyDeDupe supports deduplication of instruction tuning data, which is a common use case in NLP.
18+
19+
## Installation
20+
21+
PolyDeDupe can be installed using pip:
22+
23+
```bash
24+
pip install polydedupe
25+
```
26+
27+
## Usage
28+
29+
```python
30+
from PolyDeDupe import deduplicate_dataset, display_dataset_entries
31+
from datasets import load_dataset
32+
33+
dataset = load_dataset("tatsu-lab/alpaca",split="train")
34+
ds_dedup, duplicate_clusters = deduplicate_dataset(newdataset, jaccard_threshold=0.90)
35+
display_dataset_entries(newdataset, duplicate_clusters)
36+
```
37+
38+
Output:
39+
40+
```bash
41+
Original dataset size: 52002
42+
Number of duplicate clusters: 40
43+
Files in duplicate cluster: 82
44+
Unique files in duplicate cluster: 52
45+
Filtered dataset size: 51972
46+
Cluster:
47+
Base Index: 1482, Data: {'instruction': 'Find the five largest cities in France.', 'input': '', 'output': 'The five largest cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.', 'text': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nFind the five largest cities in France.\n\n### Response:\nThe five largest cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.'}
48+
Base Index: 1820, Data: {'instruction': 'Name five cities in France.', 'input': '', 'output': 'The five cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.', 'text': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName five cities in France.\n\n### Response:\nThe five cities in France are Paris, Marseille, Lyon, Toulouse, and Nice.'}
49+
```
50+
51+
## Supported Langauges:
52+
53+
- **Western European Languages**:
54+
- French, German, Spanish, Portuguese, Italian, Dutch, etc.
55+
56+
- **Central European and Baltic Languages**:
57+
- Czech, Polish, Slovak, Hungarian, Croatian, Slovenian, Latvian, Lithuanian, etc.
58+
59+
- **Additional European Languages**:
60+
- Additional European languages with special characters.
61+
62+
- **Vietnamese and Some African Languages**:
63+
- Vietnamese and various African languages using extended Latin characters.
64+
65+
- **Slavic Languages Using Cyrillic Script**:
66+
- Russian, Bulgarian, Serbian, Ukrainian, Belarusian, Macedonian, etc.
67+
68+
- **Greek Language**:
69+
- Modern Greek.
70+
71+
- **Arabic Language and its Variants**:
72+
- Standard Arabic, Persian (Farsi), Urdu, Pashto, Kurdish (Sorani), etc.
73+
74+
- **Languages Using the Devanagari Script**:
75+
- Hindi, Marathi, Sanskrit, Nepali, Konkani, Bodo, etc.
76+
77+
- **Ethiopic Script Languages**:
78+
- Amharic, Tigrinya, and other languages in Ethiopia and Eritrea.
79+
80+
- **Tifinagh Script for Berber Languages**:
81+
- Berber languages in North Africa.
82+
83+
- **Vai Script**:
84+
- Used for the Vai language in West Africa.
85+
86+
- **East Asian Languages**:
87+
- Chinese, Japanese, Korean.
88+
89+
- **Dravidian Languages**:
90+
- Tamil, Telugu, Kannada, Malayalam.
91+
92+
- **Indian Languages**:
93+
- Bengali, Punjabi, Gujarati, Oriya.
94+
95+
- **General Latin, Numerals, and Underscore**:
96+
- Basic Latin characters, numbers, and underscore used globally.
97+
98+
99+
# Citation:
100+
101+
```
102+
@software{Gagan_PolyDeDupe_2023,
103+
author = {Gagan, Bhatia},
104+
doi = {10.5281/zenodo.1234},
105+
month = nov,
106+
title = {{PolyDeDupe}},
107+
url = {https://github.com/gagan3012/PolyDeDupe},
108+
version = {1.0.0},
109+
year = {2023}
110+
}
111+
```

0 commit comments

Comments
 (0)