You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Partial Attribute for datasets that might not include all attributes. For example, some social media datasets require re-clawring.
UnitAdd conversations in for tha attribute.
Source the source of the non-annotated data. ['web pages', 'public datasets', 'books', 'blogs', 'scientific articles', 'wikipedia', 'TV channels', 'LLM']
Test Split rename to Has_Splits
Domain is for the domain of the dataset i.e., ['Law', 'Science', 'News', 'General', 'Quran', 'Culture', 'General']
Collection Style should be either ['crawling', 'manual curation']
Annotation Style to better represent how the dataset was annotated, and add another option for no annotation, for example, public datasets are inheritly annotated annotated ['Machine', 'Human', 'Inehrited']
Define a list of venues for easier annotation
Rename Subsets to Dialect Subsets for better representation
Preprocessing rather than Tokenized this general attributes define the preprocessning steps for the dataset i.e. ['tokenized','normalized','remove diacritization', 'other']
Benchmarks are ill-defined in our definition; also, some datasets might include subsets other than dialects
PartialAttribute for datasets that might not include all attributes. For example, some social media datasets require re-clawring.UnitAdd conversations in for tha attribute.Sourcethe source of the non-annotated data.['web pages', 'public datasets', 'books', 'blogs', 'scientific articles', 'wikipedia', 'TV channels', 'LLM']Test Splitrename toHas_SplitsDomainis for the domain of the dataset i.e.,['Law', 'Science', 'News', 'General', 'Quran', 'Culture', 'General']Collection Styleshould be either['crawling', 'manual curation']Annotation Styleto better represent how the dataset was annotated, and add another option for no annotation, for example, public datasets are inheritly annotated annotated['Machine', 'Human', 'Inehrited']Preprocessingrather thanTokenizedthis general attributes define the preprocessning steps for the dataset i.e.['tokenized','normalized','remove diacritization', 'other']