Arabic Phonetiser Script

This Python script phonetises Arabic text from various sources and outputs the phoneme sequences. It uses the phonetiser library.

Features

• Input Flexibility: Process Hugging Face datasets, CSV files (with custom columns), or plain text files.
• Output Flexibility: Save to CSV, plain text, or push directly to Hugging Face Hub.
• Configurable: Specify text column names for Hugging Face datasets and CSVs.
• Error Handling: Provides clear messages for missing inputs or columns.

Prerequisites

• Python 3.8+
• pip

Installation

1. Clone the repository:

   git clone https://github.com/mostafashahin/MSA_phonetiser.git
   cd MSA_phonetiser
   

2. Install packages:

   pip install datasets pandas
   

3. Hugging Face Login (if pushing to Hub):

   huggingface-cli login
   

Usage

Run the script with command-line arguments:

python phonetiser_tool.py --input_type [huggingface|csv|text] \
                         --input_path <input_data> \
                         --output_type [csv|text|huggingface] \
                         [--output_path <output_file>] \
                         [--column_id <csv_id_col>] \
                         [--column_text <csv_text_col>] \
                         [--hf_text_column <hf_text_col>] \
                         [--hf_output_repo_id <hf_repo_id>]

Argument Descriptions

• --input_type: huggingface, csv, or text (Required)
• --input_path: Path to input file or Hugging Face dataset name (Required)
• --output_type: csv, text, or huggingface (Required)
• --output_path: Output file path (for csv/text output)
• --column_id: CSV ID column name (default: ID)
• --column_text: CSV Arabic text column name (default: Arabic text)
• --hf_text_column: Hugging Face dataset's Arabic text column name (infers if not provided)
• --hf_output_repo_id: Hugging Face repo ID for output (for huggingface output)

Examples

1. Hugging Face to CSV

   python phonetiser_tool.py --input_type huggingface --input_path "IqraEval/Iqra_train" \
       --hf_text_column "tashkeel_sentence" --output_type csv --output_path "output_phonemes.csv"
   

2. Hugging Face to Hugging Face Hub

   python phonetiser_tool.py --input_type huggingface --input_path "IqraEval/Iqra_train" \
       --hf_text_column "tashkeel_sentence" --output_type huggingface \
       --hf_output_repo_id "your-huggingface-username/my-phonetised-arabic-dataset"
   

   (Ensure you're logged into Hugging Face CLI)

3. Custom CSV to Text File

   # my_texts.csv: record_id,arabic_sentence_data
   python phonetiser_tool.py --input_type csv --input_path "sample.csv" \
       --column_id "record_id" --column_text "arabic_sentence_data" \
       --output_type text --output_path "phoneme_output.txt"
   

4. Plain Text File to CSV

   # input_sentences.txt: one sentence per line
   python phonetiser_tool.py --input_type text --input_path "input_sentences.txt" \
       --output_type csv --output_path "phonetised_from_text.csv"
   

Important Notes

• Relies on the phonetiser.phonetise_Arabic module.
• Hugging Face authentication is required for Hub pushes.
• Verify all column names carefully.