finalize 4.9.0

Author: huseinzol05

README-pypi.rst

@@ -46,7 +46,7 @@ Features
 -  **Emotion Analysis**, detect and recognize 6 different emotions of texts using finetuned Transformer-Bahasa.
 -  **Entities Recognition**, seeks to locate and classify named entities mentioned in text using finetuned Transformer-Bahasa.
 -  **Generator**, generate any texts given a context using T5-Bahasa, GPT2-Bahasa or Transformer-Bahasa.
--  **Jawi-to-Rumi**, convert from Jawi to Rumi using LSTM Seq2Seq with attention state-of-art.
+-  **Jawi-to-Rumi**, convert from Jawi to Rumi using Transformer.
 -  **Keyword Extraction**, provide RAKE, TextRank and Attention Mechanism hybrid with Transformer-Bahasa.
 -  **Knowledge Graph**, generate Knowledge Graph using T5-Bahasa or parse from Dependency Parsing models.
 -  **Language Detection**, using Fast-text and Sparse Deep learning Model to classify Malay (formal and social media), Indonesia (formal and social media), Rojak language and Manglish.
@@ -57,7 +57,7 @@ Features
 -  **Part-of-Speech Recognition**, grammatical tagging is the process of marking up a word in a text using finetuned Transformer-Bahasa.
 -  **Question Answer**, reading comprehension using finetuned Transformer-Bahasa.
 -  **Relevancy Analysis**, detect and recognize relevancy of texts using finetuned Transformer-Bahasa.
--  **Rumi-to-Jawi**, convert from Rumi to Jawi using LSTM Seq2Seq with attention state-of-art.
+-  **Rumi-to-Jawi**, convert from Rumi to Jawi using Transformer.
 -  **Sentiment Analysis**, detect and recognize polarity of texts using finetuned Transformer-Bahasa.
 -  **Text Similarity**, provide interface for lexical similarity deep semantic similarity using finetuned Transformer-Bahasa.
 -  **Spell Correction**, using local Malaysia NLP researches hybrid with Transformer-Bahasa to auto-correct any bahasa words and NeuSpell using T5-Bahasa.

README.rst

@@ -65,7 +65,7 @@ Features
 -  **Emotion Analysis**, detect and recognize 6 different emotions of texts using finetuned Transformer-Bahasa.
 -  **Entities Recognition**, seeks to locate and classify named entities mentioned in text using finetuned Transformer-Bahasa.
 -  **Generator**, generate any texts given a context using T5-Bahasa, GPT2-Bahasa or Transformer-Bahasa.
--  **Jawi-to-Rumi**, convert from Jawi to Rumi using LSTM Seq2Seq with attention state-of-art.
+-  **Jawi-to-Rumi**, convert from Jawi to Rumi using Transformer.
 -  **Keyword Extraction**, provide RAKE, TextRank and Attention Mechanism hybrid with Transformer-Bahasa.
 -  **Knowledge Graph**, generate Knowledge Graph using T5-Bahasa or parse from Dependency Parsing models.
 -  **Language Detection**, using Fast-text and Sparse Deep learning Model to classify Malay (formal and social media), Indonesia (formal and social media), Rojak language and Manglish.
@@ -76,7 +76,7 @@ Features
 -  **Part-of-Speech Recognition**, grammatical tagging is the process of marking up a word in a text using finetuned Transformer-Bahasa.
 -  **Question Answer**, reading comprehension using finetuned Transformer-Bahasa.
 -  **Relevancy Analysis**, detect and recognize relevancy of texts using finetuned Transformer-Bahasa.
--  **Rumi-to-Jawi**, convert from Rumi to Jawi using LSTM Seq2Seq with attention state-of-art.
+-  **Rumi-to-Jawi**, convert from Rumi to Jawi using Transformer.
 -  **Sentiment Analysis**, detect and recognize polarity of texts using finetuned Transformer-Bahasa.
 -  **Text Similarity**, provide interface for lexical similarity deep semantic similarity using finetuned Transformer-Bahasa.
 -  **Spell Correction**, using local Malaysia NLP researches hybrid with Transformer-Bahasa to auto-correct any bahasa words and NeuSpell using T5-Bahasa.

docs/Api.rst

@@ -138,10 +138,10 @@ malaya.preprocessing
 .. automodule:: malaya.preprocessing
     :members:
 
-.. autoclass:: malaya.preprocessing.Tokenizer()
+.. autoclass:: malaya.preprocessing.Preprocessing()
     :members:
 
-.. autoclass:: malaya.preprocessing.Preprocessing()
+.. autoclass:: malaya.preprocessing.Demoji()
     :members:
 
 malaya.qa
@@ -219,6 +219,18 @@ malaya.tatabahasa
 .. automodule:: malaya.tatabahasa
     :members:
 
+malaya.tokenizer
+--------------------
+
+.. automodule:: malaya.tokenizer
+    :members:
+
+.. autoclass:: malaya.tokenizer.Tokenizer()
+    :members:
+
+.. autoclass:: malaya.tokenizer.SentenceTokenizer()
+    :members:
+
 malaya.summarization.abstractive
 ---------------------------------
 
@@ -452,7 +464,10 @@ malaya.model.tf
 .. autoclass:: malaya.model.tf.Seq2SeqLSTM()
     :members:
 
-.. autoclass:: malaya.model.tf.Seq2SeqLSTM_Split()
+.. autoclass:: malaya.model.tf.JawiRumi()
+    :members:
+
+.. autoclass:: malaya.model.tf.RumiJawi()
     :members:
 
 malaya.model.xlnet

docs/README.rst

@@ -65,7 +65,7 @@ Features
 -  **Emotion Analysis**, detect and recognize 6 different emotions of texts using finetuned Transformer-Bahasa.
 -  **Entities Recognition**, seeks to locate and classify named entities mentioned in text using finetuned Transformer-Bahasa.
 -  **Generator**, generate any texts given a context using T5-Bahasa, GPT2-Bahasa or Transformer-Bahasa.
--  **Jawi-to-Rumi**, convert from Jawi to Rumi using LSTM Seq2Seq with attention state-of-art.
+-  **Jawi-to-Rumi**, convert from Jawi to Rumi using Transformer.
 -  **Keyword Extraction**, provide RAKE, TextRank and Attention Mechanism hybrid with Transformer-Bahasa.
 -  **Knowledge Graph**, generate Knowledge Graph using T5-Bahasa or parse from Dependency Parsing models.
 -  **Language Detection**, using Fast-text and Sparse Deep learning Model to classify Malay (formal and social media), Indonesia (formal and social media), Rojak language and Manglish.
@@ -76,7 +76,7 @@ Features
 -  **Part-of-Speech Recognition**, grammatical tagging is the process of marking up a word in a text using finetuned Transformer-Bahasa.
 -  **Question Answer**, reading comprehension using finetuned Transformer-Bahasa.
 -  **Relevancy Analysis**, detect and recognize relevancy of texts using finetuned Transformer-Bahasa.
--  **Rumi-to-Jawi**, convert from Rumi to Jawi using LSTM Seq2Seq with attention state-of-art.
+-  **Rumi-to-Jawi**, convert from Rumi to Jawi using Transformer.
 -  **Sentiment Analysis**, detect and recognize polarity of texts using finetuned Transformer-Bahasa.
 -  **Text Similarity**, provide interface for lexical similarity deep semantic similarity using finetuned Transformer-Bahasa.
 -  **Spell Correction**, using local Malaysia NLP researches hybrid with Transformer-Bahasa to auto-correct any bahasa words and NeuSpell using T5-Bahasa.