Skip to content

tinh2044/AWESOME-NLP-PAPERS

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

7 Commits
.gitignore
.gitignore
Β 
Β 
CONTRIBUTING.md
CONTRIBUTING.md
Β 
Β 
LICENSE
LICENSE
Β 
Β 
README.md
README.md
Β 
Β 

Repository files navigation

πŸš€ Awesome NLP Papers

Awesome
License: MIT
Made With Love

πŸ“š Awesome NLP is a curated collection of high-quality resources, papers, libraries, tools, and datasets for Natural Language Processing (NLP). Whether you're a beginner exploring the basics or an expert diving into cutting-edge research, this repository has something for everyone.

πŸ“– Contents

1. Introduction

Natural Language Processing (NLP) is a fast-evolving field at the intersection of πŸ—£οΈ linguistics, πŸ€– artificial intelligence, and 🧠 deep learning. It powers various applications, from πŸ’¬ chatbots and 🌍 machine translation to ✍️ automated text generation and πŸ” information retrieval.

This repository organizes NLP research into key areas, making it easier for students, researchers, and practitioners to find relevant πŸ“„ papers, πŸ› οΈ tools, and πŸ“š datasets. Below is an overview of the main sections:

  • 🧠 Fundamentals of Deep Learning: Covers the core concepts of deep learning, including neural networks, activation functions, backpropagation, and optimization techniques.
  • ⏳ Sequence Modeling: Focuses on sequential data processing, including Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), and Transformer-based architectures.
  • πŸ“ Word Representations: Explores word embedding techniques, including static embeddings (Word2Vec, GloVe) and contextualized embeddings (BERT, ELMo).
  • πŸ“ Evaluation: Discusses how to measure NLP model performance, including accuracy, BLEU, ROUGE, and fairness metrics.
  • 🎯 Tasks: A collection of research papers on key NLP applications such as πŸ“ text generation, 🏷️ classification, πŸ” named entity recognition (NER), ❓ question answering, and 🌍 machine translation.
  • πŸ€– Models: Covers state-of-the-art NLP models such as BERT, GPT-3, RoBERTa, T5, and many others, providing links to research papers and implementations.
  • πŸ“‚ Datasets: A list of public datasets commonly used in NLP research, categorized by task (e.g., 🏷️ text classification, πŸ” NER, 🌍 machine translation).
  • πŸ‡»πŸ‡³ NLP in Vietnamese: Focuses on Vietnamese NLP research, including πŸ”„ text preprocessing, πŸ”€ embeddings, 🏷️ sentiment analysis, and 🌍 translation.

This structured collection makes it easier to πŸ“– understand fundamental NLP concepts, πŸš€ explore the latest research, and βš™οΈ apply NLP techniques to real-world problems.

2. How to Use

This repository is designed to be a comprehensive reference for NLP research and applications. Here’s how you can make the most of it:

1️⃣ Learn the Basics

If you're new to NLP, start with the Fundamentals of Deep Learning section. It provides a foundation in deep learning concepts that are essential for understanding modern NLP techniques.

2️⃣ Explore NLP Architectures

Read about different sequence modeling techniques in the Sequence Modeling section. This will introduce you to RNNs, LSTMs, the Attention Mechanism, and the Transformer model, which forms the basis of most modern NLP models.

3️⃣ Understand Word Representations

Check out the Word Representations section to learn how text is transformed into numerical vectors, including static embeddings (Word2Vec, GloVe) and contextualized embeddings (BERT, ELMo, GPT).

4️⃣ Assess Model Performance

Visit the Evaluation section to understand how NLP models are evaluated. This section covers common metrics such as BLEU for translation, ROUGE for summarization, and fairness metrics.

5️⃣ Find NLP Research Papers by Task

Browse the Tasks section for papers related to text classification, question answering, machine translation, and more.

6️⃣ Explore State-of-the-Art NLP Models

Visit the Models section to find research papers on models like BERT, GPT-3, RoBERTa, T5, and others.

7️⃣ Discover NLP Datasets

If you're looking for training datasets, check out the Datasets section, which categorizes datasets based on NLP tasks.

8️⃣ Explore Vietnamese NLP Research

For researchers focusing on Vietnamese NLP, the NLP in Vietnamese section includes papers and resources on Vietnamese text preprocessing, NER, sentiment analysis, and machine translation.

9️⃣ Stay Updated

The field of NLP is evolving rapidly. Keep an eye on new research papers and updates to this repository.

πŸ”Ÿ Contribute and Collaborate

If you have found a useful NLP paper or tool, consider contributing! See the Contributing section for details.

3. Contributing

We welcome contributions to make this repository better! Here’s how you can help:

  1. Suggest Papers or Resources:
    Found an important NLP paper, dataset, or tool? Open an issue or submit a pull request.

  2. Report Issues:
    Noticed a broken link or incorrect information? Let us know by opening an issue.

  3. Enhance Documentation:
    Help improve descriptions, summaries, or structure.

  4. Submit Pull Requests:

    • Fork the repository.
    • Create a new branch for your changes.
    • Commit your updates, ensuring they follow the existing format.
    • Submit a pull request with a clear description of your contribution.

Check out our Contribution Guidelines for detailed instructions.

Next Steps

  • If you find this repository helpful, star ⭐ it on GitHub and share it with the NLP community.
  • Start exploring topics from the table of contents.
  • Feel free to contribute by adding new papers, tools, or datasets.

Happy Learning! πŸš€

4. Fundamentals of Deep Learning

This section covers the foundational concepts of deep learning, including neural networks, activation functions, backpropagation, gradient descent, and optimization techniques. Each subsection includes links to important research papers and descriptions for further reading.

4.1 Neural Networks and Deep Learning

Explore the fundamental building blocks of deep learning and their applications across various domains.

4.2 Activation Functions

Learn about the key role of activation functions in neural networks and their impact on model performance.

4.3 Backpropagation and Gradient Descent

Explore the mathematics and algorithms that drive neural network training.

  • A Mathematical Theory of Communication
    πŸ–ŠοΈ Authors: Claude Shannon
    πŸ“– Description: This seminal work laid the foundation for information theory, which is crucial for neural networks.

  • Learning Internal Representations by Error Propagation
    πŸ–ŠοΈ Authors: David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams
    πŸ“– Description: Introduced the backpropagation algorithm, a powerful method for training multi-layer perceptrons.

  • On the Convergence Properties of the Back-Propagation Algorithm
    πŸ–ŠοΈ Authors: Y. LeCun, L. D. Jackel, L. Bottou
    πŸ“– Description: Investigates the convergence properties of the backpropagation algorithm, providing insights into its strengths and limitations.

  • An overview of gradient descent optimization algorithms
    πŸ–ŠοΈ Authors: Sebastian Ruder
    πŸ“– Description: Compares various gradient descent optimization algorithms, including standard gradient descent, Momentum, Adagrad, RMSprop, and Adam. It explores their mechanisms, advantages, and trade-offs, helping practitioners choose the best algorithm based on specific tasks. The paper also addresses challenges such as hyperparameter tuning and generalization in machine learning.

  • Efficient Backprop
    πŸ–ŠοΈ Authors: Yann LeCun, LΓ©on Bottou, Yoshua Bengio, Patrick Haffner
    πŸ“– Description: Explores techniques for improving the efficiency of backpropagation, which is crucial for training large neural networks.

  • Asynchronous stochastic gradient descent with decoupled backpropagation and layer-wise updates
    πŸ–ŠοΈ Authors: Cabrel Teguemne Fokam, Khaleelulla Khan Nazeer, Lukas KΓΆnig, David Kappel, Anand Subramoney
    πŸ“– Description: Presents a novel asynchronous approach to stochastic gradient descent, which decouples backpropagation across layers to improve efficiency in deep networks.

  • Generalizing Backpropagation for Gradient-Based Interpretability
    πŸ–ŠοΈ Authors: Kevin Du, Lucas Torroba Hennigen, Niklas Stoehr, Alexander Warstadt, Ryan Cotterell
    πŸ“– Description: Explores the concept of backpropagation and its generalization to understand gradient-based interpretability in machine learning models.

  • Gradient Descent based Optimization Algorithms for Deep Learning Models Training
    πŸ–ŠοΈ Authors: Jiawei Zhang
    πŸ“– Description: Explores gradient descent optimization techniques for training deep learning models, highlighting common methods like Momentum, Adagrad, Adam, and Gadam. It discusses how these algorithms improve training efficiency and performance, especially for complex models and high-dimensional data.

4.4 Optimization Techniques

Learn about optimization methods that improve training efficiency and performance in deep learning.

5. Sequence Modeling

This section explores models and techniques for handling sequential data, such as text, speech, or time-series, including RNNs, LSTMs, sequence-to-sequence models, attention mechanisms, and transformers.

5.1 RNNs and LSTMs

Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are widely used for processing sequential data. Below are key papers on their development and applications:

5.2 Sequence Models

Sequence models, such as sequence-to-sequence (seq2seq) architectures, handle input-output pairs with sequential relationships. Below are key papers on this topic:

5.3 Attention Mechanism

Attention mechanisms enable models to focus on the most relevant parts of the input when making predictions. This subsection includes research on various attention techniques:

5.4 Transformers

Transformers are state-of-the-art architectures in sequence modeling, built around the self-attention mechanism. Below are significant papers that outline their theory and applications:

  • Attention Is All You Need
    πŸ–ŠοΈ Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
    πŸ“– Description: The foundational paper introducing the transformer architecture. It details self-attention, encoder-decoder structure, and positional encodings, which are pivotal in sequence modeling tasks.

  • Understanding How Positional Encodings Work in Transformer Models
    πŸ–ŠοΈ Authors: T Miyazaki, H Mino, H Kaneko
    πŸ“– Description: Examines the functionality of positional encodings in self-attention and cross-attention blocks of transformer architectures, exploring their integration in encoder-decoder models.

  • Universal Transformers
    πŸ–ŠοΈ Authors: M Dehghani, S Gouws, O Vinyals, J Uszkoreit
    πŸ“– Description: Introduces a universal transformer that extends the standard model by incorporating recurrence in the self-attention mechanism, enhancing its theoretical depth and reasoning capabilities.

  • Position Information in Transformers: An Overview
    πŸ–ŠοΈ Authors: P Dufter, M Schmitt, H SchΓΌtze
    πŸ“– Description: Systematically reviews positional encoding techniques in transformers, analyzing over 30 models to understand their role in encoding positional information for attention mechanisms.

  • Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation
    πŸ–ŠοΈ Authors: TC Chi, TH Fan, AI Rudnicky, PJ Ramadge
    πŸ“– Description: Explores how transformer working memory interacts with self-attention to enable reasoning in regular languages and length extrapolation in NLP tasks.

  • Understanding the Failure of Batch Normalization for Transformers in NLP
    πŸ–ŠοΈ Authors: J Wang, J Wu, L Huang
    πŸ“– Description: Investigates the challenges batch normalization introduces to self-attention and proposes alternatives for stabilizing transformer training in NLP tasks.

  • Activating Self-Attention for Multi-Scene Absolute Pose Regression
    πŸ–ŠοΈ Authors: M Lee, J Kim, JP Heo
    πŸ“– Description: Details the functionality of self-attention and positional encoding in transformer encoders and cross-attention modules, applied to multi-scene regression tasks.

  • Aiatrack: Attention in Attention for Transformer Visual Tracking
    πŸ–ŠοΈ Authors: S Gao, C Zhou, C Ma, X Wang, J Yuan
    πŸ“– Description: Explores self-attention and cross-attention mechanisms within the encoder-decoder structure of transformers, focusing on applications in tracking tasks.

  • Why Transformers Are Obviously Good Models of Language
    πŸ–ŠοΈ Authors: F Hill
    πŸ“– Description: Discusses theoretical justifications for transformers' success in NLP, emphasizing the role of self-attention and cross-attention in language modeling.

  • Learning Deep Learning: Theory and Practice of Neural Networks, Transformers, and NLP
    πŸ–ŠοΈ Authors: M Ekman
    πŸ“– Description: Provides a comprehensive overview of transformers' components, including detailed discussions on self-attention, cross-attention, and encoder-decoder interactions in NLP.

6. Word Representations

Word representations are the foundation of many natural language processing tasks. This section is divided into three key areas: Static Word Embeddings, Contextualized Embeddings, and Subword-Based Representations, covering both classical and cutting-edge methods for representing words in vector spaces.

6.1 Static Word Embeddings

Static word embeddings, such as Word2Vec, GloVe, and FastText, represent each word with a fixed vector. Below are notable papers discussing their applications and limitations:

6.2 Contextualized Embeddings

Contextualized word embeddings, such as those generated by BERT, GPT, or ELMo, vary depending on the context in which the word appears. These embeddings capture semantic and syntactic nuances, making them ideal for a wide range of NLP tasks.

6.3 Subword-Based Representations

Subword-based representations break down words into smaller units, such as character n-grams or byte pair encodings (BPE). These methods are particularly useful for handling rare or unseen words, as well as morphologically rich languages.

7. Evaluation

Evaluation is a critical aspect of Natural Language Processing (NLP) to assess the effectiveness, robustness, and fairness of models. This section covers evaluation metrics, model validation techniques, and fairness metrics that ensure NLP models are measured accurately and ethically.

7.1 Evaluation Metrics (Accuracy, BLEU, ROUGE, etc.)

Evaluation metrics like BLEU, ROUGE, and METEOR are widely used to measure the quality of NLP systems, especially for tasks like summarization, machine translation, and text generation.

7.2 Model Validation and Cross-validation in NLP

Model validation ensures that NLP systems perform reliably across various datasets and settings. Techniques like cross-validation are crucial for optimizing models and preventing overfitting.

7.3 Bias and Fairness Metrics

Bias and fairness metrics evaluate how equitably NLP models perform across different groups and ensure that systems do not perpetuate or amplify societal biases.

8. Tasks

This section explores major NLP tasks, from foundational challenges like text classification and named entity recognition to advanced applications such as machine translation and question answering. Each task highlights methodologies, benchmarks, and state-of-the-art approaches that drive innovation in understanding, generating, and transforming human language computationally.

8.1 Text Generation

The automated creation of human-like text, such as stories, dialogue, or code. Modern models generate context-aware content for chatbots, creative writing, or code completion, balancing coherence and creativity while minimizing repetition or factual errors.

  • Generation - A New Frontier of Natural Language Processing?
    πŸ–ŠοΈ Authors: A. Joshi
    πŸ“– Description: Discusses the theoretical underpinnings of text generation in NLP, exploring its significance as a foundational component of linguistic processing.

  • Automated Title Generation in English Language Using NLP
    πŸ–ŠοΈ Authors: N. Sethi, P. Agrawal, V. Madaan, S.K. Singh
    πŸ“– Description: Presents a methodological framework for generating concise and relevant titles from English text using NLP techniques.

  • Applied Text Generation
    πŸ–ŠοΈ Authors: O. Rambow, T. Korelsky
    πŸ“– Description: Introduces a system for applying text generation to practical tasks, offering insights into its flexibility and adaptability across applications.

  • The Survey: Text Generation Models in Deep Learning
    πŸ–ŠοΈ Authors: T. Iqbal, S. Qureshi
    πŸ“– Description: Provides an in-depth analysis of text generation models, discussing deep learning-based methods and their theoretical advancements.

  • Controlled Text Generation with Adversarial Learning
    πŸ–ŠοΈ Authors: F. Betti
    πŸ“– Description: Explores conditional and controlled text generation, leveraging adversarial learning to refine outputs for specific contexts.

  • Neural Text Generation: Past, Present, and Beyond
    πŸ–ŠοΈ Authors: S. Lu, Y. Zhu, W. Zhang, J. Wang, Y. Yu
    πŸ“– Description: Surveys neural text generation, highlighting historical advancements, current methodologies, and future challenges.

  • A Theoretical Analysis of the Repetition Problem in Text Generation
    πŸ–ŠοΈ Authors: Z. Fu, W. Lam, A.M.C. So, B. Shi
    πŸ“– Description: Presents a theoretical framework for addressing repetition in generated text, a common issue in neural language models.

  • Natural Language Generation
    πŸ–ŠοΈ Authors: E. Reiter
    πŸ“– Description: Explores the fundamentals of natural language generation, detailing its applications and challenges in connecting linguistic theory with practical systems.

  • Evaluation of Text Generation: A Survey
    πŸ–ŠοΈ Authors: A. Celikyilmaz, E. Clark, J. Gao
    πŸ“– Description: Analyzes evaluation metrics for text generation, providing theoretical insights into how generated text quality is assessed in NLP.

  • Pre-trained Language Models for Text Generation: A Survey
    πŸ–ŠοΈ Authors: J. Li, T. Tang, W.X. Zhao, J.Y. Nie, J.R. Wen
    πŸ“– Description: Examines pre-trained language models for text generation, focusing on their underlying mechanisms and theoretical implications.

8.2 Text Classification

Assigning labels (e.g., sentiment, topic) to text segments. Used to categorize emails, analyze opinions, or detect spam by training models to recognize patterns in unstructured data.

8.3 Named Entity Recognition (NER)

Identifying and classifying entities (e.g., people, locations) in text. Critical for extracting structured information from documents, enabling applications like search optimization and knowledge graph construction.

8.4 Question Answering

Answering natural language questions by extracting or generating responses from a given context. Powers virtual assistants and tools requiring precise retrieval of facts or reasoning over multiple sources.

8.5 Fill Mask

A pre-training task where models predict masked words in sentences. Helps learn contextual relationships between words, forming the basis for training robust language models like BERT.

8.6 Machine Translation

Translating text between languages while preserving meaning. Advances in neural models enable fluent translations, addressing challenges like idiomatic expressions and low-resource language support.

9. Models

This section provides an overview of popular NLP models, ranging from foundational architectures to state-of-the-art models used for tasks like language generation, translation, classification, and more. Each model includes a brief description of its purpose, capabilities, and advancements.

9.1 BERT

BERT (Bidirectional Encoder Representations from Transformers) is a revolutionary transformer-based model developed by Google. Unlike traditional models, BERT uses bidirectional context, allowing it to capture dependencies from both left and right sides of a token. It is widely used for tasks like text classification, question answering, and named entity recognition.

9.2 GPT-3 (GPT)

GPT-3 (Generative Pre-trained Transformer 3), developed by OpenAI, is a large language model known for its impressive ability to generate coherent, human-like text. GPT-3 is widely used for tasks like text completion, question answering, and creative content generation. It builds on the generative pre-training concept introduced in GPT-2.

  • Language Models Are Few-Shot Learners
    πŸ–ŠοΈ Authors: Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
    πŸ“– Description: This seminal paper introduces GPT-3, a large-scale transformer-based language model. It demonstrates state-of-the-art performance on a variety of NLP tasks using few-shot, one-shot, and zero-shot learning paradigms.

  • What Makes Good In-Context Examples for GPT-3?
    πŸ–ŠοΈ Authors: J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin
    πŸ“– Description: Investigates the effectiveness of example selection in few-shot settings for GPT-3, offering theoretical insights and practical strategies for better performance.

  • Who is GPT-3? An Exploration of Personality, Values, and Demographics
    πŸ–ŠοΈ Authors: M. Miotto, N. Rossberg, B. Kleinberg
    πŸ“– Description: Explores the personality and ethical considerations of GPT-3 by analyzing its outputs and implicit biases.

  • GPT-3: Implications and Challenges for Machine Text
    πŸ–ŠοΈ Authors: Y. Dou, M. Forbes, R. Koncel-Kedziorski
    πŸ“– Description: Evaluates the text generated by GPT-3 for linguistic and stylistic coherence, and highlights challenges in distinguishing machine-generated text from human-written content.

9.3 GPT-2

GPT-2 (Generative Pre-trained Transformer 2) is the predecessor to GPT-3, with fewer parameters but still a powerful model for text generation. GPT-2 demonstrated the potential of transformer-based models to generate coherent and contextually relevant text, sparking advancements in generative AI.

9.4 RoBERTa

RoBERTa (Robustly Optimized BERT Pretraining Approach) is an improved version of BERT developed by Facebook AI. It modifies the pretraining process with larger datasets, longer training times, and other optimizations, resulting in improved performance across many NLP tasks.

  • RoBERTa: A Robustly Optimized BERT Pretraining Approach
    πŸ–ŠοΈ Authors: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov
    πŸ“– Description: This paper enhances the BERT model by optimizing pretraining strategies, such as dynamic masking, increased training data, and larger batch sizes. RoBERTa outperforms BERT on multiple benchmarks, showcasing the benefits of improved pretraining techniques.

  • Sentiment Classification with Modified RoBERTa and RNNs
    πŸ–ŠοΈ Authors: R. Cheruku, K. Hussain, I. Kavati, A.M. Reddy
    πŸ“– Description: Demonstrates the use of RoBERTa in combination with recurrent neural networks to improve sentiment analysis.

  • Robust Multilingual NLU with RoBERTa
    πŸ–ŠοΈ Authors: A. Conneau, A. Lample
    πŸ“– Description: Extends RoBERTa's capabilities to multilingual natural language understanding tasks, showing its flexibility across languages.

  • Aspect-Based Sentiment Analysis Using RoBERTa
    πŸ–ŠοΈ Authors: G.R. Narayanaswamy
    πŸ“– Description: Explores how RoBERTa can enhance sentiment classification with a focus on aspect-based analysis.

9.5 T5

T5 (Text-to-Text Transfer Transformer), developed by Google, frames every NLP task as a text-to-text problem. This unified approach allows T5 to perform tasks like translation, summarization, and question answering with remarkable efficiency and flexibility.

9.6 DistilBERT

DistilBERT is a smaller, faster, and more lightweight version of BERT. Developed by Hugging Face, it uses knowledge distillation to retain most of BERT's accuracy while reducing its size and computational requirements, making it suitable for real-time applications.

9.7 ALBERT

ALBERT (A Lite BERT) is a smaller and more efficient variant of BERT. It reduces the number of parameters through techniques like factorized embedding parameterization and shared parameters across layers, achieving faster training and inference without significant performance loss.

  • ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
    πŸ–ŠοΈ Authors: Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut
    πŸ“– Description: This paper introduces ALBERT, a lightweight and efficient variant of BERT. ALBERT reduces model size significantly while maintaining state-of-the-art performance using parameter sharing and factorized embeddings.

  • Performance and Scalability of ALBERT in Question Answering Tasks
    πŸ–ŠοΈ Authors: J. Liu, Z. Zhao, T. Chen
    πŸ“– Description: Explores the use of ALBERT in question-answering tasks, highlighting its efficiency and scalability across diverse datasets.

  • ALBERT for Biomedical Named Entity Recognition
    πŸ–ŠοΈ Authors: H. Wang, S. Wu, R. Zhang
    πŸ“– Description: Adapts ALBERT to biomedical NLP tasks, demonstrating its effectiveness in named entity recognition for domain-specific datasets.

  • Efficient Fine-tuning with ALBERT
    πŸ–ŠοΈ Authors: Y. Chen, F. Zhang, S. Guo
    πŸ“– Description: Proposes strategies for efficient fine-tuning of ALBERT, showcasing reduced computational costs and improved adaptability.

9.8 BART

BART (Bidirectional and Auto-Regressive Transformers), developed by Facebook AI, is a versatile transformer model designed for text generation tasks. It combines the strengths of both bidirectional models like BERT and auto-regressive models like GPT, making it effective for summarization, translation, and more.

9.9 ELECTRA

ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) is an alternative to masked language modeling. Instead of masking tokens, it trains a model to detect replaced tokens, resulting in faster and more efficient pretraining with strong downstream performance.

  • ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
    πŸ–ŠοΈ Authors: Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning
    πŸ“– Description: Introduces ELECTRA, a model that replaces the generator-discriminator setup in pretraining. It achieves higher efficiency compared to BERT while maintaining strong performance on NLP tasks.

  • An Analysis of ELECTRA for Sentiment Classification
    πŸ–ŠοΈ Authors: S. Zhang, H. Yu, G. Zhu
    πŸ“– Description: Explores ELECTRA’s application in sentiment classification of Chinese text, emphasizing its efficiency in handling short comments.

  • ELECTRA-Based Neural Coreference Resolution

    πŸ–ŠοΈ Authors: F. Gargiulo, A. Minutolo, R. Guarasci, E. Damiano
    πŸ“– Description: Leverages ELECTRA for coreference resolution tasks, demonstrating its potential in improving co-reference accuracy in text.

  • ELECTRA for Biomedical Named Entity Recognition
    πŸ–ŠοΈ Authors: S. Wang, T. Zhang
    πŸ“– Description: Adapts ELECTRA for biomedical text processing, focusing on named entity recognition in domain-specific corpora.

  • Fine-Tuning ELECTRA for Efficient Text Summarization
    πŸ–ŠοΈ Authors: A. Banerjee, L. White
    πŸ“– Description: Presents fine-tuning methods for ELECTRA to improve its performance on text summarization tasks efficiently.

9.10 XLNet

XLNet is a transformer-based model that addresses the limitations of BERT by leveraging a permutation-based training objective. This allows XLNet to capture bidirectional context while avoiding the masking limitations of BERT, resulting in improved performance on various NLP tasks.

  • XLNet: Generalized Autoregressive Pretraining for Language Understanding
    πŸ–ŠοΈ Authors: Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le
    πŸ“– Description: Introduces XLNet, which integrates autoregressive and autoencoding objectives to overcome limitations in BERT. It uses permutation-based training to improve context understanding."

  • XLNet for Text Classification
    πŸ–ŠοΈ Authors: F. Shi, S. Kai, J. Zheng, Y. Zhong
    πŸ“– Description: Explores fine-tuning XLNet for text classification tasks, demonstrating significant improvements over baseline models."

  • Comparing XLNet and BERT for Computational Characteristics
    πŸ–ŠοΈ Authors: H. Li, J. Choi, S. Lee, J.H. Ahn
    πŸ“– Description: Compares XLNet and BERT from the perspective of computational efficiency, emphasizing training speed and resource utilization."

  • XLNet-CNN: Combining Global Context with Local Context for Text Classification
    πŸ–ŠοΈ Authors: A. Shahriar, D. Pandit, M.S. Rahman
    πŸ“– Description: Combines XLNet with convolutional neural networks to capture both global and local contexts, enhancing text classification accuracy."

  • DialogXL: Emotion Recognition in Conversations
    πŸ–ŠοΈ Authors: W. Shen, J. Chen, X. Quan, Z. Xie
    πŸ“– Description: Proposes DialogXL, an extended XLNet framework tailored for emotion recognition in multi-party conversations."

9.11 BERTweet

BERTweet is a transformer model specifically pre-trained on a large corpus of English tweets. It is optimized for tasks in the social media domain, such as sentiment analysis, hate speech detection, and user intent classification.

9.12 BlenderBot

BlenderBot, developed by Facebook AI, is an open-domain chatbot capable of engaging in human-like conversations. It combines the conversational abilities of retrieval-based models with generative approaches, enabling it to generate more contextually appropriate and engaging responses.

  • BlenderBot: Towards a More Open-Domain, Conversational AI Model
    πŸ–ŠοΈ Authors: Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, Jason Weston
    πŸ“– Description: Introduces BlenderBot, an open-domain chatbot designed to deliver engaging and knowledgeable conversations by fine-tuning conversational datasets with enhanced generative capabilities.

  • BlenderBot 3: A Conversational Agent for Responsible Engagement
    πŸ–ŠοΈ Authors: Kurt Shuster, Jing Xu, Morteza Komeili, Emily Smith, Jason Weston
    πŸ“– Description: Details the advancements in BlenderBot 3, focusing on continual learning, safety mechanisms, and the model’s ability to adapt to user feedback in real-time.

  • Empirical Analysis of BlenderBot 2.0 for Open-Domain Conversations
    πŸ–ŠοΈ Authors: J Lee, M Shim, S Son, Y Kim, H Lim
    πŸ“– Description: Examines the shortcomings of BlenderBot 2.0 across model, data, and user-centric approaches, offering insights for improvements in future iterations.

  • GE-Blender: Graph-Based Knowledge Enhancement for Blender
    πŸ–ŠοΈ Authors: X Lian, X Tang, Y Wang
    πŸ“– Description: Proposes a graph-based knowledge-enhancement framework to improve BlenderBot’s ability to provide more accurate and contextually enriched responses.

  • Enhancing Commonsense Knowledge in BlenderBot
    πŸ–ŠοΈ Authors: O Kobza, D Herel, J Cuhel, T Gargiani, J Pichl, P Marek
    πŸ“– Description: Explores methods to augment commonsense knowledge in BlenderBot, improving conversational consistency and user engagement.

9.13 DeBERTa

DeBERTa (Decoding-enhanced BERT with Disentangled Attention) improves upon BERT and RoBERTa by introducing disentangled attention mechanisms and an enhanced mask decoder. These innovations allow DeBERTa to achieve state-of-the-art results on a variety of NLP benchmarks.

9.14 BigBird

BigBird is a sparse attention transformer designed to handle long sequences efficiently. It is particularly useful for tasks involving long documents, such as summarization and question answering, where standard transformers struggle due to memory constraints.

  • Big Bird: Transformers for Longer Sequences
    πŸ–ŠοΈ Authors: Manzil Zaheer, Guru Guruganesh, Kaushik Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed
    πŸ“– Description: This paper introduces BigBird, a transformer model designed for efficient handling of longer sequences using a sparse attention mechanism, reducing computational complexity from quadratic to linear.

  • ICDBigBird: A Contextual Embedding Model for ICD Code Classification
    πŸ–ŠοΈ Authors: G. Michalopoulos, M. Malyska, N. Sahar, A. Wong
    πŸ“– Description: Proposes a BigBird-based contextual embedding model tailored for ICD code classification in medical records, showcasing the model's capacity for domain-specific applications.

  • Clinical-longformer and Clinical-BigBird: Transformers for Long Clinical Sequences
    πŸ–ŠοΈ Authors: Y. Li, R. Wehbe, F. Ahmad, H. Wang, Y. Luo
    πŸ“– Description: Develops Clinical-BigBird for processing long clinical text sequences, highlighting its performance improvements compared to other transformer models.

  • Attention-Free BigBird Transformer for Long Document Text Summarization
    πŸ–ŠοΈ Authors: G. Mishra, N. Sethi, A. Loganathan
    πŸ“– Description: Introduces a modified BigBird transformer for document summarization, removing attention-based mechanisms for better efficiency.

  • Vision BigBird: Random Sparsification for Full Attention
    πŸ–ŠοΈ Authors: Z. Zhang, X. Gong
    πŸ“– Description: Applies BigBird concepts to vision transformers, proposing a random sparsification mechanism to optimize full attention for vision tasks.

9.15 PEGASUS

PEGASUS is a transformer model developed for abstractive summarization tasks. It uses a novel pretraining objective called "Gap Sentences Generation" to better understand document structure and generate high-quality summaries.

9.16 FLAN-T5

FLAN-T5 is a fine-tuned version of T5 that incorporates instruction tuning across multiple NLP tasks. This makes it more versatile and capable of zero-shot or few-shot learning for new tasks, improving its generalization capabilities.

9.17 MobileBERT

MobileBERT is a compact version of BERT optimized for mobile and edge devices. It maintains strong performance on NLP tasks while being significantly smaller and faster, making it ideal for resource-constrained environments.

9.18 GPT-Neo

GPT-Neo is an open-source alternative to GPT-3, developed by EleutherAI. It offers a similar architecture and is pre-trained on large datasets, enabling it to perform generative NLP tasks like text completion and summarization.

9.19 Longformer

Longformer addresses the limitations of standard transformers with sparse attention, enabling it to process long sequences efficiently. It is suitable for tasks like document classification, summarization, and long-context question answering.

  • Longformer: The Long-Document Transformer
    πŸ–ŠοΈ Authors: Iz Beltagy, Matthew E. Peters, Arman Cohan
    πŸ“– Description: This paper introduces Longformer, a transformer model optimized for long documents. It uses a sparse attention mechanism that scales linearly with sequence length, making it suitable for processing thousands of tokens efficiently.

  • Long Range Arena: A Benchmark for Efficient Transformers
    πŸ–ŠοΈ Authors: Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler
    πŸ“– Description: Provides a systematic benchmark to evaluate transformer models, including Longformer, for long-range attention tasks, emphasizing efficiency and performance.

  • Longformer for Multi-Document Summarization
    πŸ–ŠοΈ Authors: F. Yang, S. Liu
    πŸ“– Description: Applies Longformer to extractive summarization of multiple documents, showcasing its ability to handle large-scale text summarization tasks effectively.

  • Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
    πŸ–ŠοΈ Authors: P. Zhang, X. Dai, J. Yang
    πŸ“– Description: Adapts Longformer concepts for vision tasks, focusing on encoding high-resolution images with sparse attention for computational efficiency.

  • Longformer for Dense Document Retrieval
    πŸ–ŠοΈ Authors: J. Yang, Z. Liu, G. Sun
    πŸ“– Description: Explores Longformer as a dense document retrieval model, demonstrating its ability to process and retrieve information from long-form text effectively.

9.20 XLM-RoBERTa

XLM-RoBERTa is a multilingual variant of RoBERTa designed to handle over 100 languages. It is highly effective for cross-lingual understanding tasks, such as translation and multilingual question answering.

9.21 DialoGPT

DialoGPT, developed by Microsoft, is a conversational version of GPT-2 fine-tuned on dialogue datasets. It is designed to generate engaging, context-aware conversational responses for chatbots and other interactive applications.

9.22 MarianMT

MarianMT is a neural machine translation model developed by Facebook. It supports many language pairs and is optimized for low-resource languages, making it an excellent tool for translation tasks.

9.23 Falcon

Falcon is an open-source generative language model known for its lightweight architecture and efficient training. It is particularly useful for generating text with constrained computational resources.

9.24 CodeGen

CodeGen is a transformer model optimized for code generation tasks. It has been fine-tuned on programming-related datasets, enabling it to write code snippets in languages like Python, JavaScript, and more.

9.25 ByT5

ByT5 is a byte-level version of the T5 model. It eliminates the need for tokenization by processing raw byte inputs, making it especially effective for multilingual tasks and handling unseen text encodings.

9.26 PhoBERT

PhoBERT is a pre-trained language model tailored for Vietnamese. It is optimized for NLP tasks in Vietnamese, such as sentiment analysis, text classification, and named entity recognition.

9.27 Funnel Transformer

Funnel Transformer introduces a pooling mechanism to reduce the computational complexity of transformers. This hierarchical approach improves scalability while maintaining performance for long-sequence tasks.

9.28 T5v1.1

T5v1.1 is an improved version of the original T5 model. It features architectural changes and optimizations, resulting in enhanced performance and better efficiency for a wide range of NLP tasks.

  • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
    πŸ–ŠοΈ Authors: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
    πŸ“– Description: This foundational paper introduces the T5 framework, which forms the basis for T5v1.1. It treats all NLP tasks as a text-to-text problem, enabling seamless multitask learning and fine-tuning.

  • Improved Fine-Tuning and Parameter Sharing in T5 Models
    πŸ–ŠοΈ Authors: V. Lialin, K. Zhao, N. Shivagunde
    πŸ“– Description: Proposes refinements for the T5 architecture, including T5v1.1, focusing on enhanced parameter sharing and optimized fine-tuning strategies.

  • T5v1.1 for Low-Resource Language Understanding
    πŸ–ŠοΈ Authors: D. Mehra, L. Xie, E. Hofmann-Coyle
    πŸ“– Description: Explores the use of T5v1.1 in low-resource language tasks, demonstrating its ability to adapt and perform well on limited data.

  • Enhanced Dialogue State Tracking Using T5v1.1
    πŸ–ŠοΈ Authors: P. Lesci, Y. Fujinuma, M. Hardalov, C. Shang
    πŸ“– Description: Demonstrates the efficiency of T5v1.1 for dialogue state tracking tasks, leveraging its text-to-text capabilities for complex conversational scenarios.

  • T5v1.1 in Scientific Document Summarization
    πŸ–ŠοΈ Authors: R. Uppaal, Y. Li, J. Hu
    πŸ“– Description: Applies T5v1.1 for summarizing scientific documents, emphasizing its superior abstractive summarization performance compared to baseline models.

9.29 RoFormer

RoFormer (Rotary Position Embeddings Transformer) incorporates rotary position embeddings to improve positional encoding in transformers. This innovation enhances its capability to handle longer sequences and tasks like language modeling and translation.

9.30 MBart and MBart-50

MBart (Multilingual BART) and its extension MBart-50 are encoder-decoder models optimized for multilingual tasks, including translation across 50 languages. They are pre-trained on large-scale multilingual data and fine-tuned for tasks like summarization and language generation.

10. Datasets

Datasets play a crucial role in training and evaluating NLP models. The choice of dataset depends on the specific NLP task, as different datasets cater to different use cases, such as text generation, classification, named entity recognition, question answering, and more. Below, we provide a categorized list of commonly used datasets for various NLP tasks.

10.1 Text Generation Datasets

These datasets are used to train models that generate coherent and contextually relevant text based on a given input. Common applications include dialogue systems, story generation, and code completion.

10.2 Text Classification Datasets

Text classification datasets help train models to categorize text into predefined labels. These datasets are used in applications like sentiment analysis, spam detection, and topic classification.

10.3 Named Entity Recognition Datasets

Named Entity Recognition (NER) datasets are used for extracting named entities such as persons, locations, organizations, and dates from text. These datasets are crucial for tasks like information retrieval and knowledge extraction.

10.4 Question Answering Datasets

Question Answering (QA) datasets enable models to generate answers based on a given question and context. These datasets are widely used in search engines, virtual assistants, and automated customer support systems.

10.5 Fill Mask Datasets

Fill Mask datasets are used for training masked language models (MLMs) where a model learns to predict missing words in a given sentence. These datasets help improve contextualized word representations.

10.6 Machine Translation Datasets

Machine translation datasets provide parallel corpora for training models to translate text between different languages. These datasets are fundamental in developing multilingual NLP systems.

11. NLP in Vietnamese

Vietnamese NLP presents unique challenges due to the language's lack of word boundaries, tonal nature, and rich morphology. This section provides a collection of papers, tools, and datasets specifically tailored for Vietnamese NLP research and applications.

11.1 Vietnamese Text Preprocessing

Vietnamese text preprocessing involves tasks such as tokenization, stopword removal, and diacritic normalization. Due to the lack of explicit word boundaries, word segmentation is a critical preprocessing step in Vietnamese NLP.

11.2 Vietnamese Word Representations

Word embeddings and contextualized word representations trained specifically for Vietnamese text improve NLP performance. This includes models like Word2Vec, FastText, and transformer-based embeddings such as PhoBERT.

11.3 Vietnamese Named Entity Recognition (NER)

Named Entity Recognition (NER) identifies entities such as names, organizations, and locations within Vietnamese text. Challenges include handling ambiguous entity boundaries and diacritic variations.

11.4 Vietnamese Part-of-Speech Tagging

Part-of-Speech (POS) tagging in Vietnamese requires models to correctly classify words into grammatical categories despite the language’s complex morphology and word segmentation issues.

11.5 Vietnamese Syntax and Parsing

Vietnamese dependency parsing and constituency parsing help analyze sentence structures, enabling downstream applications like machine translation and question answering.

11.6 Machine Translation for Vietnamese

Machine translation between Vietnamese and other languages (e.g., English, French, Chinese) is an active research area. Transformer-based models like MarianMT and multilingual BERT-based models improve translation quality.

11.7 Vietnamese Question Answering

Question Answering (QA) systems in Vietnamese involve answering questions based on structured or unstructured text. QA models require high-quality annotated datasets for accurate responses.

11.8 Vietnamese Text Summarization

Text summarization generates concise and informative summaries from long Vietnamese documents. Extractive and abstractive summarization techniques are commonly used for this task.

11.9 Resources for Vietnamese NLP

A collection of open-source tools, frameworks, and datasets for Vietnamese NLP, including word segmentation tools, language models, and benchmark datasets.

11.10 Challenges in Vietnamese NLP

Discusses the key challenges in Vietnamese NLP, such as handling tonal variations, segmentation difficulties, data scarcity, and the need for high-quality annotated datasets.

About

A curated collection of NLP research papers, models, datasets, and tools covering fundamentals, advanced techniques, and real-world applications. πŸš€

Topics

nlp natural-language-processing deep-learning transformers neural-networks awesome-list bert vietnamese-nlp sequence-modeling nlp-papers vn-nlp

Resources

Readme

License

Apache-2.0 license

Contributing

Contributing
Activity

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors