Learn
Natural Language Processing (NLP) Basics

Top 20 NLP Models to Empower Your ML Application

Nov 13, 20237 min read

Learn about the 10 most popular LLMs taking 2023 by storm and another 10 basic NLP models.

Read the entire series

In our previous post, we delved into the fascinating world of Natural Language Processing (NLP) and explored its myriad real-world applications. In this installment, let's take a closer look at some of the most fundamental NLP models, like BERT and XLNet, and some cutting-edge large language models, like GPT and PaLM, that have taken the world by storm in 2023.

10 Large Language Models taking 2023 by storm

A large language model, or LLM, is a machine learning model that can perform various natural language processing (NLP) tasks, like translating texts, answering questions conversationally, and classifying and generating words based on the knowledge gained from different datasets. The term "large" here refers to the number of parameters used in its architecture, with some of the most common LLMs having billions of them. Below are some of the most well-known LLMs.

GPT series by OpenAI (Generative Pre-trained Transformer)

GPT-3

Released in 2021 with a staggering 175 billion parameters.
Capable of language translation, question answering, essay writing, and even code generation.
Uses a decoder-only transformer architecture.
The last of the GPT models in which OpenAI made the parameter counts publicly available.
Exclusive use by Microsoft since September 2022.

GPT-3.5

An upgraded version with fewer parameters, introduced in 2022.
Powers ChatGPT and gained immense popularity, acquiring one million users in five days and 100 million users within two months.
Training data extends to September 2021.
Integrated into the Bing search engine but has since been replaced with GPT-4.

GPT-4

The latest in the GPT series, released in 2023.
A multimodal model responding to both text and images.
Trained in Microsoft Azure AI supercomputers with a focus on creativity and collaboration.

PaLM 2 by Google

Introduced in 2023, building on Google's legacy in machine learning and responsible AI.
Pre-trained on parallel multilingual text and a larger corpus than its predecessor.
Excels in advanced reasoning, translation, and code generation.

LLama2 by Meta and Microsoft

Released in 2023 in three model sizes: 7, 13, and 70 billion parameters.
Includes both foundational models and models fine-tuned for dialog, called LLama 2 Chat.
Versatile and powerful, designed for tasks like query resolution and natural language comprehension.
Meta's specialized focus on educational applications makes LLaMA-2 an ideal AI assistant for EdTech platforms.

Claude 2 by Anthropic

Released in 2023 by Anthropic, excelling at complex reasoning tasks.
Focuses on constitutional AI, guiding AI outputs to be helpful, harmless, and accurate.
Acts as a friendly assistant for various tasks instructed in natural language.

Grok-1 by xAI

Announced in 2023 by Elon Musk’s xAI, designed to answer almost any question with wit.
Modeled after the Hitchhiker's Guide to the Galaxy.
Real-time knowledge of the world via the 𝕏 platform.

Falcon by Technology Innovation Institute

An open-source model announced in 2023.
Boasts 180 billion parameters, surpassing Llama on the Hugging Face Open LLM Leaderboard.
Trained on a high-quality dataset with a mix of text and code, covering various languages and dialects.

Cohere by Cohere

An open-source multi-language model introduced in 2022 by a Canadian startup.
Trained on a diverse and inclusive dataset, making it excellent for understanding texts in over 100 languages.
Embedded into Oracle and Salesforce products for tasks like language generation, text summarization, and sentiment analysis.

10 Basic NLP models

BERT (Bidirectional Encoder Representations from Transformers)

Bidirectional Encoder Representations from Transformers (BERT) was first proposed in 2018 by Jacob Devlin in his paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
The main breakthrough of the BERT model is that it scans text in a bidirectional way rather than a left-to-right or combined left-to-right and right-to-left sequence when looking at texts during training.
There are two general types of BERT: BERT (base) and BERT (large). The difference is in configurable parameters: base-110 million parameter, large-345 million.

XLNet

XLNet was published in the paper XLNet: Generalized Autoregressive Pretraining for Language Understanding in 2019.
XLNet outperforms BERT by large margins in 20 benchmark tests as it leverages the best of both autoregressive models and bidirectional context modeling. XLNet adopts a newly proposed modeling method called "permutation language modeling."
Unlike traditional tokenization in a language model that predicts the word in a sentence based on the context of the previous token, XLNet's permutation language modeling considers the interdependency between tokens.
XLNet achieves a 2-15% performance test improvement over BERT.

RoBERTa (Robustly Optimized BERT Approach)

RoBERTa was proposed in the paper RoBERTa: A Robustly Optimized BERT Pretraining Approach in 2019.
RoBERTa makes changes to the architecture and training procedures of BERT. Specifically, RoBERTa removes the next sentence prediction (NSP) objective, uses a much larger dataset than BERT, and replaces static masking with dynamic masking.
RoBERTa achieves a performance test result of 2-20% improvement over BERT.

ALBERT (A Lite BERT)

The ALBERT model was proposed in the paper ALBERT: A Lite BERT for Self-supervised Learning of Language Representations in 2019.
ALBERT is developed based on the BERT model. Its major breakthrough is that it brings a significant parameter reduction but maintains the same level of performance compared to BERT.
In ALBERT, parameters are shared across 12 layers of transformer encoders, while in the original BERT, each layer of encoders has a unique set of parameters.

StructBERT

StructBERT was proposed in the paper StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding in 2019.
StructBERT further extends BERT by incorporating language structure into the training procedure.
StructBERT also introduces the word structural objective (WSO), which helps the model to learn the ordering of words.

T5 (Text-to-Text Transfer Transformer)

T5 was introduced in the paper Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer in 2019. T5 is the short form for "Text-to-Text Transfer Transformer".
A clean, massive, open-source dataset C4(Colossal Clean Crawled Corpus) is released in T5.
T5 categorizes all NLP tasks as "text-to-text" tasks.
There are five different sizes of the T5 model, each with a different number of parameters: T5-small (60 million parameters), T5-base (220 million parameters), T5-large (770 million parameters), T5-3B (3 billion parameters), T5-11B (11 billion parameters).

SentenceTransformers

SentenceTransformers’ initial work is described in the paper Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks published in 2019.
SentenceTransformers is a Python framework for sentence, text, and image embeddings.
SentenceTransformers can compute sentence/text embeddings for more than 100 languages.
The framework is based on PyTorch and Transformers and offers many pre-trained models tuned for various tasks.

ERNIE (Enhanced Representation through kNowledge Integration)

ERNIE, developed by Baidu, was introduced in a research paper titled ERNIE: Enhanced Language Representation with Informative Entities, which was presented by researchers from Baidu at the Association for Computational Linguistics (ACL) conference in 2019.
ERNIE incorporates world knowledge into pre-trained language models and is designed to understand the nuances of human language and improve the performance of various NLP tasks.
There are different versions of ERNIE, and the model has been updated and refined over time to achieve better performance on a wide range of NLP tasks.

CTRL (Controllable Text Generation)

CTRL was introduced by Salesforce Research in a paper titled CTRL: A Conditional Transformer Language Model, presented at NeurIPS (Conference on Neural Information Processing Systems) in 2019.
CTRL is a natural language processing (NLP) model that allows users to control the style and content of the generated text.
The CTRL model is designed to generate diverse and controlled text. It allows users to specify the style or bias of the generated text, giving more control over the language generation process.

ELECTRA

ELECTRA was proposed in the paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators in 2020.
ELECTRA proposes a new framework for pre-training that combines a generator and discriminator.
ELECTRA changes the training method of masked language models to replace token detection.
ELECTRA performs better on small-sized models.

Angela Ni

Next: Unveiling the Power of Natural Language Processing: Top 10 Real-World Applications