Learn
Large Language Models (LLMs) 101

Introduction to the Falcon 180B Large Language Model (LLM)

Mar 18, 20248 min read

Falcon 180B is an open-source large language model (LLM) with 180B parameters trained on 3.5 trillion tokens. Learn its architecture and benefits in this blog.

By Frank Liu

Read the entire series

Latest Update: July 31, 2024

Falcon LLM is a generative LLM that includes Falcon Mamba 7B, Falcon 2, 180B, 40B, 7.5B, 1.3B parameter AI models. Falcon 180B is a super powerful language model which is open-source with 180B parameters trained on 3.5 trillion tokens. Learn its architecture and benefits in this blog.

Introduction to Falcon AI and the Falcon LLM Family

The global AI landscape exploded after a GPT initiative started in early 2000. Since then, several generative models have appeared. As AI became progressively more powerful, it began to resemble human intelligence. Nevertheless, the Open AI community closed the GPT family based on powerful linguistic models.

Falcon180b AI, a scalable and powerful generative language model, is free from GitHub and an "an open access model for research and commercial use" according the Technology Innovation Institute. The Falcon series of large language models (LLMs) represents a significant advancement in developing and deploying large-scale language models. This blog condenses and elaborates on the key aspects of the Falcon 180B model, including its model architecture, dataset considerations, training strategy, and resultant performance metrics.

Falcon 180B Model and Its Architecture

Like GPT, Claude, Pi, and other well-known LLMs, the Falcon models are still based on the autoregressive (decoder-only) transformer architecture with some macro changes driven by scalability and efficiency. This section provides an in-depth exploration of Falcon’s model architecture, highlighting the features and changes that go into it and the motivation behind these changes. From this perspective, the Falcon models try to find a good middle ground regarding model performance versus inference speeds.

Falcon-40B, developed by the Technology Innovation Institute, is a significant model in the Falcon family, known for its computational efficiency and robust performance in various applications. Falcon 180B, a larger model with 180 billion parameters, is an open-source large language model (LLM) trained on 3.5 trillion tokens. It’s the largest publicly available language model, and its architecture is based on GPT-3 with some key differences. Falcon 180B’s extensive training gives it exceptional language understanding and versatility in various applications. It’s available for both research and commercial use and performs well in tasks like reasoning, coding, proficiency, and knowledge tests.

Here, I’ll highlight several key architectural decisions that go into Falcon.

Multiquery and Multigroup Attention

One of the hallmark features of Falcon architecture is the adoption and extension of multi-query attention into multigroup attention. The idea stems from recognizing that while the multi head attention mechanism is powerful, it can be optimized for efficiency without sacrificing performance.

Multiquery Attention: This adaptation simplifies the attention mechanism by sharing keys and values across all heads, drastically reducing memory consumption and computational overhead. This is particularly beneficial for large models during inference, where the reduction in memory footprint directly translates to faster, more efficient generation tasks.
Multigroup Attention: Building on multiquery attention, the Falcon LLMs introduces multigroup attention, where the number of key-value pairs is equal to the degree of tensor parallelism. This further optimizes the model for distributed training environments, reducing the need for complex synchronization and communication between parallel processes. It aligns the architecture with modern hardware accelerators, ensuring efficient scaling across numerous GPUs.

Rotary Positional Embeddings (RoPE)

The Falcon models utilizes RoPE to encode positional information within sequences, a departure from the absolute positional vector embeddings traditionally used in Transformers. RoPE offers several advantages:

Relative Positional Information: RoPE embeds the relative positions of tokens in a sequence, facilitating the model's understanding of sequence structure and context. This is particularly beneficial for tasks involving nuanced understanding of language structure.
Efficiency and Performance: Despite its sophistication, RoPE is designed to be computationally efficient, ensuring that the additional positional context does not come at the expense of training or inference speed.

Activation Functions: GELU over GLU

The choice of activation function is critical in the model's ability to learn complex patterns. GELU (Gaussian Error Linear Unit) is selected for its proven effectiveness in deep learning models; GELU provides a non-linear activation that allows the model to learn more complex functions than traditional ReLU without the additional computational burden imposed by GLUs (Gated Linear Units).

Parallelization and Efficiency

Parallel Attention and MLP Layers

The Falcon architecture employs parallel processing of attention and MLP (multi-layer perceptron) layers, a design choice significantly reducing training time. By parallelizing access to these components, Falcon minimizes the bottlenecks associated with sequential processing, allowing for faster forward and backward passes during training.

No Biases in Linear Layers

In a move to streamline the model and improve stability, the Falcon series omits biases in linear layers:

Simplicity and Stability: This simplification reduces the number of parameters and potential sources of instability during training, contributing to the model's robustness and efficiency.
Architecture Innovations: The Falcon models' architectural innovations are not arbitrary but are deeply motivated by the goals of scalability, efficiency, and performance. Each design decision, from multigroup attention to parallel processing layers, is made with scalability in mind. The architecture is crafted to ensure that as the model size increases, it remains trainable and efficient on available hardware. Inference efficiency takes a high priority as well, particularly for models intended for wide deployment. The Falcon models addresses this through optimizations like multiquery attention and RoPE, ensuring that the model can deliver real-time responses even in complex generative tasks. Pure performance does take a hit, but Falcon's architecture is optimized to maintain or improve performance across a range of natural language processing(NLP) tasks, ensuring that the Falcon series models are competitive with the state-of-the-art.

The Falcon creators adopted a forward-thinking approach to designing large-scale language models. Through a combination of innovative attention mechanisms, efficient positional embeddings, and streamlined network components, the Falcon series models sets a new standard for what is possible in natural language processing.

The Dataset Composition

The dataset composition and deduplication strategy developed for the Falcon series of language models represent critical development aspects underpinning the model's performance and efficiency.

High-Quality Web Data

The Falcon models leverages an extensive English web dataset, amassing over 5,000 billion tokens. This dataset is curated through stringent filtering processes to ensure high quality, challenging the conventional necessity of including curated corpora from sources like books, technical papers, and other traditionally "high-quality" content. The focus on web data arises from a nuanced understanding that with adequate processing, web data can yield competitive, if not superior, model performance.

Focus on Scalability and Quality

The dataset's scale and quality are balanced to optimize model training efficiency and performance. The preference for web data is also strategic, aiming to mitigate the inference burden that typically grows with model size. Increasing the pretraining dataset size is notably advantageous as it is decoupled from inference costs, unlike model size increments.

Strategic Composition

The dataset composition is a testament to the Falcon team's commitment to leveraging scalable data collection and processing methods. It reflects a comprehensive approach where the breadth of the English web is distilled into a potent training dataset through processes that prioritize data quality and relevance.

The Deduplication Strategy

Rigorous Deduplication

Deduplication stands as a cornerstone of the Falcon dataset's integrity. The strategy involves two stages of deduplication to rigorously ensure that no data instance is repeated during the model's training. This approach addresses the degradation in model performance associated with data repetition and is pivotal in maintaining the dataset's quality.

Motivation and Implementation

The deduplication strategy is motivated by research indicating that naive repetition of data can degrade model performance, leading to concerns about the sustainability of scaling datasets. Falcon's deduplication process involves sophisticated filtering and identification techniques to remove duplicates effectively.

Benefits and Outcomes

By eliminating redundancies, the Falcon series conserves computational resources and ensures that the training process is focused on diverse data instances, enhancing the model's ability to generalize from its own training data corpus. This meticulous approach to deduplication contributes significantly to the model's impressive performance metrics, particularly in zero-shot learning and few-shot generalizations.

Key Insights and Innovations of Falcon Models

Innovation in Web Data Utilization: Falcon’s dataset composition strategy showcases an innovative approach to using web data to train state-of-the-art language models. By demonstrating that web data, when properly filtered and deduplicated, can rival or surpass the quality of curated datasets, the Falcon series challenges prevailing norms in dataset composition for large language models). The Falcon models are a valuable tool for researchers and developers, offering flexibility and robust performance essential for enhancing various applications in natural language processing.

Scalability and Efficiency: The emphasis on deduplication and quality over sheer quantity aligns with the broader design philosophy of the Falcon series, which prioritizes scalability and computational efficiency. This approach ensures that advancements in dataset processing and model architecture sustainably support the growth in model capabilities.

Impact on Model Performance: Deduplication of the dataset directly impacts the performance of the Falcon models. The creators include a large-scale deduplication process to ensure the model is trained on diverse data. Additionally, the instruct models within the Falcon series demonstrate strong effectiveness in reasoning and providing truthful answers, making them competitive within the market and encouraging users to fine-tune their own models for better results.

The Falcon series’ dataset composition and deduplication strategy exemplify cutting-edge practices in developing large-scale language models, combining innovation in data processing with a whole community and steadfast commitment to quality and efficiency.

Wrapping up

The Falcon models demonstrate remarkable performance across various datasets and tasks, mainly showcasing their strength in zero-shot and few-shot settings. Their design and training strategies yield models that advance the state of the art in natural language processing and improve model training and deployment efficiency and scalability.

The Falcon series emphasizes data quality, architectural optimizations, and systematic training strategies and sets a new benchmark for large-scale language model development.

In addition to these strengths, Falcon models are great at text generation and text generation inference, facilitating high-quality and efficient text generation run inference processes. They are also adept at language translation, showcasing their versatility in various NLP tasks. Keep an eye on the MTEB leaderboard hosted by Hugging Face to see how Falcon 180b does against the other models.

Frank Liu
Frank Liu is the Director of Operations & ML Architect at Zilliz, where he serves as a maintainer for the Towhee open-source project. Prior to Zilliz, Frank co-founded Orion Innovations, an ML-powered indoor positioning startup based in Shanghai and worked as an ML engineer at Yahoo in San Francisco. In his free time, Frank enjoys playing chess, swimming, and powerlifting. Frank holds MS and BS degrees in Electrical Engineering from Stanford University.

Next: OpenAI Whisper: Transforming Speech-to-Text with Advanced AI

Content

Start Free, Scale Easily

Try the fully-managed vector database built for your GenAI applications.

Try Zilliz Cloud for Free