1042
Last week, Meta released a new AI model, the Llama 3.1 405B. The company immediately began to position it as the most powerful open source model that is freely available.
What is the difference between an open model and a closed one? What can Meta's Llama 3.1 405B do? And how will this innovation affect the development of artificial intelligence in the future? IT-News editorial office provided answers to these and other questions.
Model architecture
The first thing that distinguishes the new artificial intelligence model Llama from Meta is its architecture and learning process. Llama 3.1 405B consists of a decoder that supports learning stability.
The model was trained on 15 trillion tokens. To enable the processing of information of this scale, training was transferred to 16 thousand H100 graphics processors, and the model itself was quantized from 16-bit to 8-bit.
The training process was carried out in two stages:
-
Previous training. The researchers tokenized the textual materials and forced the LLMs to perform special tasks to understand the structure of the language.
-
Post-training. Developers fine-tuned the model, leveling its response to information requests.
Special attention was paid to the chat. The context window of the model was increased to 128 thousand tokens. This means that Llama 3.1 405B understands large texts better and correctly evaluates the context.
How powerful is the Llama 3.1 405B model
150 benchmark data sets were used to evaluate performance. The results showed that open source model Llama 3.1 405B can compete with leading closed source models such as GPT-4 and Claude 3.5 Sonnet.
Benefit to users
Llama 3.1 405B received 405 billion parameters. She shows excellent skills in working with general knowledge, she is easy to manage. The model perfectly understands complex contexts, knows eight languages and can solve mathematical equations. It generates text quickly, can summarize large volumes of data and responds quickly to user requests. This makes it an indispensable assistant in work and study.
A profitable tool for developers
Llama 3.1 405B can generate synthetic data. Thanks to them, you can quickly train new models of generative AI. In addition, Llama 3.1 is endowed with model distillation, which allows you to transfer its functions to other models of artificial intelligence.
The Llama 3.1 405B can connect to external instruments to expand its capabilities. For example, use search optimization tools, write code, etc.
Will the Llama 3.1 405B be a driver of progress
Meta notes that the launch of Llama 3.1 405B will accelerate the development of innovation, as it provides unprecedented opportunities for rapid software development.
The open source AI model is unique in that its architecture can be copied and modified as needed. Llama 3.1 405B can run on one server in the developer environment, which means that the information processed by this model will not get to the general server of the developer and will not be intentionally or unintentionally used.
Llama 3.1 405B, capable of creating synthetic data that can be used to create new AI models with unique features.
Open source ensures that more people will have access to the generative functions of artificial intelligence. In turn, more users will get unique opportunities to create and integrate innovative tools.
