pretraining

Efficient Pre-training of Llama 3-like model architectures using torchtitan on Amazon SageMaker | Amazon Web Services

by
Efficient Pre-training of Llama 3-like model architectures using torchtitan on Amazon SageMaker | Amazon Web Services

This post is co-written with Less Wright and Wei Feng from Meta Pre-training large language models (LLMs) is the first step in developing powerful AI systems that can understand and generate human-like text. By exposing models… Article Source https://aws.amazon.com/blogs/machine-learning/efficient-pre-training-of-llama-3-like-model-architectures-using-torchtitan-on-amazon-sagemaker/

Improve Mixtral 8x7B pre-training speed with expert parallelism on Amazon SageMaker | Amazon Web Services

by
Improve Mixtral 8x7B pre-training speed with expert parallelism on Amazon SageMaker | Amazon Web Services

Mixture of Experts (MoE) architectures are gaining popularity for large language models (LLMs) due to their ability to increase model capacity and computational efficiency compared to fully dense models. MoE models utilize sparse expert subnetworks that process different subsets of tokens, allowing for a higher number of parameters with less computation per token during training … Read more