8x7B

Mixture of Experts (MoE) architectures are gaining popularity for large language models (LLMs) due to their ability to increase model capacity and computational efficiency compared to fully dense models. MoE models utilize sparse expert subnetworks that process different subsets of tokens, allowing for a higher number of parameters with less computation per token during training … Read more