P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM | Amazon Web Services
EAGLE is the state-of-the-art method for speculative decoding in large language model (LLM) inference, but its autoregressive drafting creates a…
Tag: vLLM
Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon Bedrock | Amazon Web Services
Organizations and individuals running multiple custom AI models, especially recent Mixture of Experts (MoE) model families, can face the challenge…
Red Hat OpenShift 4.20: AI, post-quantum, and broader VM assist
Red Hat has launched model 4.20 of OpenShift. The answer will get new AI tooling, post-quantum encryption, and extra in…
Microsoft expands AKS with RAG functionality and vLLM support
During KubeCon, Microsoft announced that it supports Retrieval Augmented Generation (RAG) in KAITO on Azure Kubernetes Service (AKS) clusters. In…
Inferencing with vLLM and Triton on NVIDIA Jetson AGX Orin
NVIDIA’s Triton Inference Server is an open-source inference service framework designed to facilitate the rapid development of AI/ML inference applications.…