Hosting and Scaling GenAI Workloads on Amazon EKS
Pre-conference workshop (2.5 hours)
Abstract
Deploy, serve, and scale GenAI models on Amazon EKS with NVIDIA GPUs. This workshop covers GPU cluster setup, model serving with vLLM, observability, and your choice of scaling with Ray Serve or inference optimization with KV cache offloading.
What the workshop covers
- GPU cluster setup — stand up GPU-enabled Amazon EKS clusters for GenAI workloads.
- Model serving with vLLM — deploy and serve models on NVIDIA GPUs.
- Observability — instrument and observe inference workloads.
- Scaling and optimization — scale out with Ray Serve, or optimize inference with KV cache offloading.
Facilitators
- Frank Fan · Principal Solutions Architect, Containers, AWS
- Sheng Chen · Sr. Infrastructure Modernization Specialist SA, AWS
- Navya Srivastava · Senior ISV Solutions Architect, AWS
- Parikshit Patel · Sr. Cloud Support Engineer (Containers), AWS
Prerequisites
Bring your own laptop.