Gajan Kugamoorthy - Vivanti

The Self-Hosted LLMs: Secure & Scalable Inference with Kubernetes

Abstract

For organizations concerned with data privacy, cost control, or model customisation, relying on third-party AI services isn't always an option. This reality places the challenge of building a private, efficient, and secure platform directly onto their engineers, architects, and teams. This talk delivers the technical blueprint to meet that challenge: a practical framework covering the architecture, security guardrails, and scalable operational practices. In this talk, you will learn how to build a production-ready LLM inference platform using a powerful, purely open-source stack on Kubernetes. You will learn how to deploy leading models like Llama 3 with high-performance servers, implement dynamic, cost-saving autoscaling with event-driven tools (e.g. KEDA), and enforce robust security guardrails using Kubernetes-native policy engines (e.g. Kyverno). Through a live, interactive demo, you will witness these principles in action: from deploying a model and watching it scale under load, to seeing a non-compliant deployment get blocked in real-time. You will leave with the practical knowledge to build your own secure, high-performance LLM infrastructure.

Speaker Bio

As a Senior Consultant at Vivanti, Gajan specialises in cloud infrastructure, security, and cloud data platforms. He has extensive experience leading large-scale Kubernetes platform modernisation programs for major Australian banks and architecting secure cloud data platforms for health device manufacturers. He is one of only a handful of CNCF certified Kubestronauts in Australia. His deep technical expertise is built on a foundation of over 7 years at IBM, where he developed cloud-native security solutions and contributed to more than 15 patents. Recognised by IBM as an award-winning open-source contributor, Gajan is a key contributor to a diverse range of projects, including OpenSSF's Minder, Red Hat's open-cluster-management.io, and Integrity Shield. This hands-on, real-world experience gives him a practical perspective on building the secure platforms he will discuss in his talk.