Building Hybrid AI Systems: Edge Inferencing Meets Cloud Reasoning
Full length talk (20 minutes)
Abstract
Modern AI applications increasingly need to run closer to where data is generated and decisions are made. This session presents an architectural pattern for hybrid AI systems that split latency-sensitive inference at the edge from cloud-native reasoning, connected through a shared memory layer.
Using a real-world warehouse safety monitoring system as the case study, we examine the trade-offs for placing a Vision Language Model (VLM) on edge GPUs using a hybrid Kubernetes architecture, while delegating summarisation and conversational retrieval to a cloud-based agent runtime.
We cover the decision framework for what runs where, agentic memory strategies, and cross-environment observability.
Key takeaways:
- A decision framework for what belongs at the edge vs. the cloud
- Choosing the right models for your edge applications
- Squeezing performance with quantisation, decoding and cold-start optimisation
- Observing GPU and model health across hybrid environments
- Using a shared memory layer to turn edge inference into business insight
Attendees will leave with practical patterns for building their own hybrid AI systems - from the edge/cloud decision boundary through to agentic memory and cross-environment observability.
About Sheng Chen
Sheng Chen is a Sr. Solutions Architect at AWS Australia, bringing over 20 years of experience in IT infrastructure and cloud architecture. In his current role, Sheng helps customers accelerate cloud migrations and infrastructure modernisation by leveraging cloud-native technologies. He specialises in Amazon EKS, AWS hybrid cloud services, platform engineering, HPC and AI infrastructure.
About Frank Fan
Frank Fan, an AWS Principal Container Solution Architect in Australia, has 20+ years of experience in designing and implementing large-scale technology transformations. He specializes in containerization and driving migration initiatives. He has presented at significant IT events like AWS reInvent, AWS Summit and API days. Frank's youtue channel https://www.youtube.com/@gettocloud5719