Adrian Cole - Tetrate

AI Observability with Envoy AI Gateway: From Black Box to Glass Box

Abstract

When GenAI apps break in production, traditional metrics fall short, leaving you guessing amid API chaos, cost overruns, and security gaps. This talk introduces Envoy AI Gateway—a CNCF project cofounded by Tetrate and Bloomberg—that unifies LLM traffic and turns black boxes into observable systems.

We'll explore AI-specific observability patterns using OpenTelemetry and OpenInference, making debugging straightforward again. This includes metrics that matter, such as time-to-first-token and inter-token latency. We'll review the OpenInference tracing format, which allows you to LLM-as-a-Judge your production traffic.

We’ll demo Envoy AI Gateway observability using Arize Phoenix as the LLM tracing and evaluation backend. You’ll leave with dots connected between traditional and GenAI observability, in a production gateway.

Speaker Bio

Adrian is a principal engineer working at Tetrate. He’s been a routine contributor to open source for over fifteen years. Lately, he spends most of his time on Envoy AI Gateway. His past notable project work includes OpenTelemetry, wazero, Zipkin, OpenFeign, and Apache jclouds.