Building a Scalable, Flexible, Cloud-Native GenAI Platform with Open Source Solutions

Core Architecture: Two-Tier Gateway Design ​ Tier One Gateway ​ Tier Two Gateway ​ Design Benefits ​ Routing and Traffic Management ​ Self-Hosted Model Serving with KServe ​ Observability, Control, and Optimization for Production Readiness ​ Observability ​ Control ​ Optimization ​ Pluggable and Flexible ​ Summary ​ Posted on August 28, 2025 by Takeshi Yoneda, Envoy Maintainer and Open Source Software Engineer at Tetrate AI workloads are complex, and unmanaged complexity kills velocity. Your architecture is the key to mastering it. As generative AI (GenAI) becomes foundational to modern software products, developers face a chaotic new reality, juggling different APIs from various providers while also attempting to deploy self-hosted open-source models. This leads to credential sprawl, inconsistent security policies, runaway costs, and an infrastructure that is difficult to scale and govern. Your architecture doesn’t have to be this complex. Platform engineering teams need a secure, scalable way to serve both internal and external LLMs to their users. That’s where tools such as Envoy AI Gateway come in. This reference architecture outlines how to build a flexible GenAI platform using the open source solutions Envoy AI Gateway, KServe, and complementary tools. Whether you’re self-hosting models or integrating with model-serving services on cloud providers, such as OpenAI and Anthropic, this architecture enables a unified, governable interface for LLM traffic. The foundation of this platform is a Two-Tier Gateway Architecture: Deployed in a centralized Gateway Cluster. It serves as the main API traffic entry point for client applications. Its Job: To route traffic to external LLM providers (e.