Deploy production generative AI at the edge using Amazon EKS Hybrid Nodes with NVIDIA DGX

Deploy production generative AI at the edge using Amazon EKS Hybrid Nodes with NVIDIA DGX Solution overview Prerequisites Walkthrough Prepare EKS Hybrid Nodes Install NVIDIA GPU Operator for Kubernetes Deploy NVIDIA NIM for inference on EKS Hybrid Nodes Configure centralized monitoring and observability for GPU metrics Cleaning up Conclusion About the authors Modern generative AI applications require deployment closer to where data is generated and business decisions are made, but this creates new infrastructure challenges. Organizations in manufacturing, healthcare, finance, and telecommunications need to deliver low-latency, energy-efficient AI workloads at the edge while maintaining data locality and regulatory compliance. However, managing Kubernetes on-premises adds operational complexity that can slow down innovation. You can use Amazon Elastic Kubernetes Service ( Amazon EKS ) Hybrid Nodes to address this by joining on-premises infrastructure to the Amazon EKS control plane as remote nodes. This allows you to accelerate AI workload deployment with consistent operational practices, while addressing latency, compliance, and data residency requirements. EKS Hybrid Nodes removes the complexity and burden of self-managing Kubernetes on-premises so that your team can focus on deploying AI applications and driving innovations. It provides unified workflows and tooling alongside centralized monitoring and enhanced observability across your distributed infrastructure. EKS Hybrid Nodes enables you to deliver AI capabilities wherever your business demands, such as the following use cases: Run low-latency services at on-premises locations, including real-time inference at the edge Train models with data that must remain on-premises to meet regulatory compliance requirements Deploy inference workloads near source data, such as Retrieval-Augmented Generation (RAG) applications using a local knowledge base Repurpose existing hardware investment This post demonstrates a real-world example of integrating EKS Hybrid Nodes with NVIDIA DGX Spark , a compact and energy-efficient GPU platform optimized for edge AI deployment. In this post we walk you through deploying a large language model (LLM) for low-latency generative AI inference on-premises, setting up node monitoring and GPU observability with centralized management through Amazon EKS. Although this post uses DGX Spark, the architecture and patterns discussed apply to other NVIDIA DGX systems or GPU platforms. For this demo walkthrough, you create an EKS cluster with EKS Hybrid Nodes enabled, and connect an on-premises DGX Spark as a hybrid node. You install the NVIDIA GPU Operator for Kubernetes to provision GPU resources for the local generative AI inference.