Gateway architecture
Learn about the architecture and design principles of NGINX Gateway Fabric.
The intended audience for this information is primarily the two following groups:
- Cluster Operators who would like to know how the software works and understand how it can fail.
- Developers who would like to contribute to the project.
The reader needs to be familiar with core Kubernetes concepts, such as pods, deployments, services, and endpoints. For an understanding of how NGINX itself works, you can read the “Inside NGINX: How We Designed for Performance & Scale” blog post.
NGINX Gateway Fabric is an open source project that provides an implementation of the Gateway API using NGINX as the data plane. The goal of this project is to implement the core Gateway APIs – Gateway, GatewayClass, HTTPRoute, GRPCRoute, TCPRoute, TLSRoute, and UDPRoute – to configure an HTTP or TCP/UDP load balancer, reverse proxy, or API gateway for applications running on Kubernetes. NGINX Gateway Fabric supports a subset of the Gateway API.
For a list of supported Gateway API resources and features, see the Gateway API Compatibility documentation.
NGINX Gateway Fabric separates the control plane and data plane into distinct deployments. This architectural separation enhances scalability, security, and operational isolation between the two components.
The control plane interacts with the Kubernetes API, watching for Gateway API resources. When a new Gateway resource is provisioned, it dynamically creates and manages a corresponding NGINX data plane Deployment and Service. This ensures that the system can adapt to changes in the cluster state seamlessly.
Each NGINX data plane pod consists of an NGINX container integrated with the NGINX agent. The agent securely communicates with the control plane using gRPC. The control plane translates Gateway API resources into NGINX configurations and sends these configurations to the agent to ensure consistent traffic management.
This design enables centralized management of multiple Gateways while ensuring that each NGINX instance stays aligned with the cluster’s current configuration. Labels, annotations, and infrastructure settings such as service type or replica count can be specified globally via the Helm chart or customized per Gateway using the enhanced NginxProxy CRD and the Gateway’s infrastructure section.
We have more information regarding our design principles in the project’s GitHub repository.
The NGINX Gateway Fabric architecture separates the control plane and data plane into distinct and independent Deployments, ensuring enhanced security, flexibility, and resilience.
The control plane operates as a Deployment, serving as a Kubernetes controller built with the controller-runtime library. It manages all aspects of resource provisioning and configuration for the NGINX data planes by watching Gateway API resources and other Kubernetes objects such as Services, Endpoints, and Secrets.
Key functionalities include:
- Dynamic provisioning: When a new Gateway resource is created, the control plane automatically provisions a dedicated NGINX Deployment and exposes it using a Service.
- Configuration management: Kubernetes and Gateway API resources are translated into NGINX configurations, which are securely delivered to the data plane pods via a gRPC connection to the NGINX Agent.
- Secure communication: By default, the gRPC connection uses self-signed certificates generated during installation. Integration with cert-manager is also supported for optional certificate management.
Each NGINX data plane pod is provisioned as an independent Deployment containing an nginx container. This container runs both the nginx process and the NGINX agent, which is responsible for:
- Applying configurations: The agent receives updates from the control plane and applies them to the NGINX instance.
- Handling reloads: NGINX Agent handles configuration reconciliation and reloading NGINX, eliminating the need for shared volumes or Unix signals between the control plane and data plane pods.
With this design, multiple NGINX data planes can be managed by a single control plane, enabling fine-grained, Gateway-specific control and isolation.
The architecture supports flexible operation and isolation across multiple Gateways:
- Concurrent Gateways: Multiple Gateway objects can run simultaneously within a single installation.
- 1:1 resource mapping: Each Gateway resource corresponds uniquely to a dedicated data plane deployment, ensuring clear delineation of ownership and operational segregation.
One of the primary advantages of this architecture is enhanced operational resilience and fault isolation:
In the event of a control plane failure or downtime:
- Existing data plane pods continue serving traffic using their last-valid cached configurations.
- Updates to routes or Gateways are temporarily paused, but stable traffic delivery continues without degradation.
- Recovery restores functionality, resynchronizing configuration updates seamlessly.
If a data plane pod encounters an outage or restarts:
- Only routes tied to the specific linked Gateway object experience brief disruptions.
- Configurations automatically resynchronize with the data plane upon pod restart, minimizing the scope of impact.
- Other data plane pods remain unaffected and continue serving traffic normally.
This split architecture ensures operational boundaries between the control plane and data plane, delivering improved scalability, security, and robustness while minimizing risks associated with failures in either component.
This figure depicts an example of NGINX Gateway Fabric exposing three web applications within a Kubernetes cluster to clients on the internet:
graph LR
%% Nodes and Relationships
subgraph KubernetesCluster[Kubernetes Cluster]
subgraph applications2[Namespace: applications2]
subgraph DataplaneComponentsC[Dataplane Components]
GatewayC[Gateway C
Listener: *.other-example.com]
subgraph NGINXPodC[NGINX Pod]
subgraph NGINXContainerC[NGINX Container]
NGINXProcessC(NGINX)
NGINXAgentC(NGINX Agent)
end
end
end
subgraph HTTPRouteCAndApplicationC[HTTPRoute C and Application C]
HTTPRouteC[HTTPRoute C
Host: c.other-example.com]
ApplicationC[Application C
Pods: 1]
end
end
subgraph nginx-gateway[Namespace: nginx-gateway]
NGFPod[NGF Pod]
end
subgraph applications1[Namespace: applications]
subgraph DataplaneComponentsAB[Dataplane Components]
GatewayAB[Gateway AB
Listener: *.example.com]
subgraph NGINXPodAB[NGINX Pod]
subgraph NGINXContainerAB[NGINX Container]
NGINXProcessAB(NGINX)
NGINXAgentAB(NGINX Agent)
end
end
end
subgraph HTTPRouteBAndApplicationB[HTTPRoute B and Application B]
HTTPRouteB[HTTPRoute B
Host: b.example.com]
ApplicationB[Application B
Pods: 1]
end
subgraph HTTPRouteAAndApplicationA[HTTPRoute A and Application A]
HTTPRouteA[HTTPRoute A
Host: a.example.com]
ApplicationA[Application AB
Pods: 2]
end
end
KubernetesAPI[Kubernetes API]
end
subgraph Users[Users]
ClusterOperator[Cluster Operator]
AppDevA[Application Developer A]
AppDevB[Application Developer B]
AppDevC[Application Developer C]
end
subgraph Clients[Clients]
ClientsA[Clients A]
ClientsB[Clients B]
ClientsC[Clients C]
end
subgraph "Public Endpoints"
PublicEndpointAB[Public Endpoint AB
TCP Load Balancer/NodePort]
PublicEndpointC[Public Endpoint C
TCP Load Balancer/NodePort]
end
%% Updated Traffic Flow
ClientsA == a.example.com ==> PublicEndpointAB
ClientsB == b.example.com ==> PublicEndpointAB
ClientsC == c.other-example.com ==> PublicEndpointC
PublicEndpointAB ==> NGINXProcessAB
PublicEndpointC ==> NGINXProcessC
NGINXProcessAB ==> ApplicationA
NGINXProcessAB ==> ApplicationB
NGINXProcessC ==> ApplicationC
%% Kubernetes Configuration Flow
HTTPRouteA --> GatewayAB
HTTPRouteB --> GatewayAB
HTTPRouteC --> GatewayC
NGFPod --> KubernetesAPI
NGFPod --gRPC--> NGINXAgentAB
NGINXAgentAB --> NGINXProcessAB
NGFPod --gRPC--> NGINXAgentC
NGINXAgentC --> NGINXProcessC
ClusterOperator --> KubernetesAPI
AppDevA --> KubernetesAPI
AppDevB --> KubernetesAPI
AppDevC --> KubernetesAPI
%% Styling
style ClusterOperator fill:#66CDAA,stroke:#333,stroke-width:2px
style GatewayAB fill:#66CDAA,stroke:#333,stroke-width:2px
style GatewayC fill:#66CDAA,stroke:#333,stroke-width:2px
style NGFPod fill:#66CDAA,stroke:#333,stroke-width:2px
style NGINXProcessAB fill:#66CDAA,stroke:#333,stroke-width:2px
style NGINXProcessC fill:#66CDAA,stroke:#333,stroke-width:2px
style KubernetesAPI fill:#9370DB,stroke:#333,stroke-width:2px
style HTTPRouteAAndApplicationA fill:#E0FFFF,stroke:#333,stroke-width:2px
style HTTPRouteBAndApplicationB fill:#E0FFFF,stroke:#333,stroke-width:2px
style AppDevA fill:#FFA07A,stroke:#333,stroke-width:2px
style HTTPRouteA fill:#FFA07A,stroke:#333,stroke-width:2px
style ApplicationA fill:#FFA07A,stroke:#333,stroke-width:2px
style ClientsA fill:#FFA07A,stroke:#333,stroke-width:2px
style AppDevB fill:#87CEEB,stroke:#333,stroke-width:2px
style HTTPRouteB fill:#87CEEB,stroke:#333,stroke-width:2px
style ApplicationB fill:#87CEEB,stroke:#333,stroke-width:2px
style ClientsB fill:#87CEEB,stroke:#333,stroke-width:2px
style AppDevC fill:#FFC0CB,stroke:#333,stroke-width:2px
style HTTPRouteC fill:#FFC0CB,stroke:#333,stroke-width:2px
style ApplicationC fill:#FFC0CB,stroke:#333,stroke-width:2px
style ClientsC fill:#FFC0CB,stroke:#333,stroke-width:2px
style PublicEndpointAB fill:#FFD700,stroke:#333,stroke-width:2px
style PublicEndpointC fill:#FFD700,stroke:#333,stroke-width:2px
%% Styling
classDef dashedSubgraph stroke-dasharray: 5, 5;
%% Assign Custom Style Classes
class DataplaneComponentsAB dashedSubgraph;
class DataplaneComponentsC dashedSubgraph;
Note: The figure does not show many of the necessary Kubernetes resources the Cluster Operators and Application Developers need to create, like deployment and services.
The figure shows:
- A Kubernetes cluster.
- Users Cluster Operator, Application Developer A, B and C. These users interact with the cluster through the Kubernetes API by creating Kubernetes objects.
- Clients A, B, and C connect to Applications A, B, and C respectively, which the developers have deployed.
- The NGF Pod, deployed by Cluster Operator in the namespace nginx-gateway. For scalability and availability, you can have multiple replicas. The NGF container interacts with the Kubernetes API to retrieve the most up-to-date Gateway API resources created within the cluster. When a new Gateway resource is provisioned, the control plane dynamically creates and manages a corresponding NGINX data plane Deployment and Service. It watches the Kubernetes API and dynamically configures these NGINX deployments based on the Gateway API resources, ensuring proper alignment between the cluster state and the NGINX configuration.
- The NGINX Pod consists of an NGINX container and the integrated NGINX agent, which securely communicates with the control plane over gRPC. The control plane translates Gateway API resources into NGINX configuration, and sends the configuration to the agent.
- Gateways Gateway AB and Gateway C, created by Cluster Operator, request points where traffic can be translated to Services within the cluster. Gateway AB, includes a listener with a hostname
*.example.com. Gateway C, includes a listener with a hostname*.other-example.com. Application Developers have the ability to attach their application’s routes to the Gateway AB if their application’s hostname matches*.example.com, or to Gateway C if their application’s hostname matches*.other-example.com - Application A with two pods deployed in the applications namespace by Application Developer A. To expose the application to its clients (Clients A) via the host
a.example.com, Application Developer A creates HTTPRoute A and attaches it toGateway AB. - Application B with one pod deployed in the applications namespace by Application Developer B. To expose the application to its clients (Clients B) via the host
b.example.com, Application Developer B creates HTTPRoute B and attaches it toGateway AB. - Application C with one pod deployed in the applications2 namespace by Application Developer C. To expose the application to its clients (Clients C) via the host
c.other-example.com, Application Developer C creates HTTPRoute C and attaches it toGateway C. - Public Endpoint AB, and Public Endpoint C and which fronts the NGINX AB, and NGINX C pods respectively. A public endpoint is typically a TCP load balancer (cloud, software, or hardware) or a combination of such load balancer with a NodePort service. Clients A and B connect to their applications via the Public Endpoint AB, and Clients C connect to their applications via the Public Endpoint C.
- The bold arrows represent connections related to the client traffic. Note that the traffic from Clients C to Application C is completely isolated from the traffic between Clients A and B and Application A and B respectively.
The resources within the cluster are color-coded based on the user responsible for their creation. For example, the Cluster Operator is denoted by the color green, indicating they create and manage all the green resources.
graph LR
%% Main Components
KubernetesAPI[Kubernetes API]
PrometheusMonitor[Prometheus]
F5Telemetry[F5 Telemetry Service]
NGFPod[NGF Pod]
NGINXPod[NGINX Pod]
Client[Client]
Backend[Backend]
%% NGINX Pod Grouping
subgraph NGINXPod[NGINX Pod]
NGINXAgent[NGINX Agent]
NGINXMaster[NGINX Master]
NGINXWorker[NGINX Worker]
ConfigFiles[Config Files]
ContainerRuntimeNGINX[stdout/stderr]
end
subgraph NGFPod[NGF Pod]
NGFProcess[NGF Process]
ContainerRuntimeNGF[stdout/stderr]
end
%% External Components Grouping
subgraph ExternalComponents[.]
KubernetesAPI[Kubernetes API]
PrometheusMonitor[Prometheus]
F5Telemetry[F5 Telemetry Service]
end
%% HTTPS: Communication with Kubernetes API
NGFProcess -- "(1) Reads Updates" --> KubernetesAPI
NGFProcess -- "(1) Writes Statuses" --> KubernetesAPI
%% Prometheus: Metrics Collection
PrometheusMonitor -- "(2) Fetches controller-runtime metrics" --> NGFPod
PrometheusMonitor -- "(5) Fetches NGINX metrics" --> NGINXWorker
%% Telemetry: Product telemetry data
NGFProcess -- "(3) Sends telemetry data" --> F5Telemetry
%% File I/O: Logging
NGFProcess -- "(4) Write logs" --> ContainerRuntimeNGF
NGINXMaster -- "(11) Write logs" --> ContainerRuntimeNGINX
NGINXWorker -- "(12) Write logs" --> ContainerRuntimeNGINX
%% gRPC: Configuration Updates
NGFProcess -- "(6) Sends Config to Agent" --> NGINXAgent
NGINXAgent -- "(7) Validates & Writes Config & TLS Certs" --> ConfigFiles
NGINXAgent -- "(8) Reloads NGINX" --> NGINXMaster
NGINXAgent -- "(9) Sends DataPlaneResponse" --> NGFProcess
%% File I/O: Configuration and Secrets
NGINXMaster -- "(10) Reads TLS Secrets" --> ConfigFiles
NGINXMaster -- "(11) Reads nginx.conf & NJS Modules" --> ConfigFiles
%% Signals: Worker Lifecycle Management
NGINXMaster -- "(14) Manages Workers (Update/Shutdown)" --> NGINXWorker
%% Traffic Flow
Client -- "(15) Sends Traffic" --> NGINXWorker
NGINXWorker -- "(16) Routes Traffic" --> Backend
%% Styling
classDef important fill:#66CDAA,stroke:#333,stroke-width:2px;
classDef metrics fill:#FFC0CB,stroke:#333,stroke-width:2px;
classDef io fill:#FFD700,stroke:#333,stroke-width:2px;
classDef signal fill:#87CEEB,stroke:#333,stroke-width:2px;
style ExternalComponents fill:transparent,stroke-width:0px
%% Class Assignments for Node Colors
class NGFPod,KubernetesAPI important;
class PrometheusMonitor,F5Telemetry metrics;
class ConfigFiles,NGINXMaster,NGINXWorker,NGINXAgent io;
class Client,Backend signal;
The following list describes the connections, preceeded by their types in parentheses. For brevity, the suffix “process” has been omitted from the process descriptions.
- (HTTPS)
- Read: NGF reads the Kubernetes API to get the latest versions of the resources in the cluster.
- Write: NGF writes to the Kubernetes API to update the handled resources’ statuses and emit events. If there’s more than one replica of NGF and leader election is enabled, only the NGF pod that is leading will write statuses to the Kubernetes API.
- (HTTP, HTTPS) Prometheus fetches the
controller-runtimemetrics via an HTTP endpoint that NGF exposes (:9113/metricsby default). Prometheus is not required by NGINX Gateway Fabric, and its endpoint can be turned off. - (HTTPS) NGF sends product telemetry data to the F5 telemetry service.
- (File I/O) NGF writes logs to its stdout and stderr, which are collected by the container runtime.
- (HTTP, HTTPS) Prometheus fetches the NGINX metrics via an HTTP endpoint that NGINX exposes (
:9113/metricsby default). Prometheus is not required by NGINX, and its endpoint can be turned off. - (gRPC) NGF generates NGINX configuration based on the cluster resources and sends them to NGINX Agent over a secure gRPC connection.
- NGF sends a message containing file metadata to all pods (subscriptions) for the deployment.
- Agent receives a ConfigApplyRequest with the list of file metadata.
- Agent calls GetFile for each file in the list, which NGF sends back to the agent.
- (File I/O)
- Write: _NGINX Agent validates the received configuration, and then writes and applies the config if valid. It also writes TLS certificates and keys from TLS secrets referenced in the accepted Gateway resource.
- (Signal) To reload NGINX, Agent sends the reload signal to the NGINX master.
- (gRPC) Agent responds to NGF with a DataPlaneResponse.
- (File I/O)
- Read: The NGINX master reads configuration files and the TLS cert and keys referenced in the configuration when it starts or during a reload.
- (File I/O)
- Read: The NGINX master reads the
nginx.conffile from the/etc/nginxdirectory. This file contains the global and http configuration settings for NGINX. In addition, NGINX master reads the NJS modules referenced in the configuration when it starts or during a reload. NJS modules are stored in the/usr/lib/nginx/modulesdirectory.
- Read: The NGINX master reads the
- (File I/O) The NGINX master sends logs to its stdout and stderr, which are collected by the container runtime.
- (File I/O) An NGINX worker writes logs to its stdout and stderr, which are collected by the container runtime.
- (Signal) The NGINX master controls the lifecycle of NGINX workers it creates workers with the new configuration and shutdowns workers with the old configuration.
- (HTTP, HTTPS) A client sends traffic to and receives traffic from any of the NGINX workers on ports 80 and 443.
- (HTTP, HTTPS) An NGINX worker sends traffic to and receives traffic from the backends.
The previous diagram depicts NGINX Gateway Fabric using NGINX Open Source. NGINX Gateway Fabric with NGINX Plus has the following difference:
- An admin can connect to the NGINX Plus API using port 8765. NGINX only allows connections from localhost.
The normal process to update any changes to NGINX is to write the configuration files and reload NGINX. However, when using NGINX Plus, we can take advantage of the NGINX Plus API to limit the amount of reloads triggered when making changes to NGINX. Specifically, when the endpoints of an application in Kubernetes change (Such as scaling up or down), the NGINX Plus API is used to update the upstream servers in NGINX with the new endpoints without a reload. This reduces the potential for a disruption that could occur when reloading.
The nginx-gateway container exposes a readiness endpoint at /readyz. During startup, a readiness probe periodically checks this endpoint. The probe returns a 200 OK response once the control plane initializes successfully and is ready to begin configuring NGINX. At that point, the pod is marked as ready.