Gateway API Inference Extension

Learn how to use NGINX Gateway Fabric with the Gateway API Inference Extension to optimize traffic routing to self-hosted Generative AI models on Kubernetes.

Overview

The Gateway API Inference Extension is an official Kubernetes project that aims to provide optimized load-balancing for self-hosted generative AI models on Kubernetes. The project’s goal is to improve and standardize routing to inference workloads across the ecosystem.

Coupled with the provided llm-d Router, NGINX Gateway Fabric becomes an Inference Gateway. An Inference Gateway adds AI-specific traffic management features, such as model-aware routing, serving priority for models, and model rollouts.

Set up

Install the Gateway API Inference Extension CRDs:

kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/inference-extension/?ref=v2.8.0" | kubectl apply -f -

To enable the Gateway API Inference Extension, install NGINX Gateway Fabric with these modifications:

• Using Helm: set the nginxGateway.gwAPIInferenceExtension.enable=true Helm value.
• Using Kubernetes manifests: set the --gateway-api-inference-extension flag in the nginx-gateway container argument, update the ClusterRole RBAC to add the inferencepools:

- apiGroups:
    - inference.networking.k8s.io
    resources:
    - inferencepools
    verbs:
    - get
    - list
    - watch
- apiGroups:
    - inference.networking.k8s.io
    resources:
    - inferencepools/status
    verbs:
    - update

See this example manifest for clarification.

Important

To verify the llm-d router certificate, attach a BackendTLSPolicy to the llm-d router Service. If no BackendTLSPolicy targets the llm-d router Service, NGINX Gateway Fabric connects over TLS but skips certificate verification. For the steps, see Configure TLS verification for the llm-d router.

You can also change how NGINX Gateway Fabric connects to the llm-d router:

• To skip certificate verification even when a BackendTLSPolicy exists, set the nginxGateway.gwAPIInferenceExtension.endpointPicker.skipVerify=true Helm value or the --endpoint-picker-tls-skip-verify=true flag.
• To turn off TLS, set the nginxGateway.gwAPIInferenceExtension.endpointPicker.disableTLS=true Helm value or the --endpoint-picker-disable-tls flag. Use this option only for development, testing, or a service mesh that encrypts the traffic.

Set up cert-manager

This guide uses cert-manager to issue the llm-d router certificate from a local certificate authority (CA). If your cluster already has cert-manager and the local-ca-issuer ClusterIssuer, go to Create a TLS certificate for the llm-d router.

Install cert-manager onto the cluster using Helm with Gateway API features enabled.

• Add the Helm repository.

  Copy

  helm repo add jetstack https://charts.jetstack.io
  helm repo update

• Install cert-manager, and enable the GatewayAPI feature gate:

  Copy

  helm install \
    cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \
    --set config.kind="ControllerConfiguration" \
    --set config.enableGatewayAPI=true \
    --set crds.enabled=true

Create a self-signed ClusterIssuer, a CA Certificate, and a CA-backed ClusterIssuer. cert-manager uses the resulting local-ca-issuer to sign certificates in any namespace:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: local-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: LocalCA
  secretName: local-ca-secret
  issuerRef:
    name: selfsigned-cluster-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: local-ca-issuer
spec:
  ca:
    secretName: local-ca-secret
EOF

Create a TLS certificate for the llm-d router

Create a cert-manager Certificate with Subject Alternative Names (SANs) that match the llm-d router Service. The epp-ca Secret that cert-manager creates holds the certificate, the private key, and the CA certificate:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: epp-cert
  namespace: default
spec:
  secretName: epp-ca
  issuerRef:
    name: local-ca-issuer
    kind: ClusterIssuer
  dnsNames:
  - vllm-qwen3-32b-epp.default.svc
  - vllm-qwen3-32b-epp.default.svc.cluster.local
  - vllm-qwen3-32b-epp
EOF

Confirm that cert-manager issued the certificate:

kubectl get certificate epp-cert

Deploy a sample model server

The vLLM simulator model server does not use GPUs and is ideal for test/development environments. To deploy the vLLM simulator, run the following command:

kubectl apply -f https://raw.githubusercontent.com/llm-d/llm-d-router/refs/tags/v1.0.0/config/manifests/vllm/sim-deployment.yaml

Deploy the InferencePool and llm-d router

The InferencePool is a Gateway API Inference Extension resource that represents a set of Inference-focused Pods. With InferencePool, you can configure a routing extension as well as inference-specific routing optimizations. For more information on this resource, refer to the Gateway API Inference Extension InferencePool documentation.

Install an InferencePool named vllm-qwen3-32b that selects from endpoints with label app: vllm-qwen3-32b and listening on port 8000. The Helm install command automatically installs the llm-d router and InferencePool.

NGINX queries the llm-d Router to find the pod endpoint that gets the traffic. The llm-d Router picks from the ready pods that the InferencePool selector field matches. For more information, see the README for the llm-d router’s Endpoint Picker.

Warning

The llm-d router is a third-party application written and provided by the llm-d project. NGINX Gateway Fabric connects to the llm-d router over TLS. To verify the llm-d router certificate, follow the steps in Configure TLS verification for the llm-d router. NGINX Gateway Fabric is not responsible for any threats or risks associated with using this third-party llm-d router application.

For all chart values, see the llm-d Router Helm charts.

helm install vllm-qwen3-32b  \
--set router.modelServers.matchLabels.app=vllm-qwen3-32b \
--version v0.11.0 \
--set-json 'router.epp.volumes=[{"name":"tls","secret":{"secretName":"epp-ca"}}]' \
--set-json 'router.epp.volumeMounts=[{"name":"tls","mountPath":"/etc/tls","readOnly":true}]' \
--set router.epp.flags.cert-path=/etc/tls \
oci://ghcr.io/llm-d/charts/llm-d-router-gateway

The volumes and volumeMounts values mount the epp-ca Secret in the llm-d router container. The cert-path flag points the llm-d router to the mounted certificate and key. Without the flag, the llm-d router uses a self-signed certificate, and certificate verification fails.

For test environments, lower the CPU and memory requests and limits to reduce resource use:

Copy

helm install vllm-qwen3-32b  \
--set router.modelServers.matchLabels.app=vllm-qwen3-32b \
--version v0.11.0 \
--set router.epp.resources.requests.cpu=100m \
--set router.epp.resources.requests.memory=512Mi \
--set router.epp.resources.limits.memory=2Gi \
--set-json 'router.epp.volumes=[{"name":"tls","secret":{"secretName":"epp-ca"}}]' \
--set-json 'router.epp.volumeMounts=[{"name":"tls","mountPath":"/etc/tls","readOnly":true}]' \
--set router.epp.flags.cert-path=/etc/tls \
oci://ghcr.io/llm-d/charts/llm-d-router-gateway

Confirm that the llm-d router was deployed and is running:

kubectl describe deployment vllm-qwen3-32b-epp

Configure TLS verification for the llm-d router

NGINX Gateway Fabric connects to the llm-d router over TLS. If no BackendTLSPolicy targets the llm-d router Service, NGINX Gateway Fabric skips certificate verification. To verify the certificate, attach a BackendTLSPolicy to the llm-d router Service. NGINX Gateway Fabric then checks the llm-d router certificate against the CA and hostname in the policy.

The llm-d router must serve a certificate that the referenced CA signed. The certificate must also include the hostname that you set in the policy. NGINX Gateway Fabric doesn’t manage the llm-d router, so set up server-side TLS in the llm-d router deployment.

Create a BackendTLSPolicy that targets the llm-d router Service. Your InferencePool names this Service in its endpointPickerRef field. The following example targets the vllm-qwen3-32b-epp Service and checks its certificate against the CA in the epp-ca Secret:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: epp-tls
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: vllm-qwen3-32b-epp
  validation:
    caCertificateRefs:
    - name: epp-ca
      group: ''
      kind: Secret
    hostname: vllm-qwen3-32b-epp.default.svc
EOF

Deploy an Inference Gateway

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
EOF

Confirm that the Gateway was assigned an IP address and reports a Programmed=True status:

kubectl describe gateways.gateway.networking.k8s.io inference-gateway

Status:
  Addresses:
    Type:   IPAddress
    Value:  10.96.36.219
  Conditions:
    Last Transition Time:  2026-01-09T05:40:37Z
    Message:               The Gateway is accepted
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Last Transition Time:  2026-01-09T05:40:37Z
    Message:               The Gateway is programmed
    Observed Generation:   1
    Reason:                Programmed
    Status:                True
    Type:                  Programmed

Save the public IP address and port(s) of the Gateway into shell variables:

GW_IP=XXX.YYY.ZZZ.III
GW_PORT=<PORT_NUMBER>

Replace <PORT_NUMBER> with the port number you use to access your Gateway.

Deploy an HTTPRoute

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: vllm-qwen3-32b
    matches:
    - path:
        type: PathPrefix
        value: /
EOF

Confirm that the HTTPRoute status conditions include Accepted=True and ResolvedRefs=True:

kubectl describe httproute llm-route

Try it out

Send traffic to the Gateway:

curl -i $GW_IP:$GW_PORT/v1/completions -H 'Content-Type: application/json' -d '{
"model": "food-review-1",
"prompt": "Write as if you were a critic: San Francisco",
"max_tokens": 100,
"temperature": 0
}'

Cleanup

Uninstall the InferencePool and model server resources:

helm uninstall vllm-qwen3-32b
kubectl delete -f https://raw.githubusercontent.com/llm-d/llm-d-router/refs/tags/v1.0.0/config/manifests/vllm/sim-deployment.yaml

Uninstall the Gateway API Inference Extension CRDs:

kubectl delete -k https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd --ignore-not-found

Uninstall Inference Gateway and HTTPRoute:

kubectl delete gateway inference-gateway
kubectl delete httproute llm-route

Uninstall NGINX Gateway Fabric:

helm uninstall ngf -n nginx-gateway

If needed, replace ngf with your chosen release name.

Remove namespace and NGINX Gateway Fabric CRDs:

kubectl delete ns nginx-gateway
kubectl delete -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v2.8.0/deploy/crds.yaml

Remove the BackendTLSPolicy and certificate:

kubectl delete backendtlspolicy epp-tls
kubectl delete certificate epp-cert
kubectl delete secret epp-ca --ignore-not-found

Remove the Gateway API CRDs:

Warning

This step will remove all corresponding custom resources in your entire cluster, across every namespace.

Ensure you don’t have any custom resources you need to keep, and confirm that there are no other Gateway API implementations active in your cluster.

To uninstall the Gateway API resources, run the following:

kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v2.8.0" | kubectl delete -f -

customresourcedefinition.apiextensions.k8s.io "backendtlspolicies.gateway.networking.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "gateways.gateway.networking.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "grpcroutes.gateway.networking.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "httproutes.gateway.networking.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "referencegrants.gateway.networking.k8s.io" deleted

If you installed the Gateway APIs from the experimental channel, run the following instead:

kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v2.8.0" | kubectl delete -f -

See also

• Gateway API Inference Extension Introduction: for introductory details to the project.
• Gateway API Inference Extension API Overview: for an API overview.
• Gateway API Inference Extension User Guides: for additional use cases and guides.
• llm-d: for information on the llm-d project.
• Securing backend traffic using mutual TLS: for more on BackendTLSPolicy and backend certificate validation.