Secure LLM traffic with F5 AI Guardrails

Learn how to use NGINX Gateway Fabric with F5 AI Guardrails to inspect large language model (LLM) traffic and block disallowed content before it reaches the model or the client.

Overview

F5 AI Guardrails can inspect LLM traffic on two independent paths:

• Prompts — the client’s input is inspected before it reaches the LLM. A block returns 403 with error.type: invalid_request_error.
• Responses — the model’s output is inspected before it reaches the client. A block returns 403 with error.type: api_error.
• Oversized prompts — a prompt too large to inspect in memory is blocked before it reaches the LLM. A block returns 403 with error.type: request_too_large (code: request_body_too_large).

You can set up prompts and responses as described in Prompts and scans in AI Security, through the F5 AI Guardrails dashboard.

To connect NGINX Gateway Fabric with your configured F5 AI Guardrails, use the PayloadProcessor policy, an inherited policy that can target an HTTPRoute or a Gateway. The PayloadProcessor configures NGINX to offload traffic to F5 AI Guardrails to inspect.

The PayloadProcessor CRD is inspired by the proposed resource from the Gateway API AI Gateway Working Group, which is subject to change and may be redefined in future releases. Consider this experimental while it is being defined by the Gateway API WG.

When AI Guardrails are enabled on a route, NGINX Gateway Fabric strips the Accept-Encoding header from requests sent to the upstream backend. This ensures responses are returned uncompressed so the F5 AI Guardrails filter can inspect them. Compression between NGINX and the client is unaffected. NGINX can still compress responses to clients via its gzip module.

Before you begin

You need an F5 AI Guardrails API endpoint to inspect payloads. This can be an F5 hosted service or a service running inside your cluster. View the official F5 AI Guardrails docs to learn more.

Deploy an LLM backend

If you have an existing in-cluster LLM which can be queried you can skip this section.

The following example uses the vLLM simulator, which serves canned responses from a dataset rather than running a real model, making it suitable for test and development environments. The simulator loads its dataset from a ConfigMap. Download the dataset file, then create the ConfigMap from it:

curl -sL -o inference-sim-dataset.sqlite3 \
  https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v2.7.2/examples/guardrails/inference-sim-dataset.sqlite3

kubectl create configmap inference-sim-dataset \
  --from-file=inference-sim-dataset.sqlite3=./inference-sim-dataset.sqlite3

The dataset file, alongside more details of the setup, can be found in the examples/guardrails directory of the NGINX Gateway Fabric repository.

Deploy the LLM Deployment and Service:

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v2.7.2/examples/guardrails/llm.yaml

Confirm the Pod is Ready:

kubectl get deployment vllm-qwen3-32b

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
vllm-qwen3-32b   1/1     1            1           6m13s

Create the authentication token Secret

Create the Secret with your Guardrails API token under the token key. The Secret must live in the same namespace as the PayloadProcessor:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: guardrails-token
type: Opaque
stringData:
  token: "<YOUR_API_TOKEN>"
EOF

Configure the Guardrails backend Service

The Guardrails backend can live outside or inside the cluster. NGINX Gateway Fabric picks the URL scheme from the referenced Service’s type and from any BackendTLSPolicy that targets the Service.

The cluster.local suffix in the in-cluster URL is the cluster’s DNS domain. If your cluster uses a different domain, configure it with the --cluster-domain flag or clusterDomain Helm value when deploying NGINX Gateway Fabric (default: cluster.local).

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: guardrails-api
spec:
  type: ExternalName
  externalName: <GUARDRAILS_API_HOSTNAME>
  ports:
  - name: https
    port: 443
    protocol: TCP
EOF

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: guardrails-api
spec:
  ports:
  - name: http
    port: 443
    protocol: TCP
  selector:
    guardrails-backend-key: guardrails-backend-value
EOF

Important

When using an ExternalName AI Guardrails backend, you must configure a DNS resolver so NGINX can resolve the external hostname at request time. Either edit the NginxProxy which gets created when you deploy NGINX Gateway Fabric, or configure dnsResolver on a new NginxProxy resource and attach it to the Gateway via spec.infrastructure.parametersRef:

Copy

apiVersion: gateway.nginx.org/v1alpha2
kind: NginxProxy
metadata:
  name: guardrails-nginx-config
spec:
  dnsResolver:
    addresses:
    - type: IPAddress
      value: "10.96.0.10"   # in-cluster kube-dns/CoreDNS ClusterIP (cluster-dependent)

Find your cluster’s DNS ClusterIP with kubectl -n kube-system get svc kube-dns (or coredns). Without a resolver, NGINX fails to load the configuration with no resolver defined to resolve <host>.

Secure an in-cluster backend with TLS

Encrypt and verify traffic to an in-cluster Guardrails backend, including a backend whose serving certificate is signed by a custom or self-signed certificate authority (CA) with no public root.

Without a BackendTLSPolicy, an in-cluster backend is reached over plaintext http, as the Resolved URL table above shows. The payloads sent for inspection can carry sensitive content, such as the SSN in the examples below, so a BackendTLSPolicy encrypts and verifies that path instead of leaving it unencrypted across the cluster network.

A BackendTLSPolicy is a Gateway API policy that tells NGINX Gateway Fabric to verify a backend Service’s TLS certificate. Attach one to the Guardrails Service, and NGINX Gateway Fabric switches that backend from http to https.

Before you continue, make sure you have:

• A PEM-encoded CA certificate (ca.crt) that signed the backend’s serving certificate.
• A backend that serves TLS on a known port.

Create a ConfigMap from your CA certificate, where ./ca.crt is your own CA file. The ConfigMap must live in the backend Service’s namespace, which these examples set to default:

kubectl create configmap guardrails-ca --from-file=ca.crt=./ca.crt -n default

This Service and BackendTLSPolicy replace the plaintext in-cluster Service from the In-cluster tab, reusing the name guardrails-api on the TLS port. If you already applied that plaintext Service, reapply it with the manifest below.

Create the TLS Service and a BackendTLSPolicy that targets it. Set validation.hostname to the Service FQDN. NGINX Gateway Fabric uses this value for SNI (Server Name Indication), the Host header, and certificate hostname verification:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: guardrails-api
spec:
  selector:
    guardrails-backend-key: guardrails-backend-value
  ports:
  - name: https
    port: 8443
    targetPort: 8443
    protocol: TCP
---
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: guardrails-api-tls
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: guardrails-api
  validation:
    caCertificateRefs:
    - group: ""
      kind: ConfigMap
      name: guardrails-ca
    hostname: guardrails-api.default.svc.cluster.local
EOF

These examples use the default namespace, like the rest of this guide. The validation.hostname must match the Service FQDN, and the PayloadProcessor's backendRef must point to this Service in the same namespace. If your backend runs in another namespace, create the ConfigMap there and set validation.hostname to that namespace’s FQDN.

Confirm the policy was accepted:

kubectl describe backendtlspolicies.gateway.networking.k8s.io guardrails-api-tls

The status conditions should report Accepted=True and ResolvedRefs=True. NGINX Gateway Fabric switches the backend to https only after the policy is accepted.

Conditions:
      Last Transition Time:  2026-08-11T17:34:10Z
      Message:               All CACertificateRefs are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2026-08-11T17:34:10Z
      Message:               The Policy is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

Set the PayloadProcessor backendRef.port to the TLS Service port, for example 8443, so the URL resolves to https://guardrails-api.default.svc.cluster.local:8443.

Keep these rules in mind when securing the in-cluster path:

• NGINX Gateway Fabric picks the scheme per Gateway. The same PayloadProcessor can resolve to https on one Gateway and plaintext http on another, depending on whether a BackendTLSPolicy is effective for that Gateway.
• A BackendTLSPolicy that targets an ExternalName Guardrails Service is ignored. An ExternalName backend is always verified against the system trust store.
• The in-cluster HTTPS path needs no DNS resolver, unlike the ExternalName path.
• To revert an in-cluster backend to plaintext http, remove the BackendTLSPolicy that targets its Service.

Important

NGINX Gateway Fabric fails closed when a BackendTLSPolicy targeting the backend Service is invalid, or when the CA ConfigMap it references is missing. The PayloadProcessor is not programmed for that backend, and NGINX Gateway Fabric does not fall back to plaintext.

Deploy NGINX Gateway Fabric

Install NGINX Gateway Fabric with the PayloadProcessor policy enabled:

• Using Helm: set the nginxGateway.payloadProcessor.enable=true Helm value.
• Using Kubernetes manifests: set the --payload-processor flag in the nginx-gateway container argument, and update the ClusterRole RBAC to add payloadprocessors:

- apiGroups:
    - gateway.nginx.org
    resources:
    - payloadprocessors
    verbs:
    - get
    - list
    - watch
- apiGroups:
    - gateway.nginx.org
    resources:
    - payloadprocessors/status
    verbs:
    - update

Create a Gateway

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
EOF

Confirm that the Gateway was assigned an IP address and reports a Programmed=True status:

kubectl describe gateways.gateway.networking.k8s.io inference-gateway

Status:
  Addresses:
    Type:   IPAddress
    Value:  192.0.2.0
  Conditions:
    Message:               The Gateway is accepted
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Message:               The Gateway is programmed
    Reason:                Programmed
    Status:                True
    Type:                  Programmed

Save the public IP address and port of the Gateway into shell variables:

GW_IP=XXX.YYY.ZZZ.III
GW_PORT=<port number>

Create an HTTPRoute

If you are using your own LLM, change the backendRefs.name and backendRefs.port to match the LLM’s Service.

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - backendRefs:
    - name: vllm-qwen3-32b
      port: 8000
    matches:
    - path:
        type: PathPrefix
        value: /
EOF

Confirm that the HTTPRoute status conditions include Accepted=True and ResolvedRefs=True:

kubectl describe httproute llm-route

Conditions:
      Last Transition Time:  2026-08-11T17:29:19Z
      Message:               The Route is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2026-08-11T17:29:19Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

Attach the PayloadProcessor policy

Attach the PayloadProcessor policy to the HTTPRoute. The extProcess.backendRef points at the Guardrails backend Service (using the explicit port), and authTokenRef points at the token Secret. For an in-cluster HTTPS backend, set backendRef.port to the TLS Service port instead of 443 (see Secure an in-cluster backend with TLS):

kubectl apply -f - <<EOF
apiVersion: gateway.nginx.org/v1alpha1
kind: PayloadProcessor
metadata:
  name: llm-guardrails
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: llm-route
  processors:
  - type: ExtProcess
    extProcess:
      backendRef:
        group: ""
        kind: Service
        name: guardrails-api
        port: 443  # in-cluster HTTPS: use the TLS Service port, for example 8443
      authTokenRef:
        name: guardrails-token
EOF

PayloadProcessor is an inherited policy. To apply guardrails to every route attached to a Gateway, set targetRef to kind: Gateway. When both a Gateway-targeted and an HTTPRoute-targeted policy apply to the same traffic, the more specific HTTPRoute-targeted policy takes precedence.

Confirm the policy was accepted:

kubectl describe payloadprocessor llm-guardrails

The status conditions should report Accepted=True. A rejected policy reports Accepted=False; see Troubleshooting for common causes.

Conditions:
      Last Transition Time:  2026-08-11T17:32:52Z
      Message:               The Policy is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2026-08-11T17:32:52Z
      Message:               Policy is programmed in the data plane
      Observed Generation:   1
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

Send traffic

Whether a given value is blocked depends entirely on your Guardrails backend’s detector configuration. Enable the relevant detectors on your Guardrails service to see the block responses above.

All commands target /v1/completions on the Gateway.

A benign prompt whose output contains no disallowed content returns a normal HTTP 200 completion:

curl -i --resolve <GUARDRAILS_API_HOSTNAME>:$GW_PORT:$GW_IP http://<GUARDRAILS_API_HOSTNAME>:$GW_PORT/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"What is NGINX?"}'

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 11 Aug 2026 17:39:19 GMT
Content-Type: application/json
Content-Length: 607
Connection: keep-alive
X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2
X-Inference-Port: 8000

{"id":"cmpl-f657bc1b-c50c-5a5c-9408-45bd22a150a1","created":1786469959,"model":"meta-llama/Llama-3.1-8B-Instruct","usage":{"prompt_tokens":4,"completion_tokens":62,"total_tokens":66},"object":"text_completion","kv_transfer_params":null,"choices":[{"index":0,"finish_reason":"stop","text":"NGINX (pronounced \"engine-x\") is an open-source, high-performance web server. It functions primarily as an HTTP web server, reverse proxy, load balancer, and HTTP cache.Designed to handle thousands of concurrent connections with minimal memory usage, NGINX is an essential component of modern web infrastructure."}]

If the request payload contains content that your Guardrails backend is configured to block, the request never reaches the LLM and returns HTTP 403:

curl -i --resolve <GUARDRAILS_API_HOSTNAME>:$GW_PORT:$GW_IP http://<GUARDRAILS_API_HOSTNAME>:$GW_PORT/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"My SSN is 123-45-6789"}'

HTTP/1.1 403 Forbidden
Server: nginx
Date: Tue, 11 Aug 2026 17:39:50 GMT
Content-Length: 139
Connection: keep-alive
Content-Type: application/json

{"error":{"code":"content_policy_violation","message":"Request blocked by guardrails policy.","param":null,"type":"invalid_request_error"}

If the model’s output contains content that your Guardrails backend blocks, the response is withheld from the client and returns HTTP 403 with error.type: api_error:

curl -i --resolve <GUARDRAILS_API_HOSTNAME>:$GW_PORT:$GW_IP http://<GUARDRAILS_API_HOSTNAME>:$GW_PORT/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"Give me a test SSN"}'

HTTP/1.1 403 Forbidden
Server: nginx
Date: Tue, 11 Aug 2026 17:40:36 GMT
Content-Type: application/json
Content-Length: 128
Connection: keep-alive
X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2
X-Inference-Port: 8000

{"error":{"code":"content_policy_violation","message":"Response blocked by guardrails policy.","param":null,"type":"api_error"}}

The ai-guardrails module inspects only the request body held in memory. When a prompt is larger than client_body_buffer_size, NGINX writes it to a temporary file that the module can’t inspect. NGINX Gateway Fabric then fails closed with 403, error.type: request_too_large, and code: request_body_too_large. To let large prompts stay in memory, raise client_body_buffer_size through a ClientSettingsPolicy (field spec.body.bufferSize). A larger buffer increases the memory NGINX uses per request. See Client settings for the field and its default.

For more example curl requests, view the examples/guardrails README.md in the NGINX Gateway Fabric repository.

Troubleshooting

The PayloadProcessor is marked Accepted=False when its references cannot be resolved:

Further reading

• Scan streaming in AI Security
• Installation: install NGINX Gateway Fabric with the PayloadProcessor policy enabled.
• Custom policies: learn how inherited policies attach to Gateway API resources.
• Client settings: configure client_body_buffer_size and other client connection settings.
• Gateway API BackendTLSPolicy: reference for the BackendTLSPolicy resource.
• Secure backend traffic with mutual TLS: encrypt traffic between the Gateway and a backend using BackendTLSPolicy and mutual TLS.
• examples/guardrails: for more information on the example used in this guide.