Secure LLM traffic with F5 AI Guardrails

Learn how to use NGINX Gateway Fabric with F5 AI Guardrails to inspect large language model (LLM) traffic and block disallowed content before it reaches the model or the client.

Overview

F5 AI Guardrails can inspect LLM traffic on two independent paths:

• Prompts — the client’s input is inspected before it reaches the LLM. A block returns 403 with error.type: invalid_request_error.
• Responses — the model’s output is inspected before it reaches the client. A block returns 403 with error.type: api_error.

You can set up prompts and responses as described in Prompts and scans in AI Security, through the F5 AI Guardrails dashboard.

To connect NGINX Gateway Fabric with your configured F5 AI Guardrails, use the PayloadProcessor policy, an inherited policy that can target an HTTPRoute or a Gateway. The PayloadProcessor configures NGINX to offload traffic to F5 AI Guardrails to inspect.

When AI Guardrails are enabled on a route, NGINX Gateway Fabric strips the Accept-Encoding header from requests sent to the upstream backend. This ensures responses are returned uncompressed so the F5 AI Guardrails filter can inspect them. Compression between NGINX and the client is unaffected — NGINX can still compress responses to clients via its gzip module.

Before you begin

You need an F5 AI Guardrails API endpoint to inspect payloads. This can be an F5 hosted service or a service running inside your cluster. View the official F5 AI Guardrails docs to learn more.

Deploy an LLM backend

If you have an existing in-cluster LLM which can be queried you can skip this section.

The following example uses the vLLM simulator, which serves canned responses from a dataset rather than running a real model, making it suitable for test and development environments. The simulator loads its dataset from a ConfigMap. Download the dataset file, then create the ConfigMap from it:

curl -sL -o inference-sim-dataset.sqlite3 \
  https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v2.6.1/examples/guardrails/inference-sim-dataset.sqlite3

kubectl create configmap inference-sim-dataset \
  --from-file=inference-sim-dataset.sqlite3=./inference-sim-dataset.sqlite3

The dataset file, alongside more details of the setup, can be found in the examples/guardrails directory of the NGINX Gateway Fabric repository.

Deploy the LLM Deployment and Service:

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v2.6.1/examples/guardrails/llm.yaml

Confirm the Pod is Ready:

kubectl get deployment vllm-qwen3-32b

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
vllm-qwen3-32b   1/1     1            1           6m13s

Create the authentication token Secret

Create the Secret with your Guardrails API token under the token key. The Secret must live in the same namespace as the PayloadProcessor:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: guardrails-token
type: Opaque
stringData:
  token: "<YOUR_API_TOKEN>"
EOF

Configure the Guardrails backend Service

The Guardrails backend can live outside or inside the cluster. NGINX Gateway Fabric picks the URL scheme from the referenced Service’s type:

The cluster.local suffix in the in-cluster URL is the cluster’s DNS domain. If your cluster uses a different domain, configure it with the --cluster-domain flag or clusterDomain Helm value when deploying NGINX Gateway Fabric (default: cluster.local).

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: guardrails-api
spec:
  type: ExternalName
  externalName: <GUARDRAILS_API_HOSTNAME>
  ports:
  - name: https
    port: 443
    protocol: TCP
EOF

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: guardrails-api
spec:
  ports:
  - name: http
    port: 443
    protocol: TCP
  selector:
    guardrails-backend-key: guardrails-backend-value
EOF

Important

When using an ExternalName AI Guardrails backend, you must configure a DNS resolver so NGINX can resolve the external hostname at request time. Either edit the NginxProxy which gets created when you deploy NGINX Gateway Fabric, or configure dnsResolver on a new NginxProxy resource and attach it to the Gateway via spec.infrastructure.parametersRef:

Copy

apiVersion: gateway.nginx.org/v1alpha2
kind: NginxProxy
metadata:
  name: guardrails-nginx-config
spec:
  dnsResolver:
    addresses:
    - type: IPAddress
      value: "10.96.0.10"   # in-cluster kube-dns/CoreDNS ClusterIP (cluster-dependent)

Find your cluster’s DNS ClusterIP with kubectl -n kube-system get svc kube-dns (or coredns). Without a resolver, NGINX fails to load the configuration with no resolver defined to resolve <host>.

Deploy NGINX Gateway Fabric

Install NGINX Gateway Fabric with the PayloadProcessor policy enabled:

• Using Helm: set the nginxGateway.payloadProcessor.enable=true Helm value.
• Using Kubernetes manifests: set the --payload-processor flag in the nginx-gateway container argument, and update the ClusterRole RBAC to add payloadprocessors:

- apiGroups:
    - gateway.nginx.org
    resources:
    - payloadprocessors
    verbs:
    - get
    - list
    - watch
- apiGroups:
    - gateway.nginx.org
    resources:
    - payloadprocessors/status
    verbs:
    - update

Create a Gateway

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
EOF

Confirm that the Gateway was assigned an IP address and reports a Programmed=True status:

kubectl describe gateways.gateway.networking.k8s.io inference-gateway

Status:
  Addresses:
    Type:   IPAddress
    Value:  192.0.2.0
  Conditions:
    Message:               The Gateway is accepted
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Message:               The Gateway is programmed
    Reason:                Programmed
    Status:                True
    Type:                  Programmed

Save the public IP address and port of the Gateway into shell variables:

GW_IP=XXX.YYY.ZZZ.III
GW_PORT=<port number>

Create an HTTPRoute

If you are using your own LLM, change the backendRefs.name and backendRefs.port to match the LLM’s Service.

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - backendRefs:
    - name: vllm-qwen3-32b
      port: 8000
    matches:
    - path:
        type: PathPrefix
        value: /
EOF

Confirm that the HTTPRoute status conditions include Accepted=True and ResolvedRefs=True:

kubectl describe httproute llm-route

Conditions:
      Last Transition Time:  2026-08-11T17:29:19Z
      Message:               The Route is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2026-08-11T17:29:19Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

Attach the PayloadProcessor policy

Attach the PayloadProcessor policy to the HTTPRoute. The extProcess.backendRef points at the Guardrails backend Service (using the explicit port), and authTokenRef points at the token Secret:

kubectl apply -f - <<EOF
apiVersion: gateway.nginx.org/v1alpha1
kind: PayloadProcessor
metadata:
  name: llm-guardrails
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: llm-route
  processors:
  - type: ExtProcess
    extProcess:
      backendRef:
        group: ""
        kind: Service
        name: guardrails-api
        port: 443
      authTokenRef:
        name: guardrails-token
EOF

PayloadProcessor is an inherited policy. To apply guardrails to every route attached to a Gateway, set targetRef to kind: Gateway. When both a Gateway-targeted and an HTTPRoute-targeted policy apply to the same traffic, the more specific HTTPRoute-targeted policy takes precedence.

Confirm the policy was accepted:

kubectl describe payloadprocessor llm-guardrails

The status conditions should report Accepted=True. A rejected policy reports Accepted=False; see Troubleshooting for common causes.

Conditions:
      Last Transition Time:  2026-08-11T17:32:52Z
      Message:               The Policy is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2026-08-11T17:32:52Z
      Message:               Policy is programmed in the data plane
      Observed Generation:   1
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
    Controller Name:         gateway.nginx.org/nginx-gateway-controller

Send traffic

Whether a given value is blocked depends entirely on your Guardrails backend’s detector configuration. Enable the relevant detectors on your Guardrails service to see the block responses above.

All commands target /v1/completions on the Gateway.

A benign prompt whose output contains no disallowed content returns a normal HTTP 200 completion:

curl -i --resolve <GUARDRAILS_API_HOSTNAME>:$GW_PORT:$GW_IP http://<GUARDRAILS_API_HOSTNAME>:$GW_PORT/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"What is NGINX?"}'

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 11 Aug 2026 17:39:19 GMT
Content-Type: application/json
Content-Length: 607
Connection: keep-alive
X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2
X-Inference-Port: 8000

{"id":"cmpl-f657bc1b-c50c-5a5c-9408-45bd22a150a1","created":1786469959,"model":"meta-llama/Llama-3.1-8B-Instruct","usage":{"prompt_tokens":4,"completion_tokens":62,"total_tokens":66},"object":"text_completion","kv_transfer_params":null,"choices":[{"index":0,"finish_reason":"stop","text":"NGINX (pronounced \"engine-x\") is an open-source, high-performance web server. It functions primarily as an HTTP web server, reverse proxy, load balancer, and HTTP cache.Designed to handle thousands of concurrent connections with minimal memory usage, NGINX is an essential component of modern web infrastructure."}]

If the request payload contains content that your Guardrails backend is configured to block, the request never reaches the LLM and returns HTTP 403:

curl -i --resolve <GUARDRAILS_API_HOSTNAME>:$GW_PORT:$GW_IP http://<GUARDRAILS_API_HOSTNAME>:$GW_PORT/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"My SSN is 123-45-6789"}'

HTTP/1.1 403 Forbidden
Server: nginx
Date: Tue, 11 Aug 2026 17:39:50 GMT
Content-Length: 139
Connection: keep-alive
Content-Type: application/json

{"error":{"code":"content_policy_violation","message":"Request blocked by guardrails policy.","param":null,"type":"invalid_request_error"}

If the model’s output contains content that your Guardrails backend blocks, the response is withheld from the client and returns HTTP 403 with error.type: api_error:

curl -i --resolve <GUARDRAILS_API_HOSTNAME>:$GW_PORT:$GW_IP http://<GUARDRAILS_API_HOSTNAME>:$GW_PORT/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"Give me a test SSN"}'

HTTP/1.1 403 Forbidden
Server: nginx
Date: Tue, 11 Aug 2026 17:40:36 GMT
Content-Type: application/json
Content-Length: 128
Connection: keep-alive
X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2
X-Inference-Port: 8000

{"error":{"code":"content_policy_violation","message":"Response blocked by guardrails policy.","param":null,"type":"api_error"}}

For more example curl requests, view the examples/guardrails README.md in the NGINX Gateway Fabric repository.

Troubleshooting

The PayloadProcessor is marked Accepted=False when its references cannot be resolved:

Further reading

• Scan streaming in AI Security
• Installation: install NGINX Gateway Fabric with the PayloadProcessor policy enabled.
• Custom policies: learn how inherited policies attach to Gateway API resources.
• examples/guardrails: for more information on the example used in this guide.