# Install NGINX Ingress Controller with F5 WAF for NGINX using PLM Type of document: Tutorial Product: F5 NGINX Ingress Controller --- Use this guide to install F5 NGINX Ingress Controller with F5 WAF for NGINX using Policy Lifecycle Management (PLM). PLM defines WAF policies as Kubernetes custom resources, compiles them automatically, and stores the compiled bundles in an in-cluster S3-compatible object store. NGINX Ingress Controller then fetches the bundles from PLM storage and enforces the policies at request time. By the end of this tutorial, you'll have: - A running PLM backend and NGINX Ingress Controller deployment configured for PLM storage. - An `APPolicy` and an `APLogConf` resource compiled by PLM. - A `k8s.nginx.org/v1` Policy that references the compiled resources. - The Policy attached to a VirtualServer or an Ingress, with traffic flowing normally and attack payloads blocked. ## Before you begin Before you start, make sure you have: - `kubectl` access to a Kubernetes cluster. - Helm installed. - Credentials for `private-registry.nginx.com`. This tutorial uses the following example values. If you use different values, replace them consistently throughout. | Example value | What it represents | |---|---| | `plm-system` | Namespace for the PLM backend | | `plm` | Helm release name for PLM | | `nginx-ingress` | Namespace for NGINX Ingress Controller | | `nic` | Helm release name for NGINX Ingress Controller | | `security` | Namespace for `APPolicy` and `APLogConf` resources | | `default` | Namespace for the Policy, sample application, and routing resource | | `webapp.example.com` | Example hostname for VirtualServer routing | | `cafe.example.com` | Example hostname for Ingress routing | ## Deploy PLM infrastructure Install the PLM backend before installing NGINX Ingress Controller. The PLM backend provisions the App Protect v1 CRDs, the Policy Controller, the compiler service, and the SeaweedFS storage backend in the `plm-system` namespace. The Policy Lifecycle Manager (PLM) backend runs as a Kubernetes operator. It watches WAF custom resources and compiles WAF policies into bundles. The Policy Controller delegates compilation to a separate compiler service over gRPC. The resulting bundles are stored in an embedded SeaweedFS S3-compatible object store. F5 WAF for NGINX is installed using a separate Helm chart from your NGINX data plane. The steps in this section install only the F5 WAF for NGINX PLM components and do not affect your existing NGINX installation. ### Create the registry pull secret Create a namespace for the PLM components, store your JWT in a Kubernetes Secret, then create the registry pull secret for the private F5 container registry. 1. Create the namespace and store your JWT. The following commands assume your JWT file is named `license.jwt`: ```shell kubectl create namespace plm-system kubectl create secret generic jwt-reg-secret \ --namespace plm-system \ --from-file=license.jwt ``` 2. Retrieve the JWT from the Secret and create the registry pull secret: ```shell JWT=$(kubectl get secret jwt-reg-secret \ --namespace plm-system \ -o jsonpath='{.data.license\.jwt}' | base64 -d) kubectl create secret docker-registry regcred \ --namespace plm-system \ --docker-server=private-registry.nginx.com \ --docker-username="$JWT" \ --docker-password=none \ --dry-run=client --output yaml | kubectl apply -f - ``` ### Install the Policy Controller Create a values file for the Helm installation. The `securityUpdatesRepo.cert` and `securityUpdatesRepo.key` fields are optional. They are only required if your signature repository needs certificate-based authentication. The Policy Controller starts successfully with these fields left empty. If your signature repository requires them, replace `` and `` with the base64-encoded contents of your `nginx-repo.crt` and `nginx-repo.key` files. To encode them, run: ```shell base64 --wrap=0 < nginx-repo.crt base64 --wrap=0 < nginx-repo.key ``` Create `/tmp/plm-values.yaml`: ```yaml imagePullSecrets: - name: regcred securityUpdatesRepo: cert: "" # optional: only needed for authenticated signature repository access key: "" # optional: only needed for authenticated signature repository access policyController: image: tag: "" compiler: image: tag: "" seaweedfsOperatorConfig: seaweedfs: image: tag: "" seaweedfs-operator: image: tag: "" pullSecrets: regcred ``` #### Enable TLS for PLM storage (optional) By default, communication between PLM components and the SeaweedFS object store uses unencrypted HTTP. To enable TLS, add a `certificates` block to `/tmp/plm-values.yaml`: ```yaml seaweedfsOperatorConfig: seaweedfs: certificates: enabled: true ``` **Note:** The PLM chart does not generate certificates. You must create the five Secrets listed in the commands below before running `helm upgrade --install`. If any Secret is missing, the SeaweedFS pods will fail to mount their certificates and will not start. **Note:** If you're enabling TLS on an existing installation, the storage backend restarts and objects written before the switch can become orphaned. See the [APPolicy shows `invalid` with `unexpected EOF` after enabling TLS](#troubleshoot-the-deployment) entry in the troubleshooting section. A fresh installation with TLS enabled from the start doesn't have this issue. Create the Secrets from your CA and certificate files before installing. The chart expects Secret names in the form `-f5-waf-seaweedfs-` — for the `plm` release name used in this tutorial, those are: ```shell kubectl create secret generic plm-f5-waf-seaweedfs-ca-cert \ --namespace plm-system \ --from-file=tls.crt= \ --from-file=ca.crt= kubectl create secret tls plm-f5-waf-seaweedfs-master-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-volume-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-filer-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-client-cert \ --namespace plm-system \ --cert= \ --key= ``` The CA Secret requires both `tls.crt` and `ca.crt` keys, both pointing to the same CA certificate file. The PLM chart mounts the CA using `tls.crt` into the Policy Controller, compiler, and SeaweedFS pods. The data plane's S3 client reads `ca.crt` from the same Secret when verifying the storage endpoint. The four component Secrets use `kubectl create secret tls`, which produces `tls.crt` and `tls.key` — no `ca.crt` key is needed for them. Replace each `` placeholder with the path to the corresponding certificate and key file from your PKI. The CA must sign all component certificates. If you don't have an existing PKI, generate a CA and sign the five component certificates before proceeding. #### Install the chart Add the NGINX Helm repository and install the chart: ```shell helm repo add nginx-stable https://helm.nginx.com/stable helm repo update nginx-stable helm upgrade --install plm nginx-stable/f5-waf-policy-controller \ --version \ --namespace plm-system \ --values /tmp/plm-values.yaml ``` To see all available configuration options for the PLM chart, run: ```shell helm show values nginx-stable/f5-waf-policy-controller --version ``` ### Verify the deployment Wait for all PLM components to become ready. The Policy Controller's init container waits for both the compiler service and the SeaweedFS S3 endpoint to be available before it starts, so the controller pod will show `Init:0/1` until SeaweedFS is ready. Wait for the SeaweedFS storage backend: ```shell kubectl rollout status deployment/plm-seaweedfs-operator \ --namespace plm-system --timeout=120s ``` The SeaweedFS operator creates the SeaweedFS pods after it reconciles the SeaweedFS custom resource, so there is a window where the operator deployment is ready but no SeaweedFS pods exist yet. Poll until the pods appear and are ready: ```shell end=$((SECONDS + 300)) until kubectl wait pods \ --selector app.kubernetes.io/name=seaweedfs \ --for=condition=Ready \ --namespace plm-system \ --timeout=10s 2>/dev/null; do if [ $SECONDS -ge $end ]; then echo "Timed out waiting for SeaweedFS pods" exit 1 fi sleep 5 done ``` Wait for the Policy Controller: ```shell kubectl rollout status deployment/plm-f5-waf-policy-controller \ --namespace plm-system --timeout=180s ``` Confirm all pods are running: ```shell kubectl get pods --namespace plm-system ``` Example output: ```text NAME READY STATUS RESTARTS plm-f5-waf-compiler-service-xxxxx 1/1 Running 0 plm-f5-waf-policy-controller-xxxxx 1/1 Running 0 plm-seaweedfs-operator-xxxxx 1/1 Running 0 plm-f5-waf-seaweed-master-0 1/1 Running 0 plm-f5-waf-seaweed-filer-0 1/1 Running 0 plm-f5-waf-seaweed-volume-0 1/1 Running 0 plm-f5-waf-seaweed-volume-1 1/1 Running 0 plm-f5-waf-seaweed-volume-2 1/1 Running 0 ``` Confirm the CRDs are present: ```shell kubectl get crd | grep appprotect.f5.com ``` Expected output: ```text aplogconfs.appprotect.f5.com appolicies.appprotect.f5.com apsignatures.appprotect.f5.com apusersigs.appprotect.f5.com ``` All eight pods running and all four CRDs present confirms the PLM backend is ready. ### Update the CRDs **Note:** Skip this step on a fresh install — Helm installs the CRDs automatically. Only follow these steps when upgrading an existing PLM installation. When upgrading PLM, apply the CRDs manually before running `helm upgrade`: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/waf-policy-controller//manifests/1-deploy-crds.yaml ``` ### Troubleshoot the deployment These are the most common failures during PLM installation, roughly in order of likelihood. #### Pods stuck in `ImagePullBackOff` The JWT is wrong, expired, or contains a line break. Check the events log: ```shell kubectl get events --namespace plm-system --field-selector reason=Failed ``` Use the full JWT string as the registry username. Use the literal string `none` as the password. #### Policy Controller stuck in `Init:0/1` The `Init:0/1` state is expected during startup. The init container waits for the compiler service and the S3 endpoint before it starts. If the pod stays in `Init:0/1` for more than a few minutes, check that the SeaweedFS pods are `Running`: ```shell kubectl get pods --namespace plm-system --selector app.kubernetes.io/name=seaweedfs ``` The most common cause is PVCs stuck in `Pending` because the cluster has no default StorageClass. #### SeaweedFS pods `Pending` SeaweedFS pods stay `Pending` when the cluster has no default StorageClass or insufficient capacity. Check the PVCs and available storage classes: ```shell kubectl get pvc --namespace plm-system kubectl get storageclass ``` #### `APPolicy` shows `invalid` with `unexpected EOF` after enabling TLS Enabling TLS on an existing installation restarts the storage backend. Objects written before TLS was enabled can become orphaned. Check the filer log: ```shell kubectl logs --namespace plm-system plm-f5-waf-seaweed-filer-0 | grep "not found" ``` If the output contains `volume N not found`, orphaned objects exist. Delete the affected `APPolicy` resource and reapply it. The Policy Controller regenerates the bundle. #### Helm install fails on a ClusterRole If the error references `seaweed-editor-role` or `seaweed-viewer-role`, another PLM installation already exists in the cluster. Only one PLM installation is supported per cluster. Remove the existing release before installing. #### Check the Policy Controller logs Use the Policy Controller logs to diagnose any policy-related failure: ```shell kubectl logs --namespace plm-system deploy/plm-f5-waf-policy-controller -c policy-controller ``` **Note:** The `-c policy-controller` flag is required because the pod has more than one container. The containers are distroless, so `kubectl exec` isn't available for interactive debugging. Confirm the CRDs are installed: ```shell kubectl get crd | grep appprotect.f5.com ``` Expected output: ```text aplogconfs.appprotect.f5.com appolicies.appprotect.f5.com apsignatures.appprotect.f5.com apusersigs.appprotect.f5.com ``` ## Look up the PLM storage endpoint and credentials NGINX Ingress Controller connects to PLM's SeaweedFS filer over S3 to fetch compiled bundles. Before you install the controller, collect these four values from the PLM installation: 1. **PLM storage URL**: the SeaweedFS filer endpoint (HTTPS or HTTP). 2. **Credentials Secret**: the S3 credentials Secret. The access key ID is `admin` by default. The secret access key is in the `seaweedfs_admin_secret` field. 3. **CA Secret** (HTTPS only): verifies the SeaweedFS filer certificate. 4. **Client TLS Secret** (mutual TLS only): presented by NGINX Ingress Controller when it connects to the filer. List the Services PLM created and identify the filer: ```shell kubectl get service --namespace plm-system ``` Expected output includes an entry similar to: ```text NAME TYPE CLUSTER-IP PORT(S) plm-f5-waf-seaweed-filer ClusterIP 10.0.0.10 8333/TCP,9333/TCP,... ``` Assemble the URL from the service name, namespace, and port. Use `9333` for HTTPS and `8333` for HTTP: - HTTPS (mTLS): `https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333` - HTTP: `http://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:8333` List the Secrets PLM created: ```shell kubectl get secret --namespace plm-system ``` The default PLM install creates three Secrets that NGINX Ingress Controller references: - `plm-f5-waf-seaweedfs-auth`: SeaweedFS credentials. - `plm-f5-waf-seaweedfs-ca-cert`: CA certificate for the HTTPS filer. - `plm-f5-waf-seaweedfs-client-cert`: client TLS certificate for mTLS. Record the Secret references in `` form. You'll pass all four values to NGINX Ingress Controller using `--set controller.appprotect.plmStorage.*` flags. ## Install NGINX Ingress Controller with PLM storage Because PLM owns the `appprotect.f5.com/v1` CRDs, you must apply the controller's own CRDs before running `helm install`. The `deploy/crds.yaml` bundle contains every CRD the controller needs (`VirtualServer`, `VirtualServerRoute`, `Policy`, `TransportServer`, `GlobalConfiguration`, `DNSEndpoint`). The bundle deliberately excludes the App Protect CRDs, which PLM owns. Download your NGINX Ingress Controller subscription’s JSON Web Token and rename it to `nginx-repo.jwt`. ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` Create a namespace and image pull Secret for NGINX Ingress Controller: ```shell kubectl create namespace nginx-ingress kubectl create secret docker-registry regcred \ --namespace nginx-ingress \ --docker-server=private-registry.nginx.com \ --docker-username=$(cat nginx-repo.jwt) \ --docker-password=none kubectl create secret generic license-token \ --namespace nginx-ingress \ --from-file=license.jwt=nginx-repo.jwt \ --type=nginx.com/license ``` Add the NGINX Helm repository: ```shell helm repo add nginx-stable https://helm.nginx.com/stable helm repo update nginx-stable ``` Install NGINX Ingress Controller with PLM storage turned on. This example uses HTTPS with mutual TLS. For HTTP storage, set only `controller.appprotect.plmStorage.url` and `controller.appprotect.plmStorage.credentialsSecret`. ```shell helm install nic nginx-stable/nginx-ingress \ --namespace nginx-ingress \ --skip-crds \ --set controller.image.repository="private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress" \ --set controller.image.tag="" \ --set controller.nginxplus=true \ --set controller.appprotect.enable=true \ --set controller.appprotect.v5=true \ --set controller.appprotect.plmStorage.url="https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333" \ --set controller.appprotect.plmStorage.credentialsSecret="plm-system/plm-f5-waf-seaweedfs-auth" \ --set controller.appprotect.plmStorage.caSecret="plm-system/plm-f5-waf-seaweedfs-ca-cert" \ --set controller.appprotect.plmStorage.clientSSLSecret="plm-system/plm-f5-waf-seaweedfs-client-cert" \ --set controller.appprotect.plmStorage.insecureSkipVerify=false \ --set controller.serviceAccount.imagePullSecretName=regcred ``` Wait for the controller pod to become ready. Each pod runs three containers: `nginx-ingress`, `waf-enforcer`, and `waf-config-mgr`. ```shell kubectl wait --for=condition=Ready pods \ --namespace nginx-ingress \ --selector app.kubernetes.io/name=nginx-ingress \ --timeout=180s kubectl get pods --namespace nginx-ingress ``` Expected output: ```text NAME READY STATUS RESTARTS AGE nic-nginx-ingress-controller-xxxxx 3/3 Running 0 2m ``` Save the public IP address and HTTP port of the NGINX Ingress Controller LoadBalancer service to shell variables: ```shell IC_IP= IC_HTTP_PORT= ``` ## Define the WAF policy Create the `security` namespace to hold the `APPolicy` and `APLogConf` resources: ```shell kubectl create namespace security ``` Save the following as `waf-resources.yaml`. The file defines an `APPolicy` that blocks attack signatures and masks credit card numbers and social security numbers in responses. It also defines an `APLogConf` for security logging. ```yaml apiVersion: appprotect.f5.com/v1 kind: APPolicy metadata: name: dataguard-blocking namespace: security spec: policy: name: dataguard-blocking template: name: POLICY_TEMPLATE_NGINX_BASE applicationLanguage: utf-8 enforcementMode: blocking blocking-settings: violations: - name: VIOL_DATA_GUARD alarm: true block: true data-guard: enabled: true maskData: true creditCardNumbers: true usSocialSecurityNumbers: true --- apiVersion: appprotect.f5.com/v1 kind: APLogConf metadata: name: log-default namespace: security spec: content: format: default max_message_size: 64k max_request_size: any filter: request_type: all ``` Apply the file: ```shell kubectl apply -f waf-resources.yaml ``` Wait for both resources to reach `ready`: ```shell kubectl wait --for=jsonpath='{.status.bundle.state}'=ready \ appolicy/dataguard-blocking --namespace security --timeout=180s kubectl wait --for=jsonpath='{.status.bundle.state}'=ready \ aplogconf/log-default --namespace security --timeout=180s ``` Confirm the compiled bundle location: ```shell kubectl get appolicy dataguard-blocking --namespace security \ --output jsonpath='State: {.status.bundle.state}{"\n"}Location: {.status.bundle.location}{"\n"}' ``` ## Create the Policy Save the following as `waf-policy.yaml`. The `apPolicy` and `apLogConf` fields accept `[/]`. When you omit the namespace, NGINX Ingress Controller defaults to the Policy's own namespace. ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: waf-policy namespace: default spec: waf: enable: true apPolicy: "security/dataguard-blocking" securityLogs: - enable: true apLogConf: "security/log-default" logDest: "syslog:server=syslog-svc.default:514" ``` Apply the file: ```shell kubectl apply -f waf-policy.yaml ``` Wait for the Policy to become `Valid`: ```shell kubectl wait --for=jsonpath='{.status.state}'=Valid \ policy/waf-policy --namespace default --timeout=180s ``` If the Policy stays in `Warning` with a `BundleFetchFailed` reason, see [Troubleshooting](#troubleshooting). Choose how to attach the Policy: - [Attach to a VirtualServer](#attach-to-a-virtualserver) - [Attach to an Ingress](#attach-to-an-ingress) The Policy resource is the same for both. ## Attach to a VirtualServer Save the following as `webapp.yaml`. The file defines the sample application Deployment, its Service, and a VirtualServer that references the `waf-policy` Policy. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: webapp namespace: default spec: replicas: 1 selector: matchLabels: app: webapp template: metadata: labels: app: webapp spec: containers: - name: webapp image: nginxdemos/nginx-hello:plain-text ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: webapp-svc namespace: default spec: selector: app: webapp ports: - name: http port: 80 targetPort: 8080 --- apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: webapp namespace: default spec: host: webapp.example.com policies: - name: waf-policy upstreams: - name: webapp service: webapp-svc port: 80 routes: - path: / action: pass: webapp ``` Apply the file: ```shell kubectl apply -f webapp.yaml ``` Send a normal request to confirm the application responds: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP \ http://webapp.example.com:$IC_HTTP_PORT/ ``` Expected output: ```text Server address: 10.0.0.1:8080 Server name: webapp-xxxxx ``` Send a request that triggers the data guard violation: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP \ "http://webapp.example.com:$IC_HTTP_PORT/" ``` Expected output: ```text Request Rejected The requested URL was rejected. Please consult with your administrator. ... ``` ## Attach to an Ingress Save the following as `cafe.yaml`. The file defines the sample cafe application (`coffee` and `tea` Deployments and Services) and an Ingress. The `nginx.com/policies` annotation attaches the `waf-policy` Policy to every route on the Ingress. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: coffee namespace: default spec: replicas: 2 selector: matchLabels: app: coffee template: metadata: labels: app: coffee spec: containers: - name: coffee image: nginxdemos/hello:plain-text ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: coffee-svc namespace: default spec: selector: app: coffee ports: - port: 80 targetPort: 80 name: http --- apiVersion: apps/v1 kind: Deployment metadata: name: tea namespace: default spec: replicas: 2 selector: matchLabels: app: tea template: metadata: labels: app: tea spec: containers: - name: tea image: nginxdemos/hello:plain-text ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: tea-svc namespace: default spec: selector: app: tea ports: - port: 80 targetPort: 80 name: http --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress namespace: default annotations: nginx.com/policies: "waf-policy" spec: ingressClassName: nginx rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` Apply the file: ```shell kubectl apply -f cafe.yaml ``` Send a normal request: ```shell curl --resolve cafe.example.com:$IC_HTTP_PORT:$IC_IP \ http://cafe.example.com:$IC_HTTP_PORT/coffee ``` Send a request that triggers the data guard violation: ```shell curl --resolve cafe.example.com:$IC_HTTP_PORT:$IC_IP \ "http://cafe.example.com:$IC_HTTP_PORT/coffee/" ``` The response body is `Request Rejected`. Under PLM, the Ingress-only App Protect annotations (`appprotect.f5.com/app-protect-policy` and `appprotect.f5.com/app-protect-security-log`) aren't supported. Use the `k8s.nginx.org/v1` Policy resource and the `nginx.com/policies` annotation instead, as shown above. ## Verify bundles on disk Confirm the compiled bundles are present in the ingress controller pod: ```shell NIC_POD=$(kubectl get pods --namespace nginx-ingress \ --selector app.kubernetes.io/name=nginx-ingress \ --output jsonpath='{.items[0].metadata.name}') kubectl exec --namespace nginx-ingress $NIC_POD --container nginx-ingress -- \ ls -ltr /etc/app_protect/bundles/ ``` Expected output: ```text total 1860 -rw------- 1 nginx nginx 1654 Aug 12 10:06 fetched_default_waf-policy_log_0.tgz -rw------- 1 nginx nginx 1898698 Aug 12 13:32 fetched_default_waf-policy_policy.tgz ``` Check the Policy status: ```shell kubectl describe policy waf-policy ``` A `State: Valid` and `Reason: AddedOrUpdated` status confirms the bundles were fetched successfully. ## Troubleshooting - **Policy status is `Warning` with reason `BundleFetchFailed`.** Run `kubectl describe appolicy --namespace security` and confirm `status.bundle.state` is `ready`. If PLM hasn't compiled the resource yet, the Policy fetch can't proceed. - **NGINX Ingress Controller reports the referenced namespace isn't watched.** If `controller.watchNamespace` is set, include the namespace that holds the `APPolicy` and `APLogConf` resources. If `controller.watchSecretNamespace` is set, include the PLM namespace so the controller can observe storage Secret rotation.