# Upgrade NGINX Ingress Controller to F5 WAF for NGINX with PLM Type of document: How-to guide Product: F5 NGINX Ingress Controller --- Use this guide to upgrade an existing F5 NGINX Ingress Controller + F5 WAF for NGINX deployment from in-pod App Protect policy compilation to Policy Lifecycle Management (PLM). PLM compiles `APPolicy` and `APLogConf` resources in a dedicated controller and stores the resulting bundles in an in-cluster S3-compatible object store. NGINX Ingress Controller then fetches the compiled bundles instead of compiling them in the data plane. Under PLM, the fields you already write on the NGINX Ingress Controller `Policy` resource (`waf.apPolicy` and `waf.securityLogs[].apLogConf`) continue to work unchanged. You don't need to rewrite the `k8s.nginx.org/v1` Policy manifest for a VirtualServer or Ingress during the upgrade. By the end of this guide, you'll have: - The PLM backend installed alongside your existing NGINX Ingress Controller deployment. - NGINX Ingress Controller upgraded to a PLM-capable release with PLM storage configured. - Existing `APPolicy` and `APLogConf` resources adopted by PLM and compiled into bundles. - WAF-protected traffic served by NGINX Ingress Controller, with bundles fetched from PLM storage instead of compiled in the data plane. ## Before you begin Before you start, make sure you have: - An existing NGINX Ingress Controller + F5 WAF for NGINX deployment running in your cluster. - Existing `APPolicy` and `APLogConf` resources served by the `appprotect.f5.com/v1beta1` CRDs shipped with your current NGINX Ingress Controller installation. - `kubectl` and Helm access to the cluster. - Credentials for `private-registry.nginx.com`. Record your current values before you begin: | Value | Where to find it | |---|---| | NGINX Ingress Controller release name | `helm list --namespace ` | | NGINX Ingress Controller chart version | `helm list --namespace ` | | Existing NGINX Ingress Controller values | `helm get values --namespace ` | | Existing `APPolicy` resources | `kubectl get appolicy --all-namespaces` | | Existing `APLogConf` resources | `kubectl get aplogconf --all-namespaces` | This guide uses `nic` as the NGINX Ingress Controller release name, `nginx-ingress` as the NGINX Ingress Controller namespace, `plm-system` as the PLM namespace, and `plm` as the PLM release name. Replace these with your own values consistently throughout. ## Deploy PLM infrastructure Install the PLM backend into the `plm-system` namespace. The install adds the `appprotect.f5.com/v1` versions to the existing `appprotect.f5.com` CRDs. Because the v1 versions are a superset of v1beta1, adding them doesn't affect NGINX Ingress Controller while it still watches v1beta1. The Policy Lifecycle Manager (PLM) backend runs as a Kubernetes operator. It watches WAF custom resources and compiles WAF policies into bundles. The Policy Controller delegates compilation to a separate compiler service over gRPC. The resulting bundles are stored in an embedded SeaweedFS S3-compatible object store. F5 WAF for NGINX is installed using a separate Helm chart from your NGINX data plane. The steps in this section install only the F5 WAF for NGINX PLM components and do not affect your existing NGINX installation. ### Create the registry pull secret Create a namespace for the PLM components, store your JWT in a Kubernetes Secret, then create the registry pull secret for the private F5 container registry. 1. Create the namespace and store your JWT. The following commands assume your JWT file is named `license.jwt`: ```shell kubectl create namespace plm-system kubectl create secret generic jwt-reg-secret \ --namespace plm-system \ --from-file=license.jwt ``` 2. Retrieve the JWT from the Secret and create the registry pull secret: ```shell JWT=$(kubectl get secret jwt-reg-secret \ --namespace plm-system \ -o jsonpath='{.data.license\.jwt}' | base64 -d) kubectl create secret docker-registry regcred \ --namespace plm-system \ --docker-server=private-registry.nginx.com \ --docker-username="$JWT" \ --docker-password=none \ --dry-run=client --output yaml | kubectl apply -f - ``` ### Install the Policy Controller Create a values file for the Helm installation. The `securityUpdatesRepo.cert` and `securityUpdatesRepo.key` fields are optional. They are only required if your signature repository needs certificate-based authentication. The Policy Controller starts successfully with these fields left empty. If your signature repository requires them, replace `` and `` with the base64-encoded contents of your `nginx-repo.crt` and `nginx-repo.key` files. To encode them, run: ```shell base64 --wrap=0 < nginx-repo.crt base64 --wrap=0 < nginx-repo.key ``` Create `/tmp/plm-values.yaml`: ```yaml imagePullSecrets: - name: regcred securityUpdatesRepo: cert: "" # optional: only needed for authenticated signature repository access key: "" # optional: only needed for authenticated signature repository access policyController: image: tag: "" compiler: image: tag: "" seaweedfsOperatorConfig: seaweedfs: image: tag: "" seaweedfs-operator: image: tag: "" pullSecrets: regcred ``` #### Enable TLS for PLM storage (optional) By default, communication between PLM components and the SeaweedFS object store uses unencrypted HTTP. To enable TLS, add a `certificates` block to `/tmp/plm-values.yaml`: ```yaml seaweedfsOperatorConfig: seaweedfs: certificates: enabled: true ``` **Note:** The PLM chart does not generate certificates. You must create the five Secrets listed in the commands below before running `helm upgrade --install`. If any Secret is missing, the SeaweedFS pods will fail to mount their certificates and will not start. **Note:** If you're enabling TLS on an existing installation, the storage backend restarts and objects written before the switch can become orphaned. See the [APPolicy shows `invalid` with `unexpected EOF` after enabling TLS](#troubleshoot-the-deployment) entry in the troubleshooting section. A fresh installation with TLS enabled from the start doesn't have this issue. Create the Secrets from your CA and certificate files before installing. The chart expects Secret names in the form `-f5-waf-seaweedfs-` — for the `plm` release name used in this tutorial, those are: ```shell kubectl create secret generic plm-f5-waf-seaweedfs-ca-cert \ --namespace plm-system \ --from-file=tls.crt= \ --from-file=ca.crt= kubectl create secret tls plm-f5-waf-seaweedfs-master-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-volume-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-filer-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-client-cert \ --namespace plm-system \ --cert= \ --key= ``` The CA Secret requires both `tls.crt` and `ca.crt` keys, both pointing to the same CA certificate file. The PLM chart mounts the CA using `tls.crt` into the Policy Controller, compiler, and SeaweedFS pods. The data plane's S3 client reads `ca.crt` from the same Secret when verifying the storage endpoint. The four component Secrets use `kubectl create secret tls`, which produces `tls.crt` and `tls.key` — no `ca.crt` key is needed for them. Replace each `` placeholder with the path to the corresponding certificate and key file from your PKI. The CA must sign all component certificates. If you don't have an existing PKI, generate a CA and sign the five component certificates before proceeding. #### Install the chart Add the NGINX Helm repository and install the chart: ```shell helm repo add nginx-stable https://helm.nginx.com/stable helm repo update nginx-stable helm upgrade --install plm nginx-stable/f5-waf-policy-controller \ --version \ --namespace plm-system \ --values /tmp/plm-values.yaml ``` To see all available configuration options for the PLM chart, run: ```shell helm show values nginx-stable/f5-waf-policy-controller --version ``` ### Verify the deployment Wait for all PLM components to become ready. The Policy Controller's init container waits for both the compiler service and the SeaweedFS S3 endpoint to be available before it starts, so the controller pod will show `Init:0/1` until SeaweedFS is ready. Wait for the SeaweedFS storage backend: ```shell kubectl rollout status deployment/plm-seaweedfs-operator \ --namespace plm-system --timeout=120s ``` The SeaweedFS operator creates the SeaweedFS pods after it reconciles the SeaweedFS custom resource, so there is a window where the operator deployment is ready but no SeaweedFS pods exist yet. Poll until the pods appear and are ready: ```shell end=$((SECONDS + 300)) until kubectl wait pods \ --selector app.kubernetes.io/name=seaweedfs \ --for=condition=Ready \ --namespace plm-system \ --timeout=10s 2>/dev/null; do if [ $SECONDS -ge $end ]; then echo "Timed out waiting for SeaweedFS pods" exit 1 fi sleep 5 done ``` Wait for the Policy Controller: ```shell kubectl rollout status deployment/plm-f5-waf-policy-controller \ --namespace plm-system --timeout=180s ``` Confirm all pods are running: ```shell kubectl get pods --namespace plm-system ``` Example output: ```text NAME READY STATUS RESTARTS plm-f5-waf-compiler-service-xxxxx 1/1 Running 0 plm-f5-waf-policy-controller-xxxxx 1/1 Running 0 plm-seaweedfs-operator-xxxxx 1/1 Running 0 plm-f5-waf-seaweed-master-0 1/1 Running 0 plm-f5-waf-seaweed-filer-0 1/1 Running 0 plm-f5-waf-seaweed-volume-0 1/1 Running 0 plm-f5-waf-seaweed-volume-1 1/1 Running 0 plm-f5-waf-seaweed-volume-2 1/1 Running 0 ``` Confirm the CRDs are present: ```shell kubectl get crd | grep appprotect.f5.com ``` Expected output: ```text aplogconfs.appprotect.f5.com appolicies.appprotect.f5.com apsignatures.appprotect.f5.com apusersigs.appprotect.f5.com ``` All eight pods running and all four CRDs present confirms the PLM backend is ready. ### Update the CRDs **Note:** Skip this step on a fresh install — Helm installs the CRDs automatically. Only follow these steps when upgrading an existing PLM installation. When upgrading PLM, apply the CRDs manually before running `helm upgrade`: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/waf-policy-controller//manifests/1-deploy-crds.yaml ``` ### Troubleshoot the deployment These are the most common failures during PLM installation, roughly in order of likelihood. #### Pods stuck in `ImagePullBackOff` The JWT is wrong, expired, or contains a line break. Check the events log: ```shell kubectl get events --namespace plm-system --field-selector reason=Failed ``` Use the full JWT string as the registry username. Use the literal string `none` as the password. #### Policy Controller stuck in `Init:0/1` The `Init:0/1` state is expected during startup. The init container waits for the compiler service and the S3 endpoint before it starts. If the pod stays in `Init:0/1` for more than a few minutes, check that the SeaweedFS pods are `Running`: ```shell kubectl get pods --namespace plm-system --selector app.kubernetes.io/name=seaweedfs ``` The most common cause is PVCs stuck in `Pending` because the cluster has no default StorageClass. #### SeaweedFS pods `Pending` SeaweedFS pods stay `Pending` when the cluster has no default StorageClass or insufficient capacity. Check the PVCs and available storage classes: ```shell kubectl get pvc --namespace plm-system kubectl get storageclass ``` #### `APPolicy` shows `invalid` with `unexpected EOF` after enabling TLS Enabling TLS on an existing installation restarts the storage backend. Objects written before TLS was enabled can become orphaned. Check the filer log: ```shell kubectl logs --namespace plm-system plm-f5-waf-seaweed-filer-0 | grep "not found" ``` If the output contains `volume N not found`, orphaned objects exist. Delete the affected `APPolicy` resource and reapply it. The Policy Controller regenerates the bundle. #### Helm install fails on a ClusterRole If the error references `seaweed-editor-role` or `seaweed-viewer-role`, another PLM installation already exists in the cluster. Only one PLM installation is supported per cluster. Remove the existing release before installing. #### Check the Policy Controller logs Use the Policy Controller logs to diagnose any policy-related failure: ```shell kubectl logs --namespace plm-system deploy/plm-f5-waf-policy-controller -c policy-controller ``` **Note:** The `-c policy-controller` flag is required because the pod has more than one container. The containers are distroless, so `kubectl exec` isn't available for interactive debugging. After the install completes, PLM adopts every existing `APPolicy` and `APLogConf` resource in the cluster. It does this by adding the `appprotect.f5.com/finalizer` finalizer and compiling each resource against its current signature package. Your running NGINX Ingress Controller continues to compile the same resources in-pod. The two mechanisms operate independently until you upgrade NGINX Ingress Controller. Confirm PLM has compiled the existing resources: ```shell kubectl get appolicy --all-namespaces \ --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.bundle.state' kubectl get aplogconf --all-namespaces \ --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.bundle.state' ``` Every resource should report `STATE: ready`. If any resource isn't `ready`, inspect the PLM policy-controller logs, resolve the compilation errors, and then continue. ## Look up the PLM storage endpoint and credentials NGINX Ingress Controller connects to PLM's SeaweedFS filer over S3 to fetch compiled bundles. Before you run `helm upgrade`, collect these four values from the PLM installation: 1. **PLM storage URL**: the SeaweedFS filer endpoint (HTTPS or HTTP). 2. **Credentials Secret**: the S3 credentials Secret. The access key ID is `admin` by default. The secret access key is in the `seaweedfs_admin_secret` field. 3. **CA Secret** (HTTPS only): verifies the SeaweedFS filer certificate. 4. **Client TLS Secret** (mutual TLS only): presented by NGINX Ingress Controller when it connects to the filer. List the Services PLM created and identify the filer: ```shell kubectl get service --namespace plm-system ``` Expected output includes an entry similar to: ```text NAME TYPE CLUSTER-IP PORT(S) plm-f5-waf-seaweed-filer ClusterIP 10.0.0.10 8333/TCP,9333/TCP,... ``` Assemble the URL from the service name, namespace, and port. Use `9333` for HTTPS and `8333` for HTTP: - HTTPS (mTLS): `https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333` - HTTP: `http://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:8333` List the Secrets PLM created: ```shell kubectl get secret --namespace plm-system ``` The default PLM install creates three Secrets that NGINX Ingress Controller references: - `plm-f5-waf-seaweedfs-auth`: SeaweedFS credentials. - `plm-f5-waf-seaweedfs-ca-cert`: CA certificate for the HTTPS filer. - `plm-f5-waf-seaweedfs-client-cert`: client TLS certificate for mTLS. Record the Secret references in `/` form. When you run `helm upgrade`, pass all four values to NGINX Ingress Controller using `--set controller.appprotect.plmStorage.*` flags. ## Apply the NGINX Ingress Controller CRDs Before you run `helm upgrade`, apply the NGINX Ingress Controller CRDs from the bundled manifest for your target release. The `deploy/crds.yaml` bundle contains every CRD the controller needs (`VirtualServer`, `VirtualServerRoute`, `Policy`, `TransportServer`, `GlobalConfiguration`, `DNSEndpoint`). The bundle deliberately excludes the App Protect CRDs, which PLM owns. ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` ## Section 1: Upgrade a VirtualServer-based deployment Follow this section if your existing NGINX Ingress Controller deployment routes traffic through `k8s.nginx.org/v1` VirtualServer resources that reference a Policy resource with `waf.apPolicy` and `waf.securityLogs[].apLogConf` fields. ### Upgrade NGINX Ingress Controller with PLM storage Upgrade the NGINX Ingress Controller release. `--reuse-values` preserves your existing configuration. The `--set` flags overlay PLM storage on top. `--skip-crds` prevents Helm from touching CRDs, because you applied the NGINX Ingress Controller CRDs in the previous step and PLM owns the App Protect CRDs. This example uses HTTPS PLM storage with mutual TLS. For HTTP storage, set `controller.appprotect.plmStorage.url` to `http://:8333` and omit the `caSecret` and `clientSSLSecret` flags. ```shell helm upgrade nic nginx-stable/nginx-ingress \ --namespace nginx-ingress \ --skip-crds \ --reuse-values \ --set controller.image.repository="private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress" \ --set controller.image.tag="" \ --set controller.appprotect.plmStorage.url="https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333" \ --set controller.appprotect.plmStorage.credentialsSecret="plm-system/plm-f5-waf-seaweedfs-auth" \ --set controller.appprotect.plmStorage.caSecret="plm-system/plm-f5-waf-seaweedfs-ca-cert" \ --set controller.appprotect.plmStorage.clientSSLSecret="plm-system/plm-f5-waf-seaweedfs-client-cert" \ --set controller.appprotect.plmStorage.insecureSkipVerify=false ``` Wait for the rollout to complete: ```shell kubectl rollout status deployment/nic-nginx-ingress-controller \ --namespace nginx-ingress \ --timeout=180s ``` Confirm NGINX Ingress Controller is watching the v1 CRDs by inspecting the controller log: ```shell kubectl logs deployment/nic-nginx-ingress-controller \ --namespace nginx-ingress \ --container nginx-ingress | grep 'appprotect.f5.com/v' ``` Expected output includes: ```text Using appprotect.f5.com/v1 CRDs ``` ### Verify Policy status Check each WAF Policy referenced by a VirtualServer: ```shell kubectl get policy --all-namespaces \ --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.state,REASON:.status.reason' ``` Expected output after the first successful fetch: ```text NAMESPACE NAME STATE REASON default waf-policy Valid AddedOrUpdated ``` If a Policy remains in `Warning` with `BundlePending`, the referenced `APPolicy` or `APLogConf` isn't yet `ready` in PLM. Confirm that PLM has compiled the resource, then retry. ### Verify traffic Send a normal request to the VirtualServer and confirm the application responds: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP \ http://webapp.example.com:$IC_HTTP_PORT/ ``` Then send a request that triggers the configured WAF violation and confirm the response is `Request Rejected`. ### Confirm bundles come from PLM storage Check the bundle files in the ingress controller pod: ```shell NIC_POD=$(kubectl get pods --namespace nginx-ingress \ --selector app.kubernetes.io/name=nginx-ingress \ --output jsonpath='{.items[0].metadata.name}') kubectl exec --namespace nginx-ingress $NIC_POD --container nginx-ingress -- \ ls -l /etc/app_protect/bundles/ ``` Files named `fetched___policy.tgz` and `fetched___log_.tgz` confirm that the bundles were fetched from PLM storage. ## Section 2: Upgrade an Ingress-based deployment Follow this section if your existing NGINX Ingress Controller deployment routes traffic through Kubernetes Ingress resources. The procedure is the same as Section 1, with one additional Ingress-specific step. ### Audit Ingress annotations Under PLM, NGINX Ingress Controller doesn't support the App Protect Ingress annotations: - `appprotect.f5.com/app-protect-policy` - `appprotect.f5.com/app-protect-security-log` - `appprotect.f5.com/app-protect-security-log-enable` An Ingress that uses these annotations is accepted but produces a warning after the upgrade. WAF isn't applied to that route. Before you turn on PLM storage, migrate every such Ingress to a `k8s.nginx.org/v1` Policy resource. ### Upgrade NGINX Ingress Controller with PLM storage Run the same `helm upgrade` command shown in [Section 1, step 1](#1-upgrade-nginx-ingress-controller-with-plm-storage). ### Verify Ingress traffic For each Ingress, send a normal request and confirm the application responds. Then send a request that matches a WAF violation and confirm the response is `Request Rejected`. ### Confirm bundles come from PLM storage Use the same procedure as [Section 1, step 4](#4-confirm-bundles-come-from-plm-storage). ## Troubleshooting - **Policy stays in `BundlePending` after the upgrade.** The referenced `APPolicy` or `APLogConf` isn't `ready` in PLM. Run `kubectl describe appolicy ` and inspect the PLM policy-controller logs. - **NGINX Ingress Controller reports the referenced namespace isn't watched.** If the deployment sets `controller.watchNamespace`, include the namespace of every `APPolicy` and `APLogConf` resource. Also include the PLM namespace in `controller.watchSecretNamespace` so NGINX Ingress Controller can observe storage Secret rotation. - **Helm upgrade fails with a CRD conflict.** Confirm PLM is installed and its v1 CRDs are present, then run the upgrade with `--skip-crds`.