Upgrade NGINX Ingress Controller to F5 WAF for NGINX with PLM

Use this guide to upgrade an existing F5 NGINX Ingress Controller + F5 WAF for NGINX deployment from in-pod App Protect policy compilation to Policy Lifecycle Management (PLM). PLM compiles APPolicy and APLogConf resources in a dedicated controller and stores the resulting bundles in an in-cluster S3-compatible object store. NGINX Ingress Controller then fetches the compiled bundles instead of compiling them in the data plane.

Under PLM, the fields you already write on the NGINX Ingress Controller Policy resource (waf.apPolicy and waf.securityLogs[].apLogConf) continue to work unchanged. You don’t need to rewrite the k8s.nginx.org/v1 Policy manifest for a VirtualServer or Ingress during the upgrade.

By the end of this guide, you’ll have:

• The PLM backend installed alongside your existing NGINX Ingress Controller deployment.
• NGINX Ingress Controller upgraded to a PLM-capable release with PLM storage configured.
• Existing APPolicy and APLogConf resources adopted by PLM and compiled into bundles.
• WAF-protected traffic served by NGINX Ingress Controller, with bundles fetched from PLM storage instead of compiled in the data plane.

Before you begin

Before you start, make sure you have:

• An existing NGINX Ingress Controller + F5 WAF for NGINX deployment running in your cluster.
• Existing APPolicy and APLogConf resources served by the appprotect.f5.com/v1beta1 CRDs shipped with your current NGINX Ingress Controller installation.
• kubectl and Helm access to the cluster.
• Credentials for private-registry.nginx.com.

Record your current values before you begin:

This guide uses nic as the NGINX Ingress Controller release name, nginx-ingress as the NGINX Ingress Controller namespace, plm-system as the PLM namespace, and plm as the PLM release name. Replace these with your own values consistently throughout.

Deploy PLM infrastructure

Install the PLM backend into the plm-system namespace. The install adds the appprotect.f5.com/v1 versions to the existing appprotect.f5.com CRDs. Because the v1 versions are a superset of v1beta1, adding them doesn’t affect NGINX Ingress Controller while it still watches v1beta1.

The Policy Lifecycle Manager (PLM) backend runs as a Kubernetes operator. It watches WAF custom resources and compiles WAF policies into bundles. The Policy Controller delegates compilation to a separate compiler service over gRPC. The resulting bundles are stored in an embedded SeaweedFS S3-compatible object store.

F5 WAF for NGINX is installed using a separate Helm chart from your NGINX data plane. The steps in this section install only the F5 WAF for NGINX PLM components and do not affect your existing NGINX installation.

Create the registry pull secret

Create a namespace for the PLM components, store your JWT in a Kubernetes Secret, then create the registry pull secret for the private F5 container registry.

1. Create the namespace and store your JWT. The following commands assume your JWT file is named license.jwt:

   Copy

   kubectl create namespace plm-system
   
   kubectl create secret generic jwt-reg-secret \
     --namespace plm-system \
     --from-file=license.jwt

2. Retrieve the JWT from the Secret and create the registry pull secret:

   Copy

   JWT=$(kubectl get secret jwt-reg-secret \
     --namespace plm-system \
     -o jsonpath='{.data.license\.jwt}' | base64 -d)
   
   kubectl create secret docker-registry regcred \
     --namespace plm-system \
     --docker-server=private-registry.nginx.com \
     --docker-username="$JWT" \
     --docker-password=none \
     --dry-run=client --output yaml | kubectl apply -f -

Install the Policy Controller

Create a values file for the Helm installation.

The securityUpdatesRepo.cert and securityUpdatesRepo.key fields are optional. They are only required if your signature repository needs certificate-based authentication. The Policy Controller starts successfully with these fields left empty.

If your signature repository requires them, replace <BASE64_NGINX_REPO_CRT> and <BASE64_NGINX_REPO_KEY> with the base64-encoded contents of your nginx-repo.crt and nginx-repo.key files. To encode them, run:

base64 --wrap=0 < nginx-repo.crt
base64 --wrap=0 < nginx-repo.key

Create /tmp/plm-values.yaml:

imagePullSecrets:
  - name: regcred
securityUpdatesRepo:
  cert: "<BASE64_NGINX_REPO_CRT>"  # optional: only needed for authenticated signature repository access
  key: "<BASE64_NGINX_REPO_KEY>"   # optional: only needed for authenticated signature repository access
policyController:
  image:
    tag: "5.14.0"
compiler:
  image:
    tag: "5.14.0"
seaweedfsOperatorConfig:
  seaweedfs:
    image:
      tag: "5.14.0"
seaweedfs-operator:
  image:
    tag: "5.14.0"
    pullSecrets: regcred

Enable TLS for PLM storage (optional)

By default, communication between PLM components and the SeaweedFS object store uses unencrypted HTTP. To enable TLS, add a certificates block to /tmp/plm-values.yaml:

seaweedfsOperatorConfig:
  seaweedfs:
    certificates:
      enabled: true

Create Secrets before installing

The PLM chart does not generate certificates. You must create the five Secrets listed in the commands below before running helm upgrade --install. If any Secret is missing, the SeaweedFS pods will fail to mount their certificates and will not start.

Enabling TLS on an existing install

If you’re enabling TLS on an existing installation, the storage backend restarts and objects written before the switch can become orphaned. See the APPolicy shows invalid with unexpected EOF after enabling TLS entry in the troubleshooting section. A fresh installation with TLS enabled from the start doesn’t have this issue.

Create the Secrets from your CA and certificate files before installing. The chart expects Secret names in the form <release>-f5-waf-seaweedfs-<component> — for the plm release name used in this tutorial, those are:

kubectl create secret generic plm-f5-waf-seaweedfs-ca-cert \
  --namespace plm-system \
  --from-file=tls.crt=<PATH/TO/CA_CERT> \
  --from-file=ca.crt=<PATH/TO/CA_CERT>

kubectl create secret tls plm-f5-waf-seaweedfs-master-cert \
  --namespace plm-system \
  --cert=<PATH/TO/MASTER_CERT> \
  --key=<PATH/TO/MASTER_KEY>

kubectl create secret tls plm-f5-waf-seaweedfs-volume-cert \
  --namespace plm-system \
  --cert=<PATH/TO/VOLUME_CERT> \
  --key=<PATH/TO/VOLUME_KEY>

kubectl create secret tls plm-f5-waf-seaweedfs-filer-cert \
  --namespace plm-system \
  --cert=<PATH/TO/FILER_CERT> \
  --key=<PATH/TO/FILER_KEY>

kubectl create secret tls plm-f5-waf-seaweedfs-client-cert \
  --namespace plm-system \
  --cert=<PATH/TO/CLIENT_CERT> \
  --key=<PATH/TO/CLIENT_KEY>

The CA Secret requires both tls.crt and ca.crt keys, both pointing to the same CA certificate file. The PLM chart mounts the CA using tls.crt into the Policy Controller, compiler, and SeaweedFS pods. The data plane’s S3 client reads ca.crt from the same Secret when verifying the storage endpoint. The four component Secrets use kubectl create secret tls, which produces tls.crt and tls.key — no ca.crt key is needed for them.

Replace each <PATH/TO/*> placeholder with the path to the corresponding certificate and key file from your PKI. The CA must sign all component certificates. If you don’t have an existing PKI, generate a CA and sign the five component certificates before proceeding.

Install the chart

Add the NGINX Helm repository and install the chart:

helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update nginx-stable

helm upgrade --install plm nginx-stable/f5-waf-policy-controller \
  --version 5.14.0 \
  --namespace plm-system \
  --values /tmp/plm-values.yaml

To see all available configuration options for the PLM chart, run:

helm show values nginx-stable/f5-waf-policy-controller --version 5.14.0

Verify the deployment

Wait for all PLM components to become ready. The Policy Controller’s init container waits for both the compiler service and the SeaweedFS S3 endpoint to be available before it starts, so the controller pod will show Init:0/1 until SeaweedFS is ready.

Wait for the SeaweedFS storage backend:

kubectl rollout status deployment/plm-seaweedfs-operator \
  --namespace plm-system --timeout=120s

The SeaweedFS operator creates the SeaweedFS pods after it reconciles the SeaweedFS custom resource, so there is a window where the operator deployment is ready but no SeaweedFS pods exist yet. Poll until the pods appear and are ready:

end=$((SECONDS + 300))
until kubectl wait pods \
    --selector app.kubernetes.io/name=seaweedfs \
    --for=condition=Ready \
    --namespace plm-system \
    --timeout=10s 2>/dev/null; do
  if [ $SECONDS -ge $end ]; then
    echo "Timed out waiting for SeaweedFS pods"
    exit 1
  fi
  sleep 5
done

Wait for the Policy Controller:

kubectl rollout status deployment/plm-f5-waf-policy-controller \
  --namespace plm-system --timeout=180s

Confirm all pods are running:

kubectl get pods --namespace plm-system

Example output:

NAME                                               READY   STATUS    RESTARTS
plm-f5-waf-compiler-service-xxxxx                  1/1     Running   0
plm-f5-waf-policy-controller-xxxxx                 1/1     Running   0
plm-seaweedfs-operator-xxxxx                       1/1     Running   0
plm-f5-waf-seaweed-master-0                        1/1     Running   0
plm-f5-waf-seaweed-filer-0                         1/1     Running   0
plm-f5-waf-seaweed-volume-0                        1/1     Running   0
plm-f5-waf-seaweed-volume-1                        1/1     Running   0
plm-f5-waf-seaweed-volume-2                        1/1     Running   0

Confirm the CRDs are present:

kubectl get crd | grep appprotect.f5.com

Expected output:

aplogconfs.appprotect.f5.com
appolicies.appprotect.f5.com
apsignatures.appprotect.f5.com
apusersigs.appprotect.f5.com

All eight pods running and all four CRDs present confirms the PLM backend is ready.

Update the CRDs

Skip this step on a fresh install — Helm installs the CRDs automatically. Only follow these steps when upgrading an existing PLM installation.

When upgrading PLM, apply the CRDs manually before running helm upgrade:

kubectl apply -f https://raw.githubusercontent.com/nginx/waf-policy-controller/5.14.0/manifests/1-deploy-crds.yaml

Troubleshoot the deployment

These are the most common failures during PLM installation, roughly in order of likelihood.

kubectl get events --namespace plm-system --field-selector reason=Failed

kubectl get pods --namespace plm-system --selector app.kubernetes.io/name=seaweedfs

kubectl get pvc --namespace plm-system
kubectl get storageclass

kubectl logs --namespace plm-system plm-f5-waf-seaweed-filer-0 | grep "not found"

Check the Policy Controller logs

Use the Policy Controller logs to diagnose any policy-related failure:

kubectl logs --namespace plm-system deploy/plm-f5-waf-policy-controller -c policy-controller

The -c policy-controller flag is required because the pod has more than one container. The containers are distroless, so kubectl exec isn’t available for interactive debugging.

After the install completes, PLM adopts every existing APPolicy and APLogConf resource in the cluster. It does this by adding the appprotect.f5.com/finalizer finalizer and compiling each resource against its current signature package. Your running NGINX Ingress Controller continues to compile the same resources in-pod. The two mechanisms operate independently until you upgrade NGINX Ingress Controller.

Confirm PLM has compiled the existing resources:

kubectl get appolicy --all-namespaces \
  --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.bundle.state'

kubectl get aplogconf --all-namespaces \
  --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.bundle.state'

Every resource should report STATE: ready. If any resource isn’t ready, inspect the PLM policy-controller logs, resolve the compilation errors, and then continue.

Look up the PLM storage endpoint and credentials

NGINX Ingress Controller connects to PLM’s SeaweedFS filer over S3 to fetch compiled bundles. Before you run helm upgrade, collect these four values from the PLM installation:

1. PLM storage URL: the SeaweedFS filer endpoint (HTTPS or HTTP).
2. Credentials Secret: the S3 credentials Secret. The access key ID is admin by default. The secret access key is in the seaweedfs_admin_secret field.
3. CA Secret (HTTPS only): verifies the SeaweedFS filer certificate.
4. Client TLS Secret (mutual TLS only): presented by NGINX Ingress Controller when it connects to the filer.

List the Services PLM created and identify the filer:

kubectl get service --namespace plm-system

Expected output includes an entry similar to:

NAME                       TYPE        CLUSTER-IP   PORT(S)
plm-f5-waf-seaweed-filer   ClusterIP   10.0.0.10    8333/TCP,9333/TCP,...

Assemble the URL from the service name, namespace, and port. Use 9333 for HTTPS and 8333 for HTTP:

• HTTPS (mTLS): https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333
• HTTP: http://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:8333

List the Secrets PLM created:

kubectl get secret --namespace plm-system

The default PLM install creates three Secrets that NGINX Ingress Controller references:

• plm-f5-waf-seaweedfs-auth: SeaweedFS credentials.
• plm-f5-waf-seaweedfs-ca-cert: CA certificate for the HTTPS filer.
• plm-f5-waf-seaweedfs-client-cert: client TLS certificate for mTLS.

Record the Secret references in <namespace>/<name> form. When you run helm upgrade, pass all four values to NGINX Ingress Controller using --set controller.appprotect.plmStorage.* flags.

Apply the NGINX Ingress Controller CRDs

Before you run helm upgrade, apply the NGINX Ingress Controller CRDs from the bundled manifest for your target release. The deploy/crds.yaml bundle contains every CRD the controller needs (VirtualServer, VirtualServerRoute, Policy, TransportServer, GlobalConfiguration, DNSEndpoint). The bundle deliberately excludes the App Protect CRDs, which PLM owns.

kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v5.5.4/deploy/crds.yaml

Section 1: Upgrade a VirtualServer-based deployment

Follow this section if your existing NGINX Ingress Controller deployment routes traffic through k8s.nginx.org/v1 VirtualServer resources that reference a Policy resource with waf.apPolicy and waf.securityLogs[].apLogConf fields.

Upgrade NGINX Ingress Controller with PLM storage

Upgrade the NGINX Ingress Controller release. --reuse-values preserves your existing configuration. The --set flags overlay PLM storage on top. --skip-crds prevents Helm from touching CRDs, because you applied the NGINX Ingress Controller CRDs in the previous step and PLM owns the App Protect CRDs.

This example uses HTTPS PLM storage with mutual TLS. For HTTP storage, set controller.appprotect.plmStorage.url to http://<host>:8333 and omit the caSecret and clientSSLSecret flags.

helm upgrade nic nginx-stable/nginx-ingress \
  --namespace nginx-ingress \
  --skip-crds \
  --reuse-values \
  --set controller.image.repository="private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress" \
  --set controller.image.tag="5.5.4" \
  --set controller.appprotect.plmStorage.url="https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333" \
  --set controller.appprotect.plmStorage.credentialsSecret="plm-system/plm-f5-waf-seaweedfs-auth" \
  --set controller.appprotect.plmStorage.caSecret="plm-system/plm-f5-waf-seaweedfs-ca-cert" \
  --set controller.appprotect.plmStorage.clientSSLSecret="plm-system/plm-f5-waf-seaweedfs-client-cert" \
  --set controller.appprotect.plmStorage.insecureSkipVerify=false

Wait for the rollout to complete:

kubectl rollout status deployment/nic-nginx-ingress-controller \
  --namespace nginx-ingress \
  --timeout=180s

Confirm NGINX Ingress Controller is watching the v1 CRDs by inspecting the controller log:

kubectl logs deployment/nic-nginx-ingress-controller \
  --namespace nginx-ingress \
  --container nginx-ingress | grep 'appprotect.f5.com/v'

Expected output includes:

Using appprotect.f5.com/v1 CRDs

Verify Policy status

Check each WAF Policy referenced by a VirtualServer:

kubectl get policy --all-namespaces \
  --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.state,REASON:.status.reason'

Expected output after the first successful fetch:

NAMESPACE   NAME         STATE   REASON
default     waf-policy   Valid   AddedOrUpdated

If a Policy remains in Warning with BundlePending, the referenced APPolicy or APLogConf isn’t yet ready in PLM. Confirm that PLM has compiled the resource, then retry.

Verify traffic

Send a normal request to the VirtualServer and confirm the application responds:

curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP \
  http://webapp.example.com:$IC_HTTP_PORT/

Then send a request that triggers the configured WAF violation and confirm the response is Request Rejected.

Confirm bundles come from PLM storage

Check the bundle files in the ingress controller pod:

NIC_POD=$(kubectl get pods --namespace nginx-ingress \
  --selector app.kubernetes.io/name=nginx-ingress \
  --output jsonpath='{.items[0].metadata.name}')

kubectl exec --namespace nginx-ingress $NIC_POD --container nginx-ingress -- \
  ls -l /etc/app_protect/bundles/

Files named fetched_<namespace>_<policy-name>_policy.tgz and fetched_<namespace>_<policy-name>_log_<index>.tgz confirm that the bundles were fetched from PLM storage.

Section 2: Upgrade an Ingress-based deployment

Follow this section if your existing NGINX Ingress Controller deployment routes traffic through Kubernetes Ingress resources. The procedure is the same as Section 1, with one additional Ingress-specific step.

Audit Ingress annotations

Under PLM, NGINX Ingress Controller doesn’t support the App Protect Ingress annotations:

• appprotect.f5.com/app-protect-policy
• appprotect.f5.com/app-protect-security-log
• appprotect.f5.com/app-protect-security-log-enable

An Ingress that uses these annotations is accepted but produces a warning after the upgrade. WAF isn’t applied to that route. Before you turn on PLM storage, migrate every such Ingress to a k8s.nginx.org/v1 Policy resource.

Upgrade NGINX Ingress Controller with PLM storage

Run the same helm upgrade command shown in Section 1, step 1.

Verify Ingress traffic

For each Ingress, send a normal request and confirm the application responds. Then send a request that matches a WAF violation and confirm the response is Request Rejected.

Confirm bundles come from PLM storage

Use the same procedure as Section 1, step 4.

Troubleshooting

• Policy stays in BundlePending after the upgrade. The referenced APPolicy or APLogConf isn’t ready in PLM. Run kubectl describe appolicy <name> and inspect the PLM policy-controller logs.
• NGINX Ingress Controller reports the referenced namespace isn’t watched. If the deployment sets controller.watchNamespace, include the namespace of every APPolicy and APLogConf resource. Also include the PLM namespace in controller.watchSecretNamespace so NGINX Ingress Controller can observe storage Secret rotation.
• Helm upgrade fails with a CRD conflict. Confirm PLM is installed and its v1 CRDs are present, then run the upgrade with --skip-crds.