VirtualServer and VirtualServerRoute resources

This document is reference material for the VirtualServer and VirtualServerRoute resources used by F5 NGINX Ingress Controller.

VirtualServer and VirtualServerRoute resources are load balancing configurations recommended as an alternative to the Ingress resource.

They enable use cases not supported with the Ingress resource, such as traffic splitting and advanced content-based routing. The resources are implemented as Custom Resources.

The GitHub repository has examples of the resources for specific use cases.

VirtualServer specification

The VirtualServer resource defines load balancing configuration for a domain name, such as example.com. Below is an example of such configuration:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  listener:
    http: http-8083
    https: https-8443
  tls:
    secret: cafe-secret
  gunzip: on
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  - name: coffee
    service: coffee-svc
    port: 80
  routes:
  - path: /tea
    action:
      pass: tea
  - path: /coffee
    action:
      pass: coffee
  - path: ~ ^/decaf/.*\\.jpg$
    action:
      pass: coffee
  - path: = /green/tea
    action:
      pass: tea

VirtualServer.TLS

The tls field defines TLS configuration for a VirtualServer. For example:

secret: cafe-secret
redirect:
  enable: true

VirtualServer.TLS.Redirect

The redirect field configures a TLS redirect for a VirtualServer:

enable: true
code: 301
basedOn: scheme

VirtualServer.TLS.CertManager

The cert-manager field configures x509 automated Certificate management for VirtualServer resources using cert-manager (cert-manager.io). Please see the cert-manager configuration documentation for more information on deploying and configuring Issuers. Example:

cert-manager:
  cluster-issuer: "my-issuer-name"

VirtualServer.Listener

The listener field defines a custom HTTP and/or HTTPS listener. The respective listeners used must reference the name of a listener defined using a GlobalConfiguration resource. For example:

http: http-8083
https: https-8443

VirtualServer.ExternalDNS

The externalDNS field configures controlling DNS records dynamically for VirtualServer resources using ExternalDNS. Please see the ExternalDNS configuration documentation for more information on deploying and configuring ExternalDNS and Providers. Example:

enable: true

VirtualServer.ExternalDNS.ProviderSpecific

The providerSpecific field of the externalDNS block allows the specification of provider specific properties which is a list of key value pairs of configurations which are specific to individual DNS providers. Example:

- name: my-name
  value: my-value
- name: my-name2
  value: my-value2

VirtualServer.Policy

The policy field references a Policy resource by its name and optional namespace. For example:

name: access-control

VirtualServer.Route

The route defines rules for matching client requests to actions like passing a request to an upstream. For example:

  path: /tea
  action:
    pass: tea

* – a route must include exactly one of the following: action, splits, or route.

VirtualServerRoute specification

The VirtualServerRoute resource defines a route for a VirtualServer. It can consist of one or multiple subroutes. The VirtualServerRoute is an alternative to Mergeable Ingress types.

In the example below, the VirtualServer cafe from the namespace cafe-ns defines a route with the path /coffee, which is further defined in the VirtualServerRoute coffee from the namespace coffee-ns.

VirtualServer:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
  namespace: cafe-ns
spec:
  host: cafe.example.com
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  routes:
  - path: /tea
    action:
      pass: tea
  - path: /coffee
    route: coffee-ns/coffee

VirtualServerRoute:

apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
  name: coffee
  namespace: coffee-ns
spec:
  host: cafe.example.com
  upstreams:
  - name: latte
    service: latte-svc
    port: 80
  - name: espresso
    service: espresso-svc
    port: 80
  subroutes:
  - path: /coffee/latte
    action:
      pass: latte
  - path: /coffee/espresso
    action:
      pass: espresso

Note that each subroute must have a path that starts with the same prefix (here /coffee), which is defined in the route of the VirtualServer. Additionally, the host in the VirtualServerRoute must be the same as the host of the VirtualServer.

VirtualServerRoute.Subroute

The subroute defines rules for matching client requests to actions like passing a request to an upstream. For example:

path: /coffee
action:
  pass: coffee

* – a subroute must include exactly one of the following: action or splits.

Common VirtualServer and VirtualServerRoute specifications

Upstream

The upstream defines a destination for the routing configuration. For example:

name: tea
service: tea-svc
subselector:
  version: canary
port: 80
lb-method: round_robin
fail-timeout: 10s
max-fails: 1
max-conns: 32
keepalive: 32
connect-timeout: 30s
read-timeout: 30s
send-timeout: 30s
next-upstream: "error timeout non_idempotent"
next-upstream-timeout: 5s
next-upstream-tries: 10
client-max-body-size: 2m
tls:
  enable: true

Note: The WebSocket protocol is supported without any additional configuration.

Upstream.Buffers

The buffers field configures the buffers used for reading a response from the upstream server for a single connection:

number: 4
size: 8K

See the proxy_buffers directive for additional information.

Upstream.TLS

Upstream.Queue

The queue field configures a queue. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request:

size: 10
timeout: 60s

See queue directive for additional information.

Note: This feature is supported only in NGINX Plus.

Upstream.Healthcheck

The Healthcheck defines an active health check. In the example below we enable a health check for an upstream and configure all the available parameters, including the slow-start parameter combined with mandatory and persistent:

name: tea
service: tea-svc
port: 80
slow-start: 30s
healthCheck:
  enable: true
  path: /healthz
  interval: 20s
  jitter: 3s
  fails: 5
  passes: 5
  port: 8080
  tls:
    enable: true
  connect-timeout: 10s
  read-timeout: 10s
  send-timeout: 10s
  headers:
  - name: Host
    value: my.service
  statusMatch: "! 500"
  mandatory: true
  persistent: true
  keepalive-time: 60s

Note: This feature is supported only in NGINX Plus.

Upstream.SessionCookie

The SessionCookie field configures session persistence which allows requests from the same client to be passed to the same upstream server. The information about the designated upstream server is passed in a session cookie generated by NGINX Plus.

In the example below, we configure session persistence with a session cookie for an upstream and configure all the available parameters:

name: tea
service: tea-svc
port: 80
sessionCookie:
  enable: true
  name: srv_id
  path: /
  expires: 1h
  domain: .example.com
  httpOnly: false
  secure: true
  samesite: strict

See the sticky directive for additional information. The session cookie corresponds to the sticky cookie method.

Note: This feature is supported only in NGINX Plus.

Header

The header defines an HTTP Header:

name: Host
value: example.com

Action

The action defines an action to perform for a request.

In the example below, client requests are passed to an upstream coffee:

 path: /coffee
 action:
  pass: coffee

* – an action must include exactly one of the following: pass, redirect, return or proxy.

Action.Redirect

The redirect action defines a redirect to return for a request.

In the example below, client requests are passed to a url http://www.nginx.com:

redirect:
  url: http://www.nginx.com
  code: 301

Action.Return

The return action defines a preconfigured response for a request.

In the example below, NGINX will respond with the preconfigured response for every request:

return:
  code: 200
  type: text/plain
  body: "Hello World\n"
  headers:
  - name: x-coffee
    value: espresso

* – Supported NGINX variables: $request_uri, $request_method, $request_body, $scheme, $http_, $args, $arg_, $cookie_, $host, $request_time, $request_length, $nginx_version, $pid, $connection, $remote_addr, $remote_port, $time_iso8601, $time_local, $server_addr, $server_port, $server_name, $server_protocol, $connections_active, $connections_reading, $connections_writing and $connections_waiting.

Action.Return.Header

The header defines an HTTP Header for a canned response in an actionReturn:

name: x-coffee
value: espresso

Action.Proxy

The proxy action passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers).

In the example below, the request URI is rewritten to /, and the request and the response headers are modified:

proxy:
  upstream: coffee
  requestHeaders:
    pass: true
    set:
    - name: My-Header
      value: Value
    - name: Client-Cert
      value: ${ssl_client_escaped_cert}
  responseHeaders:
    add:
    - name: My-Header
      value: Value
    - name: IC-Nginx-Version
      value: ${nginx_version}
      always: true
    hide:
    - x-internal-version
    ignore:
    - Expires
    - Set-Cookie
    pass:
    - Server
  rewritePath: /

Action.Proxy.RequestHeaders

The RequestHeaders field modifies the headers of the request to the proxied upstream server.

Action.Proxy.RequestHeaders.Set.Header

The header defines an HTTP Header:

name: My-Header
value: My-Value

It is possible to override the default value of the Host header, which NGINX Ingress Controller sets to $host:

name: Host
value: example.com

* – Supported NGINX variables: $request_uri, $request_method, $request_body, $scheme, $http_, $args, $arg_, $cookie_, $host, $request_time, $request_length, $nginx_version, $pid, $connection, $remote_addr, $remote_port, $time_iso8601, $time_local, $server_addr, $server_port, $server_name, $server_protocol, $connections_active, $connections_reading, $connections_writing, $connections_waiting, $ssl_cipher, $ssl_ciphers, $ssl_client_cert, $ssl_client_escaped_cert, $ssl_client_fingerprint, $ssl_client_i_dn, $ssl_client_i_dn_legacy, $ssl_client_raw_cert, $ssl_client_s_dn, $ssl_client_s_dn_legacy, $ssl_client_serial, $ssl_client_v_end, $ssl_client_v_remain, $ssl_client_v_start, $ssl_client_verify, $ssl_curves, $ssl_early_data, $ssl_protocol, $ssl_server_name, $ssl_session_id, $ssl_session_reused, $jwt_claim_ (NGINX Plus only) and $jwt_header_ (NGINX Plus only).

Action.Proxy.ResponseHeaders

The ResponseHeaders field modifies the headers of the response to the client.

* – Default hidden headers are: Date, Server, X-Pad and X-Accel-....

** – The following fields can be ignored: X-Accel-Redirect, X-Accel-Expires, X-Accel-Limit-Rate, X-Accel-Buffering, X-Accel-Charset, Expires, Cache-Control, Set-Cookie and Vary.

AddHeader

The addHeader defines an HTTP Header with an optional always field:

name: My-Header
value: My-Value
always: true

* – Supported NGINX variables: $request_uri, $request_method, $request_body, $scheme, $http_, $args, $arg_, $cookie_, $host, $request_time, $request_length, $nginx_version, $pid, $connection, $remote_addr, $remote_port, $time_iso8601, $time_local, $server_addr, $server_port, $server_name, $server_protocol, $connections_active, $connections_reading, $connections_writing, $connections_waiting, $ssl_cipher, $ssl_ciphers, $ssl_client_cert, $ssl_client_escaped_cert, $ssl_client_fingerprint, $ssl_client_i_dn, $ssl_client_i_dn_legacy, $ssl_client_raw_cert, $ssl_client_s_dn, $ssl_client_s_dn_legacy, $ssl_client_serial, $ssl_client_v_end, $ssl_client_v_remain, $ssl_client_v_start, $ssl_client_verify, $ssl_curves, $ssl_early_data, $ssl_protocol, $ssl_server_name, $ssl_session_id, $ssl_session_reused, $jwt_claim_ (NGINX Plus only) and $jwt_header_ (NGINX Plus only).

Note:
If always is false, the response header is added only if the response status code is any of 200, 201, 204, 206, 301, 302, 303, 304, 307 or 308.

Split

The split defines a weight for an action as part of the splits configuration.

In the example below NGINX passes 80% of requests to the upstream coffee-v1 and the remaining 20% to coffee-v2:

splits:
- weight: 80
  action:
    pass: coffee-v1
- weight: 20
  action:
    pass: coffee-v2

Match

The match defines a match between conditions and an action or splits.

In the example below, NGINX routes requests with the path /coffee to different upstreams based on the value of the cookie user:

• user=john -> coffee-future
• user=bob -> coffee-deprecated
• If the cookie is not set or not equal to either john or bob, NGINX routes to coffee-stable

path: /coffee
matches:
- conditions:
  - cookie: user
    value: john
  action:
    pass: coffee-future
- conditions:
  - cookie: user
    value: bob
  action:
    pass: coffee-deprecated
action:
  pass: coffee-stable

In the next example, NGINX routes requests based on the value of the built-in $request_method variable, which represents the HTTP method of a request:

• all POST requests -> coffee-post
• all non-POST requests -> coffee

path: /coffee
matches:
- conditions:
  - variable: $request_method
    value: POST
  action:
    pass: coffee-post
action:
  pass: coffee

Note:
A match must include exactly one of the following: action or splits.

Condition

The condition defines a condition in a match.

Note:
a condition must include exactly one of the following: header, cookie, argument or variable.

Supported NGINX variables: $args, $http2, $https, $remote_addr, $remote_port, $query_string, $request, $request_body, $request_uri, $request_method, $scheme. Find the documentation for each variable here.

The value supports two kinds of matching:

• Case-insensitive string comparison. For example:
  • john – case-insensitive matching that succeeds for strings, such as john, John, JOHN.
  • !john – negation of the case-insensitive matching for john that succeeds for strings, such as bob, anything, '' (empty string).
• Matching with a regular expression. Note that NGINX supports regular expressions compatible with those used by the Perl programming language (PCRE). For example:
  • ~^yes – a case-sensitive regular expression that matches any string that starts with yes. For example: yes, yes123.
  • !~^yes – negation of the previous regular expression that succeeds for strings like YES, Yes123, noyes. (The negation mechanism is not part of the PCRE syntax).
  • ~*no$ – a case-insensitive regular expression that matches any string that ends with no. For example: no, 123no, 123NO.

Note:
A value must not include any unescaped double quotes (") and must not end with an unescaped backslash (\). For example, the following are invalid values: some"value, somevalue\.

ErrorPage

The errorPage defines a custom response for a route for the case when either an upstream server responds with (or NGINX generates) an error status code. The custom response can be a redirect or a canned response. See the error_page directive for more information.

path: /coffee
errorPages:
- codes: [502, 503]
  redirect:
    code: 301
    url: https://nginx.org
- codes: [404]
  return:
    code: 200
    body: "Original resource not found, but success!"

Note:
An errorPage must include exactly one of the following: return or redirect.

ErrorPage.Redirect

The redirect defines a redirect for an errorPage.

In the example below, NGINX responds with a redirect when a response from an upstream server has a 404 status code.

codes: [404]
redirect:
  code: 301
  url: ${scheme}://cafe.example.com/error.html

ErrorPage.Return

The return defines a canned response for an errorPage.

In the example below, NGINX responds with a canned response when a response from an upstream server has either 401 or 403 status code.

codes: [401, 403]
return:
  code: 200
  type: application/json
  body: |
    {\"msg\": \"You don't have permission to do this\"}    
  headers:
  - name: x-debug-original-statuses
    value: ${upstream_status}

ErrorPage.Return.Header

The header defines an HTTP Header for a canned response in an errorPage:

name: x-debug-original-statuses
value: ${upstream_status}

Using VirtualServer and VirtualServerRoute

You can use the usual kubectl commands to work with VirtualServer and VirtualServerRoute resources, similar to Ingress resources.

For example, the following command creates a VirtualServer resource defined in cafe-virtual-server.yaml with the name cafe:

kubectl apply -f cafe-virtual-server.yaml

virtualserver.k8s.nginx.org "cafe" created

You can get the resource by running:

kubectl get virtualserver cafe

NAME   STATE   HOST                   IP            PORTS      AGE
cafe   Valid   cafe.example.com       12.13.23.123  [80,443]   3m

In kubectl get and similar commands, you can use the short name vs instead of virtualserver.

Similarly, for VirtualServerRoute you can use virtualserverroute or the short name vsr.

Using Snippets

Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure several NGINX features in a VirtualServer:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
  namespace: cafe
spec:
  http-snippets: |
    limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s;
    proxy_cache_path /tmp keys_zone=one:10m;    
  host: cafe.example.com
  tls:
    secret: cafe-secret
  server-snippets: |
    limit_req zone=mylimit burst=20;    
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  - name: coffee
    service: coffee-svc
    port: 80
  routes:
  - path: /tea
    location-snippets: |
      proxy_cache one;
      proxy_cache_valid 200 10m;      
    action:
      pass: tea
  - path: /coffee
    action:
      pass: coffee

For additional information, view the Advanced configuration with Snippets topic.

Validation

Two types of validation are available for VirtualServer and VirtualServerRoute resources:

• Structural validation by the kubectl and Kubernetes API server.
• Comprehensive validation by NGINX Ingress Controller.

Structural Validation

The custom resource definitions for VirtualServer and VirtualServerRoute include structural OpenAPI schema which describes the type of every field of those resources.

If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), kubectl and Kubernetes API server will reject such a resource:

• Example of kubectl validation:

  kubectl apply -f cafe-virtual-server.yaml
  

  error: error validating "cafe-virtual-server.yaml": error validating data: ValidationError(VirtualServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.VirtualServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false
  

• Example of Kubernetes API server validation:

  kubectl apply -f cafe-virtual-server.yaml --validate=false
  

  The VirtualServer "cafe" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list:
  spec.upstreams.port in body must be of type integer: "string"
  

If a resource is not rejected (it doesn’t violate the structural schema), NGINX Ingress Controller will validate it further.

Comprehensive Validation

NGINX Ingress Controller validates the fields of the VirtualServer and VirtualServerRoute resources. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it.

You can check if NGINX Ingress Controller successfully applied the configuration for a VirtualServer. For our example cafe VirtualServer, we can run:

kubectl describe vs cafe

...
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  16s   nginx-ingress-controller  Configuration for default/cafe was added or updated

Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied.

If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a VirtualServer cafe with two upstream with the same name tea, you will get:

kubectl describe vs cafe

...
Events:
  Type     Reason    Age   From                      Message
  ----     ------    ----  ----                      -------
  Warning  Rejected  12s   nginx-ingress-controller  VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea"

Note how the events section includes a Warning event with the Rejected reason.

Additionally, this information is also available in the status field of the VirtualServer resource. Note the Status section of the VirtualServer:

kubectl describe vs cafe

...
Status:
  External Endpoints:
    Ip:        12.13.23.123
    Ports:     [80,443]
  Message:  VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea"
  Reason:   Rejected
  State:    Invalid

NGINX Ingress Controller validates VirtualServerRoute resources in a similar way.

Note: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX.

Customization using ConfigMap

You can customize the NGINX configuration for VirtualServer and VirtualServerRoutes resources using the ConfigMap. Most of the ConfigMap keys are supported, with the following exceptions:

• proxy-hide-headers
• proxy-pass-headers
• hsts
• hsts-max-age
• hsts-include-subdomains
• hsts-behind-proxy
• redirect-to-https
• ssl-redirect