TransportServer resources

This document is reference material for the TransportServer resource used by F5 NGINX Ingress Controller.

The TransportServer resource allows you to configure TCP, UDP, and TLS Passthrough load balancing. The resource is implemented as a Custom Resource.

The GitHub repository has examples of the resources for specific use cases.

Prerequisites

• For TCP and UDP, the TransportServer resource must be used in conjunction with the GlobalConfiguration resource, which must be created separately.
• For TLS Passthrough, make sure to enable the -enable-tls-passthrough command-line argument of NGINX Ingress Controller.

TransportServer Specification

The TransportServer resource defines load balancing configuration for TCP, UDP, or TLS Passthrough traffic. Below are a few examples:

• TCP load balancing:

  apiVersion: k8s.nginx.org/v1
  kind: TransportServer
  metadata:
    name: dns-tcp
  spec:
    listener:
      name: dns-tcp
      protocol: TCP
    tls:
      secret: cafe-secret
    upstreams:
    - name: dns-app
      service: dns-service
      port: 5353
    action:
      pass: dns-app
  

• UDP load balancing:

  apiVersion: k8s.nginx.org/v1
  kind: TransportServer
  metadata:
    name: dns-udp
  spec:
    listener:
      name: dns-udp
      protocol: UDP
    upstreams:
    - name: dns-app
      service: dns-service
      port: 5353
    upstreamParameters:
      udpRequests: 1
      udpResponses: 1
    action:
      pass: dns-app
  

• TLS passthrough load balancing:

  apiVersion: k8s.nginx.org/v1
  kind: TransportServer
  metadata:
    name: secure-app
  spec:
    listener:
      name: tls-passthrough
      protocol: TLS_PASSTHROUGH
    host: app.example.com
    upstreams:
    - name: secure-app
      service: secure-app
      port: 8443
    action:
      pass: secure-app
  

* – Required for TLS Passthrough load balancing.

Listener

The listener field references a listener that NGINX will use to accept incoming traffic for the TransportServer. For TCP and UDP, the listener must be defined in the GlobalConfiguration resource. When referencing a listener, both the name and the protocol must match. For TLS Passthrough, use the built-in listener with the name tls-passthrough and the protocol TLS_PASSTHROUGH.

An example:

listener:
  name: dns-udp
  protocol: UDP

TLS

The tls field defines TLS configuration for a TransportServer. Please note the current implementation supports TLS termination on multiple ports, where each application owns a dedicated port - NGINX Ingress Controller terminates TLS connections on each port, where each application uses its own cert/key, and routes connections to appropriate application (service) based on that incoming port (any TLS connection regardless of the SNI on a port will be routed to the application that corresponds to that port). An example configuration is shown below:

secret: cafe-secret

Upstream

The upstream defines a destination for the TransportServer. For example:

name: secure-app
service: secure-app
port: 8443
maxFails: 3
maxConns: 100
failTimeout: 30s
loadBalancingMethod: least_conn

Upstream.Healthcheck

The Healthcheck defines an active health check. In the example below we enable a health check for an upstream and configure all the available parameters:

name: secure-app
service: secure-app
port: 8443
healthCheck:
  enable: true
  interval: 20s
  timeout: 30s
  jitter: 3s
  fails: 5
  passes: 5
  port: 8080

Note:
This feature is only supported with NGINX Plus.

Upstream.Healthcheck.Match

The match controls the data to send and the response to expect for the healthcheck:

match:
  send: 'GET / HTTP/1.0\r\nHost: localhost\r\n\r\n'
  expect: "~200 OK"

Both send and expect fields can contain hexadecimal literals with the prefix \x followed by two hex digits, for example, \x80.

See the match directive for details.

UpstreamParameters

The upstream parameters define various parameters for the upstreams:

upstreamParameters:
  udpRequests: 1
  udpResponses: 1
  connectTimeout: 60s
  nextUpstream: true
  nextUpstreamTimeout: 50s
  nextUpstreamTries: 1

SessionParameters

The session parameters define various parameters for TCP connections and UDP sessions.

sessionParameters:
  timeout: 50s

Action

The action defines an action to perform for a client connection/datagram.

In the example below, client connections/datagrams are passed to an upstream dns-app:

action:
  pass: dns-app

Using TransportServer

You can use the usual kubectl commands to work with TransportServer resources, similar to Ingress resources.

For example, the following command creates a TransportServer resource defined in transport-server-passthrough.yaml with the name secure-app:

kubectl apply -f transport-server-passthrough.yaml

transportserver.k8s.nginx.org/secure-app created

You can get the resource by running:

kubectl get transportserver secure-app

NAME         AGE
secure-app   46sm

In the kubectl get and similar commands, you can also use the short name ts instead of transportserver.

Using Snippets

Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure access control in a TransportServer:

apiVersion: k8s.nginx.org/v1
kind: TransportServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  serverSnippets: |
    deny  192.168.1.1;
    allow 192.168.1.0/24;    
  upstreams:
  - name: tea
    service: tea-svc
    port: 80

Snippets can also be specified for a stream. In the example below, we use snippets to limit the number of connections:

apiVersion: k8s.nginx.org/v1
kind: TransportServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  streamSnippets: limit_conn_zone $binary_remote_addr zone=addr:10m;
  serverSnippets: limit_conn addr 1;
  upstreams:
  - name: tea
    service: tea-svc
    port: 80

Note:
To configure snippets in the stream context, use stream-snippets ConfigMap key.

For additional information, view the Advanced configuration with Snippets topic.

Validation

Two types of validation are available for the TransportServer resource:

• Structural validation by the kubectl and Kubernetes API server.
• Comprehensive validation by NGINX Ingress Controller.

Structural Validation

The custom resource definition for the TransportServer includes structural OpenAPI schema which describes the type of every field of the resource.

If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), kubectl and Kubernetes API server will reject such a resource:

• Example of kubectl validation:

  kubectl apply -f transport-server-passthrough.yaml
  

  error: error validating "transport-server-passthrough.yaml": error validating data: ValidationError(TransportServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.TransportServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false
  

• Example of Kubernetes API server validation:

  kubectl apply -f transport-server-passthrough.yaml --validate=false
  

  The TransportServer "secure-app" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list:
  spec.upstreams.port in body must be of type integer: "string"
  

If a resource is not rejected (it doesn’t violate the structural schema), NGINX Ingress Controller will validate it further.

Comprehensive Validation

NGINX Ingress Controller validates the fields of a TransportServer resource. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it.

You can check if NGINX Ingress Controller successfully applied the configuration for a TransportServer. For our example secure-app TransportServer, we can run:

kubectl describe ts secure-app

...
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  3s    nginx-ingress-controller  Configuration for default/secure-app was added or updated

Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied.

If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a TransportServer secure-app with a pass action that references a non-existing upstream, you will get :

kubectl describe ts secure-app

...
Events:
  Type     Reason    Age   From                      Message
  ----     ------    ----  ----                      -------
  Warning  Rejected  2s    nginx-ingress-controller  TransportServer default/secure-app is invalid and was rejected: spec.action.pass: Not found: "some-app"

Note how the events section includes a Warning event with the Rejected reason.

Note: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX.

Customization via ConfigMap

The ConfigMap keys (except for stream-snippets, stream-log-format, resolver-addresses, resolver-ipv6, resolver-valid and resolver-timeout) do not affect TransportServer resources.