# Configure tracing

Type of document: How-to guide
Product: NGINX Gateway Fabric

---


This guide explains how to enable tracing on HTTPRoutes in NGINX Gateway Fabric using the OpenTelemetry Collector. Jaeger is used to process and collect the traces.

## Overview

NGINX Gateway Fabric supports tracing using [OpenTelemetry](https://opentelemetry.io/).

The official [NGINX OpenTelemetry Module](https://github.com/nginxinc/nginx-otel) instruments the NGINX data plane to export traces to a configured collector. Tracing data can be used with an OpenTelemetry Protocol (OTLP) exporter, such as the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector).

This collector can then export data to one or more upstream collectors like [Jaeger](https://www.jaegertracing.io/), [DataDog](https://docs.datadoghq.com/tracing/), and many others. This is called the [Agent model](https://opentelemetry.io/docs/collector/deployment/agent/).

## Install the collectors

The first step is to install the collectors. NGINX Gateway Fabric will be configured to export to the OpenTelemetry Collector, which is configured to export to Jaeger. This model allows the visualization collector (Jaeger) to be swapped with something else, or to add more collectors without needing to reconfigure NGINX Gateway Fabric. It is also possible to configure NGINX Gateway Fabric to export directly to Jaeger.

Create the namespace:

```shell
kubectl create namespace tracing
```

Download the following files containing the configurations for the collectors:

- [Download: otel-collector.yaml](/ngf/otel-collector.yaml)
- [Download: jaeger.yaml](/ngf/jaeger.yaml)

**Note:**  These collectors are for demonstration purposes and are not tuned for production use. 

Then install them:

```shell
kubectl apply -f otel-collector.yaml -f jaeger.yaml -n tracing
```

Ensure the Pods are running:

```shell
kubectl -n tracing get pods
```

```text
NAME                             READY   STATUS    RESTARTS   AGE
jaeger-8469f69b86-bfpk9          1/1     Running   0          9s
otel-collector-f786b7dfd-h2x9l   1/1     Running   0          9s
```

Once running, you can access the Jaeger dashboard by using port-forwarding in the background:

```shell
kubectl port-forward -n tracing svc/jaeger 16686:16686 &
```

Visit [http://127.0.0.1:16686](http://127.0.0.1:16686) to view the dashboard.

## Enable tracing

To enable tracing, you must configure two resources:

- `NginxProxy`: This resource contains global settings relating to the NGINX data plane. It is created and managed by the [cluster operator](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/), and is referenced in the `parametersRef` field of the GatewayClass. By default, an `NginxProxy` resource is created in the same namespace where NGINX Gateway Fabric is installed, attached to the GatewayClass. You can set configuration options in the `nginx` Helm value section, and the resource will be created and attached using the set values.

When installed using the Helm chart, the NginxProxy resource is named `<release-name>-proxy-config` and is created in the release Namespace.

  The `NginxProxy` resource contains configuration for the collector, and applies to all Gateways and routes under the GatewayClass. It does not enable tracing, but is a prerequisite to the next piece of configuration.

**Note:**  You can also override the tracing configuration for a particular Gateway by manually creating and attaching specific `NginxProxy` resources to target the different Gateways. This guide covers the global tracing configuration only. 

- `ObservabilityPolicy`: This resource is a [Direct PolicyAttachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that targets HTTPRoutes or GRPCRoutes. It is created by the [application developer](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/) and enables tracing for a specific route or routes. It requires the `NginxProxy` resource to exist in order to complete the tracing configuration.

For all the possible configuration options for these resources, see the [API reference](/ngf/reference/api.md).

### Install NGINX Gateway Fabric with global tracing configuration

**Note:**  Ensure that you [install the Gateway API resources](/ngf/install/helm.md#installing-the-gateway-api-resources).

Referencing the previously deployed collector, create the following `values.yaml` file for installing NGINX Gateway Fabric:

```yaml
cat <<EOT > values.yaml
nginx:
  config:
    telemetry:
      exporter:
        endpoint: otel-collector.tracing.svc:4317
      spanAttributes:
      - key: cluster-attribute-key
        value: cluster-attribute-value
EOT
```

The span attribute will be added to all tracing spans.

To install:

```shell
helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway -f values.yaml
```

You should see the following configuration:

```shell
kubectl get nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway -o yaml
```

```yaml
apiVersion: gateway.nginx.org/v1alpha2
kind: NginxProxy
metadata:
  name: ngf-proxy-config
spec:
  telemetry:
    exporter:
      endpoint: otel-collector.tracing.svc:4317
    spanAttributes:
      - key: cluster-attribute-key
        value: cluster-attribute-value
```

```shell
kubectl get gatewayclasses.gateway.networking.k8s.io nginx -o yaml
```

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: gateway.nginx.org/nginx-gateway-controller
  parametersRef:
    group: gateway.nginx.org
    kind: NginxProxy
    name: ngf-proxy-config
status:
  conditions:
    - lastTransitionTime: "2024-05-22T15:18:35Z"
      message: GatewayClass is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-05-22T15:18:35Z"
      message: Gateway API CRD versions are supported
      observedGeneration: 1
      reason: SupportedVersion
      status: "True"
      type: SupportedVersion
    - lastTransitionTime: "2024-05-22T15:18:35Z"
      message: parametersRef resource is resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
```

If you already have NGINX Gateway Fabric installed, then you can modify the `NginxProxy` resource to include the tracing configuration:

```shell
kubectl edit nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway
```

You can now create the application, route, and tracing policy.

### Create the application and route

Create the basic **coffee** application:

```yaml
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffee
spec:
  replicas: 2
  selector:
    matchLabels:
      app: coffee
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: coffee
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: coffee
EOF
```

Create the Gateway resource and HTTPRoute for the application:

```yaml
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cafe
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: coffee
spec:
  parentRefs:
  - name: cafe
  hostnames:
  - "cafe.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee
    backendRefs:
    - name: coffee
      port: 80
EOF
```

After creating the Gateway resource, NGINX Gateway Fabric will provision an NGINX Pod and Service fronting it to route traffic. Verify the gateway is created:

```shell
kubectl describe gateways.gateway.networking.k8s.io cafe
```

Verify the status is `Accepted`:

```text
Status:
  Addresses:
    Type:   IPAddress
    Value:  10.96.36.219
  Conditions:
    Last Transition Time:  2026-01-09T05:40:37Z
    Message:               The Gateway is accepted
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Last Transition Time:  2026-01-09T05:40:37Z
    Message:               The Gateway is programmed
    Observed Generation:   1
    Reason:                Programmed
    Status:                True
    Type:                  Programmed
```

Save the public IP address and port(s) of the Gateway into shell variables:

```text
GW_IP=XXX.YYY.ZZZ.III
GW_PORT=<port number>
```

Check that traffic can flow to the application.

**Note:**  If you have a DNS record allocated for `cafe.example.com`, you can send the request directly to that hostname, without needing to resolve. 

```shell
curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee
```

You should receive a response from the coffee Pod.

```text
Server address: 10.244.0.69:8080
Server name: coffee-6b8b6d6486-k5w5w
URI: /coffee
```

You shouldn't see any information from the [Jaeger dashboard](http://127.0.0.1:16686) yet: you need to create the `ObservabilityPolicy`.

### Create the ObservabilityPolicy

To enable tracing for the coffee HTTPRoute, create the following policy:

```yaml
kubectl apply -f - <<EOF
apiVersion: gateway.nginx.org/v1alpha2
kind: ObservabilityPolicy
metadata:
  name: coffee
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: coffee
  tracing:
    strategy: ratio
    ratio: 75
    spanAttributes:
    - key: coffee-key
      value: coffee-value
EOF
```

This policy attaches to the coffee HTTPRoute and enables ratio-based tracing, sampling 75% of requests. The span attribute is only included in the spans for the routes referenced in this policy.

Check the status of the policy:

```shell
kubectl describe observabilitypolicies.gateway.nginx.org coffee
```

```text
Status:
  Ancestors:
    Ancestor Ref:
      Group:      gateway.networking.k8s.io
      Kind:       HTTPRoute
      Name:       coffee
      Namespace:  default
    Conditions:
      Last Transition Time:  2024-05-23T18:13:03Z
      Message:               Policy is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller
```

The `message` field shows the policy is accepted. Run the next command multiple times to create new traffic.

```shell
curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee
```

Once complete, refresh the Jaeger dashboard. You should see a service entry called `ngf:default:cafe`, and a few traces. The default service name is `ngf:<gateway-namespace>:<gateway-name>`.

![image](/ngf/img/jaeger-trace-overview.png)

<br></br>

Select a trace to view the attributes.

![image](/ngf/img/jaeger-trace-attributes.png)

The trace includes the attribute from the global NginxProxy resource as well as the attribute from the ObservabilityPolicy.

## See also

- [Data plane configuration](/ngf/how-to/data-plane-configuration.md): learn how to dynamically update the NGINX Gateway Fabric global data plane configuration, including how to override the telemetry configuration for a particular Gateway.
- [Custom policies](/ngf/overview/custom-policies.md): learn about how NGINX Gateway Fabric custom policies work.
- [API reference](/ngf/reference/api.md): all configuration fields for the policies mentioned in this guide