Version: Next

opentelemetry

Description#

The opentelemetry Plugin can be used to report tracing data according to the OpenTelemetry Specification.

The Plugin only supports binary-encoded OLTP over HTTP.

Configurations#

By default, configurations of the Service name, tenant ID, collector, and batch span processor are pre-configured in default configuration.

You can change this configuration of the Plugin through the endpoint apisix/admin/plugin_metadata/opentelemetry For example:

note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry -H "X-API-KEY: $admin_key" -X PUT -d '
{
    "trace_id_source": "x-request-id",
    "resource": {
      "service.name": "APISIX"
    },
    "collector": {
      "address": "127.0.0.1:4318",
      "request_timeout": 3,
      "request_headers": {
        "Authorization": "token"
      }
    },
    "batch_span_processor": {
      "drop_on_queue_full": false,
      "max_queue_size": 1024,
      "batch_timeout": 2,
      "inactive_timeout": 1,
      "max_export_batch_size": 16
    },
    "set_ngx_var": false
}'

Attributes#

Name	Type	Required	Default	Valid Values	Description
sampler	object	False	-	-	Sampling configuration.
sampler.name	string	False	`always_off`	["always_on", "always_off", "trace_id_ratio", "parent_base"]	Sampling strategy. To always sample, use `always_on`. To never sample, use `always_off`. To randomly sample based on a given ratio, use `trace_id_ratio`. To use the sampling decision of the span's parent, use `parent_base`. If there is no parent, use the root sampler.
sampler.options	object	False	-	-	Parameters for sampling strategy.
sampler.options.fraction	number	False	0	[0, 1]	Sampling ratio when the sampling strategy is `trace_id_ratio`.
sampler.options.root	object	False	-	-	Root sampler when the sampling strategy is `parent_base` strategy.
sampler.options.root.name	string	False	-	["always_on", "always_off", "trace_id_ratio"]	Root sampling strategy.
sampler.options.root.options	object	False	-	-	Root sampling strategy parameters.
sampler.options.root.options.fraction	number	False	0	[0, 1]	Root sampling ratio when the sampling strategy is `trace_id_ratio`.
additional_attributes	array[string]	False	-	-	Additional attributes appended to the trace span. Support built-in variables in values.
additional_header_prefix_attributes	array[string]	False	-	-	Headers or header prefixes appended to the trace span's attributes. For example, use `x-my-header"` or `x-my-headers-*` to include all headers with the prefix `x-my-headers-`.

Examples#

The examples below demonstrate how you can work with the opentelemetry Plugin for different scenarios.

Enable `opentelemetry` Plugin#

By default, the opentelemetry Plugin is disabled in APISIX. To enable, add the Plugin to your configuration file as such:

config.yaml

plugins:
  - ...
  - opentelemetry

Reload APISIX for changes to take effect.

See static configurations for other available options you can configure in config.yaml.

Send Traces to OpenTelemetry#

The following example demonstrates how to trace requests to a Route and send traces to OpenTelemetry.

Start an OpenTelemetry collector instance in Docker:

docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib

Create a Route with opentelemetry Plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "otel-tracing-route",
    "uri": "/anything",
    "plugins": {
      "opentelemetry": {
        "sampler": {
          "name": "always_on"
        }
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org": 1
      }
    }
  }'

Send a request to the Route:

curl "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

In OpenTelemetry collector's log, you should see information similar to the following:

2024-02-18T17:14:03.825Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
     -> telemetry.sdk.language: Str(lua)
     -> telemetry.sdk.name: Str(opentelemetry-lua)
     -> telemetry.sdk.version: Str(0.1.1)
     -> hostname: Str(e34673e24631)
     -> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
    Trace ID       : fbd0a38d4ea4a128ff1a688197bc58b0
    Parent ID      :
    ID             : af3dc7642104748a
    Name           : GET /anything
    Kind           : Server
    Start time     : 2024-02-18 17:14:03.763244032 +0000 UTC
    End time       : 2024-02-18 17:14:03.920229888 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> net.host.name: Str(127.0.0.1)
     -> http.method: Str(GET)
     -> http.scheme: Str(http)
     -> http.target: Str(/anything)
     -> http.user_agent: Str(curl/7.64.1)
     -> apisix.route_id: Str(otel-tracing-route)
     -> apisix.route_name: Empty()
     -> http.route: Str(/anything)
     -> http.status_code: Int(200)
{"kind": "exporter", "data_type": "traces", "name": "debug"}

To visualize these traces, you can export your telemetry to backend Services, such as Zipkin and Prometheus. See exporters for more details.

Using Trace Variables in Logging#

The following example demonstrates how to configure the opentelemetry Plugin to set the following built-in variables, which can be used in logger Plugins or access logs:

opentelemetry_context_traceparent: trace parent ID
opentelemetry_trace_id: trace ID of the current span
opentelemetry_span_id: span ID of the current span

Configure the plugin metadata to set set_ngx_var as true:

curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry -H "X-API-KEY: $admin_key" -X PUT -d '
{
    "set_ngx_var": true
}'

Update the configuration file as below. You should customize the access log format to use the opentelemetry Plugin variables.

conf/config.yaml

nginx_config:
  http:
    enable_access_log: true
    access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
    access_log_format_escape: json

Reload APISIX for configuration changes to take effect.

You should see access log entries similar to the following when you generate requests:

{"time": "18/Feb/2024:15:09:00 +0000","opentelemetry_context_traceparent": "00-fbd0a38d4ea4a128ff1a688197bc58b0-8f4b9d9970a02629-01","opentelemetry_trace_id": "fbd0a38d4ea4a128ff1a688197bc58b0","opentelemetry_span_id": "af3dc7642104748a","remote_addr": "172.10.0.1"}

Description#

Configurations#

note

Attributes#

Examples#

Enable opentelemetry Plugin#

Send Traces to OpenTelemetry#

Using Trace Variables in Logging#

Enable `opentelemetry` Plugin#