Version: 3.10

opentelemetry

Description#

The opentelemetry Plugin can be used to report tracing data according to the OpenTelemetry specification.

The Plugin only supports binary-encoded OLTP over HTTP.

Attributes#

Name	Type	Required	Default	Valid values	Description
sampler	object	False			Sampling configuration.
sampler.name	string	False	always_off	["always_on", "always_off", "trace_id_ratio", "parent_base"]	Sampling strategy. `always_on`: always samples, `always_off`: never samples, `trace_id_ratio`: random sampling result based on given sampling probability, `parent_base`: use parent decision if given, else determined by the root sampler.
sampler.options	object	False		{fraction = 0, root = {name = "always_off"}}	Parameters for sampling strategy.
sampler.options.fraction	number	False	0	[0, 1]	Sampling probability for `trace_id_ratio`.
sampler.options.root	object	False	{name = "always_off", options = {fraction = 0}}		Root sampler for `parent_base` strategy.
sampler.options.root.name	string	False	always_off	["always_on", "always_off", "trace_id_ratio"]	Root sampling strategy.
sampler.options.root.options	object	False	{fraction = 0}		Root sampling strategy parameters.
sampler.options.root.options.fraction	number	False	0	[0, 1]	Root sampling probability for `trace_id_ratio`.
additional_attributes	array[string]	False			Additional attributes appended to the trace span. Support built-in NGINX or APISIX variables in values, such as `http_header` or `route_id`.
additional_header_prefix_attributes	array[string]	False			Headers or header prefixes appended to the trace span's attributes. For example, use `x-my-header"` or `x-my-headers-*` to include all headers with the prefix `x-my-headers-`.

Configuring the collector#

You can set up the collector by configuring it in you configuration file (conf/config.yaml):

Name	Type	Default	Description
trace_id_source	enum	x-request-id	Source of the trace ID. Valid values are `random` or `x-request-id`. When set to `x-request-id`, the value of the `x-request-id` header will be used as trace ID. Make sure that it matches the regex pattern `[0-9a-f]{32}`.
resource	object		Additional resource appended to the trace.
collector	object	{address = "127.0.0.1:4318", request_timeout = 3}	OpenTelemetry Collector configuration.
collector.address	string	127.0.0.1:4318	Collector address. If the collector serves on https, use https://127.0.0.1:4318 as the address.
collector.request_timeout	integer	3	Report request timeout in seconds.
collector.request_headers	object		Report request HTTP headers.
batch_span_processor	object		Trace span processor.
batch_span_processor.drop_on_queue_full	boolean	false	When set to `true`, drops the span when queue is full. Otherwise, force process batches.
batch_span_processor.max_queue_size	integer	1024	Maximum queue size for buffering spans for delayed processing.
batch_span_processor.batch_timeout	number	2	Maximum time in seconds for constructing a batch.
batch_span_processor.max_export_batch_size	integer	16	Maximum number of spans to process in a single batch.
batch_span_processor.inactive_timeout	number	1	Time interval in seconds between processing batches.

note

If you find a bad argument #1 to '?' (invalid value) error triggered by the hex2bytes function in error log, it's essential to verify if your traceId matches the specified regex pattern [0-9a-f]{32}, as required by opentelemetry's traceId format.

For instance, a possible scenario occurs when the plugin attribute trace_id_source is configured as x-request-id, and requests include an x-request-id header generated by Envoy. Envoy typically uses a UUID to create this header by default. When the opentelemetry plugin adopts this UUID as the traceId, the presence of hyphens in the UUID can cause issues. Since the UUID format with hyphens does not comply with the expected traceId format, it results in errors when attempting to push traces to the collector.

You can configure these as shown below in your configuration file (conf/config.yaml):

conf/config.yaml

plugin_attr:
  opentelemetry:
    resource:
      service.name: APISIX
      tenant.id: business_id
    collector:
      address: 192.168.8.211:4318
      request_timeout: 3
      request_headers:
        foo: bar
    batch_span_processor:
      drop_on_queue_full: false
      max_queue_size: 6
      batch_timeout: 2
      inactive_timeout: 1
      max_export_batch_size: 2

Variables#

The following nginx variables are set by OpenTelemetry:

opentelemetry_context_traceparent - W3C trace context, e.g.: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
opentelemetry_trace_id - Trace Id of the current span
opentelemetry_span_id - Span Id of the current span

How to use variables? you have to add it to your configuration file (conf/config.yaml):

conf/config.yaml

http:
    enable_access_log: true
    access_log: "/dev/stdout"
    access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr","uri": "$uri"}'
    access_log_format_escape: json
plugins:
  - opentelemetry
plugin_attr:
  opentelemetry:
    set_ngx_var: true

Enable Plugin#

To enable the Plugin, you have to add it to your configuration file (conf/config.yaml):

conf/config.yaml

plugins:
  - ...
  - opentelemetry

Now, you can enable the Plugin on a specific Route:

note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

curl http://127.0.0.1:9180/apisix/admin/routes/1  -H "X-API-KEY: $admin_key" -X PUT -d '
{
    "methods": ["GET"],
    "uris": [
        "/uid/*"
    ],
    "plugins": {
        "opentelemetry": {
            "sampler": {
                "name": "always_on"
            }
        }
    },
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:1980": 1
        }
    }
}'

Delete Plugin#

To remove the opentelemetry Plugin, you can delete the corresponding JSON configuration from the Plugin configuration. APISIX will automatically reload and you do not have to restart for this to take effect.

curl http://127.0.0.1:9180/apisix/admin/routes/1  -H "X-API-KEY: $admin_key" -X PUT -d '
{
    "methods": ["GET"],
    "uris": [
        "/uid/*"
    ],
    "plugins": {
    },
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:1980": 1
        }
    }
}'