Skip to main content
Version: Next

opentelemetry

Description#

The opentelemetry Plugin can be used to report tracing data according to the OpenTelemetry specification.

The Plugin only supports binary-encoded OLTP over HTTP.

Attributes#

NameTypeRequiredDefaultValid valuesDescription
samplerobjectFalseSampling configuration.
sampler.namestringFalsealways_off["always_on", "always_off", "trace_id_ratio", "parent_base"]Sampling strategy. always_on: always samples, always_off: never samples, trace_id_ratio: random sampling result based on given sampling probability, parent_base: use parent decision if given, else determined by the root sampler.
sampler.optionsobjectFalse{fraction = 0, root = {name = "always_off"}}Parameters for sampling strategy.
sampler.options.fractionnumberFalse0[0, 1]Sampling probability for trace_id_ratio.
sampler.options.rootobjectFalse{name = "always_off", options = {fraction = 0}}Root sampler for parent_base strategy.
sampler.options.root.namestringFalsealways_off["always_on", "always_off", "trace_id_ratio"]Root sampling strategy.
sampler.options.root.optionsobjectFalse{fraction = 0}Root sampling strategy parameters.
sampler.options.root.options.fractionnumberFalse0[0, 1]Root sampling probability for trace_id_ratio.
additional_attributesarray[string]FalseAdditional attributes appended to the trace span. Support built-in NGINX or APISIX variables in values, such as http_header or route_id.
additional_header_prefix_attributesarray[string]FalseHeaders or header prefixes appended to the trace span's attributes. For example, use x-my-header" or x-my-headers-* to include all headers with the prefix x-my-headers-.

Configuring the collector#

You can set up the collector by configuring it in you configuration file (conf/config.yaml):

NameTypeDefaultDescription
trace_id_sourceenumx-request-idSource of the trace ID. Valid values are random or x-request-id. When set to x-request-id, the value of the x-request-id header will be used as trace ID. Make sure that it matches the regex pattern [0-9a-f]{32}.
resourceobjectAdditional resource appended to the trace.
collectorobject{address = "127.0.0.1:4318", request_timeout = 3}OpenTelemetry Collector configuration.
collector.addressstring127.0.0.1:4318Collector address. If the collector serves on https, use https://127.0.0.1:4318 as the address.
collector.request_timeoutinteger3Report request timeout in seconds.
collector.request_headersobjectReport request HTTP headers.
batch_span_processorobjectTrace span processor.
batch_span_processor.drop_on_queue_fullbooleanfalseWhen set to true, drops the span when queue is full. Otherwise, force process batches.
batch_span_processor.max_queue_sizeinteger1024Maximum queue size for buffering spans for delayed processing.
batch_span_processor.batch_timeoutnumber2Maximum time in seconds for constructing a batch.
batch_span_processor.max_export_batch_sizeinteger16Maximum number of spans to process in a single batch.
batch_span_processor.inactive_timeoutnumber1Time interval in seconds between processing batches.
note

If you find a bad argument #1 to '?' (invalid value) error triggered by the hex2bytes function in error log, it's essential to verify if your traceId matches the specified regex pattern [0-9a-f]{32}, as required by opentelemetry's traceId format.

For instance, a possible scenario occurs when the plugin attribute trace_id_source is configured as x-request-id, and requests include an x-request-id header generated by Envoy. Envoy typically uses a UUID to create this header by default. When the opentelemetry plugin adopts this UUID as the traceId, the presence of hyphens in the UUID can cause issues. Since the UUID format with hyphens does not comply with the expected traceId format, it results in errors when attempting to push traces to the collector.

You can configure these as shown below in your configuration file (conf/config.yaml):

conf/config.yaml
plugin_attr:
opentelemetry:
resource:
service.name: APISIX
tenant.id: business_id
collector:
address: 192.168.8.211:4318
request_timeout: 3
request_headers:
foo: bar
batch_span_processor:
drop_on_queue_full: false
max_queue_size: 6
batch_timeout: 2
inactive_timeout: 1
max_export_batch_size: 2

Variables#

The following nginx variables are set by OpenTelemetry:

  • opentelemetry_context_traceparent - W3C trace context, e.g.: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
  • opentelemetry_trace_id - Trace Id of the current span
  • opentelemetry_span_id - Span Id of the current span

How to use variables? you have to add it to your configuration file (conf/config.yaml):

conf/config.yaml
http:
enable_access_log: true
access_log: "/dev/stdout"
access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr","uri": "$uri"}'
access_log_format_escape: json
plugins:
- opentelemetry
plugin_attr:
opentelemetry:
set_ngx_var: true

Enable Plugin#

To enable the Plugin, you have to add it to your configuration file (conf/config.yaml):

conf/config.yaml
plugins:
- ...
- opentelemetry

Now, you can enable the Plugin on a specific Route:

note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
curl http://127.0.0.1:9180/apisix/admin/routes/1  -H "X-API-KEY: $admin_key" -X PUT -d '
{
"methods": ["GET"],
"uris": [
"/uid/*"
],
"plugins": {
"opentelemetry": {
"sampler": {
"name": "always_on"
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
}
}
}'

Delete Plugin#

To remove the opentelemetry Plugin, you can delete the corresponding JSON configuration from the Plugin configuration. APISIX will automatically reload and you do not have to restart for this to take effect.

curl http://127.0.0.1:9180/apisix/admin/routes/1  -H "X-API-KEY: $admin_key" -X PUT -d '
{
"methods": ["GET"],
"uris": [
"/uid/*"
],
"plugins": {
},
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
}
}
}'