Skip to main content
Version: 2.13

clickhouse-logger

Description#

clickhouse-logger is a plugin which push Log data requests to clickhouse.

Attributes#

NameTypeRequirementDefaultValidDescription
endpoint_addrstringrequiredThe clickhouse endpoint.
databasestringrequiredThe DB name to store log.
logtablestringrequiredThe table name.
userstringrequiredclickhouse user.
passwordstringrequiredclickhouse password.
timeoutintegeroptional3[1,...]Time to keep the connection alive after sending a request.
namestringoptional"clickhouse logger"A unique identifier to identity the logger.
ssl_verifybooleanoptionaltrue[true,false]verify ssl.

The plugin supports the use of batch processors to aggregate and process entries(logs/data) in a batch. This avoids frequent data submissions by the plugin, which by default the batch processor submits data every 5 seconds or when the data in the queue reaches 1000. For information or custom batch processor parameter settings, see Batch-Processor configuration section.

How To Enable#

The following is an example of how to enable the clickhouse-logger for a specific route.

curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"plugins": {
"clickhouse-logger": {
"user": "default",
"password": "a",
"database": "default",
"logtable": "test",
"endpoint_addr": "http://127.0.0.1:8123"
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
}
},
"uri": "/hello"
}'

Test Plugin#

success:

$ curl -i http://127.0.0.1:9080/hello
HTTP/1.1 200 OK
...
hello, world

Metadata#

NameTypeRequirementDefaultValidDescription
log_formatobjectoptional{"host": "$host", "@timestamp": "$time_iso8601", "client_ip": "$remote_addr"}Log format declared as key value pair in JSON format. Only string is supported in the value part. If the value starts with $, it means to get APISIX variable or Nginx variable.

Note that the metadata configuration is applied in global scope, which means it will take effect on all Route or Service which use clickhouse-logger plugin.

Example#

curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/clickhouse-logger -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"log_format": {
"host": "$host",
"@timestamp": "$time_iso8601",
"client_ip": "$remote_addr"
}
}'

create clickhouse log table

CREATE TABLE default.test (
`host` String,
`client_ip` String,
`route_id` String,
`@timestamp` String,
PRIMARY KEY(`@timestamp`)
) ENGINE = MergeTree()

On clickhouse run select * from default.test;, will got below row:

┌─host──────┬─client_ip─┬─route_id─┬─@timestamp────────────────┐
127.0.0.1 │ 127.0.0.1 │ 12022-01-17T10:03:10+08:00 │
└───────────┴───────────┴──────────┴───────────────────────────┘

Disable Plugin#

Remove the corresponding json configuration in the plugin configuration to disable the clickhouse-logger. APISIX plugins are hot-reloaded, therefore no need to restart APISIX.

$ curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/hello",
"plugins": {},
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
}
}
}'