ua-restriction

Description

The ua-restriction Plugin supports restricting access to upstream resources through either configuring an allowlist or denylist of user agents. A common use case is to prevent web crawlers from overloading the upstream resources and causing service degradation.

Attributes

Name	Type	Required	Default	Valid values	Description
bypass_missing	boolean	False	false		If true, bypass the user agent restriction check when the User-Agent header is missing.
allowlist	array[string]	False			List of user agents to allow. Support regular expressions. At least one of the allowlist and denylist should be configured, but they cannot be configured at the same time.
denylist	array[string]	False			List of user agents to deny. Support regular expressions. At least one of the allowlist and denylist should be configured, but they cannot be configured at the same time.
message	string	False	"Not allowed"		Message returned when the user agent is denied access.

Examples

The examples below demonstrate how you can configure ua-restriction for different scenarios.

note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

Reject Web Crawlers and Customize Error Message

The following example demonstrates how you can configure the Plugin to fend off unwanted web crawlers and customize the rejection message.

Create a Route and configure the Plugin to block specific crawlers from accessing resources with a customized message:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "ua-restriction-route",
    "uri": "/anything",
    "plugins": {
      "ua-restriction": {
        "bypass_missing": false,
        "denylist": [
          "(Baiduspider)/(\\d+)\\.(\\d+)",
          "bad-bot-1"
        ],
        "message": "Access denied"
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the Route:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

Send another request to the Route with a disallowed user agent:

curl -i "http://127.0.0.1:9080/anything" -H 'User-Agent: Baiduspider/5.0'

You should receive an HTTP/1.1 403 Forbidden response with the following message:

{"message":"Access denied"}

Bypass UA Restriction Checks

The following example demonstrates how to configure the Plugin to allow requests of a specific user agent to bypass the UA restriction.

Create a Route as such:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "ua-restriction-route",
    "uri": "/anything",
    "plugins": {
      "ua-restriction": {
        "bypass_missing": true,
        "allowlist": [
          "good-bot-1"
        ],
        "message": "Access denied"
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the Route without modifying the user agent:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 403 Forbidden response with the following message:

{"message":"Access denied"}

Send another request to the Route with an empty user agent:

curl -i "http://127.0.0.1:9080/anything" -H 'User-Agent: '

You should receive an HTTP/1.1 200 OK response.