Version: Next

Kubernetes

Summary#

The Kubernetes service discovery List-Watch real-time changes of Endpoints resources, then store theirs value into ngx.shared.DICT.

Discovery also provides a node query interface in accordance with the APISIX Discovery Specification.

How To Use#

Kubernetes service discovery both support single-cluster and multi-cluster modes, applicable to the case where the service is distributed in single or multiple Kubernetes clusters.

Single-Cluster Mode Configuration#

A detailed configuration for single-cluster mode Kubernetes service discovery is as follows:

discovery:
  kubernetes:
    service:
      # apiserver schema, options [http, https]
      schema: https #default https

      # apiserver host, options [ipv4, ipv6, domain, environment variable]
      host: ${KUBERNETES_SERVICE_HOST} #default ${KUBERNETES_SERVICE_HOST}

      # apiserver port, options [port number, environment variable]
      port: ${KUBERNETES_SERVICE_PORT}  #default ${KUBERNETES_SERVICE_PORT}

    client:
      # serviceaccount token or token_file
      token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      #token: |-
       # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif
       # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI

    default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0

    # kubernetes discovery support namespace_selector
    # you can use one of [equal, not_equal, match, not_match] filter namespace
    namespace_selector:
      # only save endpoints with namespace equal default
      equal: default

      # only save endpoints with namespace not equal default
      #not_equal: default

      # only save endpoints with namespace match one of [default, ^my-[a-z]+$]
      #match:
       #- default
       #- ^my-[a-z]+$

      # only save endpoints with namespace not match one of [default, ^my-[a-z]+$ ]
      #not_match:
       #- default
       #- ^my-[a-z]+$

    # kubernetes discovery support label_selector
    # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
    label_selector: |-
      first="a",second="b"

    # reserved lua shared memory size,1m memory can store about 1000 pieces of endpoint
    shared_size: 64m #default 64m

    # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints
    watch_endpoint_slices: false #default false

If the Kubernetes service discovery runs inside a pod, you can use minimal configuration:

discovery:
  kubernetes: { }

If the Kubernetes service discovery runs outside a pod, you need to create or select a specified ServiceAccount, then get its token value, and use following configuration:

discovery:
  kubernetes:
    service:
      schema: https
      host: # enter apiserver host value here
      port: # enter apiserver port value here
    client:
      token: # enter serviceaccount token value here
      #token_file: # enter file path here

Single-Cluster Mode Query Interface#

The Kubernetes service discovery provides a query interface in accordance with the APISIX Discovery Specification.

function: nodes(service_name)

description: nodes() function attempts to look up the ngx.shared.DICT for nodes corresponding to servicename, \ service_name should match pattern: [namespace]/[name]:[portName]_

namespace: The namespace where the Kubernetes endpoints is located
name: The name of the Kubernetes endpoints
portName: The ports.name value in the Kubernetes endpoints, if there is no ports.name, use targetPort, port instead. If ports.name exists, then port number cannot be used.

return value: if the Kubernetes endpoints value is as follows:

apiVersion: v1
kind: Endpoints
metadata:
  name: plat-dev
  namespace: default
subsets:
  - addresses:
      - ip: "10.5.10.109"
      - ip: "10.5.10.110"
    ports:
      - port: 3306
        name: port

a nodes("default/plat-dev:port") call will get follow result:

 {
     {
         host="10.5.10.109",
         port= 3306,
         weight= 50,
     },
     {
         host="10.5.10.110",
         port= 3306,
         weight= 50,
     },
 }

Multi-Cluster Mode Configuration#

A detailed configuration for multi-cluster mode Kubernetes service discovery is as follows:

discovery:
  kubernetes:
  - id: release  # a custom name refer to the cluster, pattern ^[a-z0-9]{1,8}
    service:
      # apiserver schema, options [http, https]
      schema: https #default https

      # apiserver host, options [ipv4, ipv6, domain, environment variable]
      host: "1.cluster.com"

      # apiserver port, options [port number, environment variable]
      port: "6443"

    client:
      # serviceaccount token or token_file
      token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      #token: |-
       # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif
       # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI

    default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0

    # kubernetes discovery support namespace_selector
    # you can use one of [equal, not_equal, match, not_match] filter namespace
    namespace_selector:
      # only save endpoints with namespace equal default
      equal: default

      # only save endpoints with namespace not equal default
      #not_equal: default

      # only save endpoints with namespace match one of [default, ^my-[a-z]+$]
      #match:
       #- default
       #- ^my-[a-z]+$

      # only save endpoints with namespace not match one of [default, ^my-[a-z]+$]
      #not_match:
       #- default
       #- ^my-[a-z]+$

    # kubernetes discovery support label_selector
    # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
    label_selector: |-
      first="a",second="b"

    # reserved lua shared memory size,1m memory can store about 1000 pieces of endpoint
    shared_size: 64m #default 64m

    # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints
    watch_endpoint_slices: false #default false

Multi-Kubernetes service discovery does not fill default values for service and client fields, you need to fill them according to the cluster configuration.

Multi-Cluster Mode Query Interface#

The Kubernetes service discovery provides a query interface in accordance with the APISIX Discovery Specification.

function: nodes(service_name)

description: nodes() function attempts to look up the ngx.shared.DICT for nodes corresponding to servicename, \ service_name should match pattern: [id]/[namespace]/[name]:[portName]_

id: value defined in service discovery configuration
namespace: The namespace where the Kubernetes endpoints is located
name: The name of the Kubernetes endpoints
portName: The ports.name value in the Kubernetes endpoints, if there is no ports.name, use targetPort, port instead. If ports.name exists, then port number cannot be used.

return value: if the Kubernetes endpoints value is as follows:

apiVersion: v1
kind: Endpoints
metadata:
  name: plat-dev
  namespace: default
subsets:
  - addresses:
      - ip: "10.5.10.109"
      - ip: "10.5.10.110"
    ports:
      - port: 3306
        name: port

a nodes("release/default/plat-dev:port") call will get follow result:

 {
     {
         host="10.5.10.109",
         port= 3306,
         weight= 50,
     },
     {
         host="10.5.10.110",
         port= 3306,
         weight= 50,
     },
 }

Select Clusters with `cluster_ids`#

In multi-cluster mode, an upstream can get its nodes from several clusters at once by listing their ids in discovery_args.cluster_ids. The service_name then uses the single-cluster pattern [namespace]/[name]:[portName], without the id prefix:

{
    "type": "roundrobin",
    "discovery_type": "kubernetes",
    "service_name": "default/plat-dev:port",
    "discovery_args": {
        "cluster_ids": ["release", "staging"]
    }
}

The upstream nodes are the union of the matching endpoints in the listed clusters, and they follow the endpoint changes in those clusters. When the same host:port is found in more than one listed cluster, it is used once.

cluster_ids is a non-empty array of unique strings. It only applies to multi-cluster mode. The Admin API rejects cluster_ids when Kubernetes service discovery is configured in single-cluster mode. If such an upstream reaches the data plane in another way, it gets no nodes and an error is logged.
Clusters that are not listed, including clusters added to the configuration later, never contribute nodes.
When cluster_ids is set, service_name must match [namespace]/[name]:[portName], so it cannot carry an id prefix. Any other value is rejected, and the error shows the service_name value.
The Admin API rejects an id that is not defined in the Kubernetes service discovery configuration of the APISIX instance that serves the Admin API, and the error lists all unknown ids. When a configuration with an unknown id still reaches the data plane, for example because it was written before a cluster was removed from the configuration, or because it comes from a standalone configuration file, the unknown ids are skipped and a warning that lists them is logged.
When none of the listed clusters has matching endpoints, the upstream has no valid nodes. Nodes from other clusters are never used instead.

Without cluster_ids, the query interface described above is unchanged.

Q&A#

Q: Why only support configuration token to access Kubernetes APIServer?

A: Usually, we will use three ways to complete the authentication of Kubernetes APIServer:

mTLS
Token
Basic authentication

Because lua-resty-http does not currently support mTLS, and basic authentication is not recommended, so currently only the token authentication method is implemented.

Q: APISIX inherits Nginx's multiple process model, does it mean that each nginx worker process will List-Watch kubernetes endpoints resources?

A: The Kubernetes service discovery only uses privileged processes to List-Watch Kubernetes endpoints resources, then store theirs value into ngx.shared.DICT, worker processes get results by querying ngx.shared.DICT.

Q: What permissions do ServiceAccount require?

A: ServiceAccount requires the permissions of cluster-level [ get, list, watch ] endpoints and endpointslices resources, the declarative definition is as follows:

kind: ServiceAccount
apiVersion: v1
metadata:
 name: apisix-test
 namespace: default
---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: apisix-test
rules:
  - apiGroups: [ "" ]
    resources: [ endpoints]
    verbs: [ get,list,watch ]
  - apiGroups: [ "discovery.k8s.io" ]
    resources: [ endpointslices ]
    verbs: [ get,list,watch ]
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: apisix-test
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: apisix-test
subjects:
 - kind: ServiceAccount
   name: apisix-test
   namespace: default

Q: How to get ServiceAccount token value?

A: Assume your ServiceAccount located in namespace apisix and name is Kubernetes-discovery, you can use the following steps to get token value.

Get secret name. You can execute the following command, the output of the first column is the secret name we want:
```
kubectl -n apisix get secrets | grep kubernetes-discovery
```
Get token value. Assume secret resources name is kubernetes-discovery-token-c64cv, you can execute the following command, the output is the service account token value we want:
```
kubectl -n apisix get secret kubernetes-discovery-token-c64cv -o jsonpath={.data.token} | base64 -d
```

Debugging API#

It also offers control api for debugging.

Memory Dump API#

To query/list the nodes discovered by kubernetes discovery, you can query the /v1/discovery/kubernetes/dump control API endpoint like so:

GET /v1/discovery/kubernetes/dump

Which will yield the following response:

{
  "endpoints": [
    {
      "endpoints": [
        {
          "value": "{\"https\":[{\"host\":\"172.18.164.170\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":6443,\"weight\":50}]}",
          "name": "default/kubernetes"
        },
        {
          "value": "{\"metrics\":[{\"host\":\"172.18.164.170\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":2379,\"weight\":50}]}",
          "name": "kube-system/etcd"
        },
        {
          "value": "{\"http-85\":[{\"host\":\"172.64.89.2\",\"port\":85,\"weight\":50}]}",
          "name": "test-ws/testing"
        }
      ],
      "id": "first"
    }
  ],
  "config": [
    {
      "default_weight": 50,
      "id": "first",
      "client": {
        "token": "xxx"
      },
      "service": {
        "host": "172.18.164.170",
        "port": "6443",
        "schema": "https"
      },
      "shared_size": "64m"
    }
  ]
}