Skip to main content

Apache APISIX

caution
  • 默认 3 etcd 会占用非常多资源,非常不划算
  • apisix 依赖 etcd 容易出问题
  • apache/apisix
    • Apache-2.0, Lua+Nginx/openresty+etcd
    • 支持 Golang 插件
    • 支持协议: HTTP, GRPC, Dubbo, MQTT
    • 也可以存储在 yaml - 但不可以编辑
      • config_center 配置
  • Admin API
    • Auth X-API-KEY: admin-ley
    • PUT 为新增
    • /apisix/admin/<资源>
      • routes, stream_routes, services, consumers, upstreams, ssl, global_rules,
      • plugin_configs, plugin_metadata
    • 信息
      • /apisix/admin/plugins/list
      • /apisix/admin/plugins/{plugin_name}
  • 参考
# 官方建议自行构建
# APISIX_VERSION=2.9 make build-on-alpine-cn
# apache/apisix:2.9-alpine
# 带 build 逻辑的 compose
# https://github.com/apache/apisix-docker/raw/master/example/docker-compose-alpine.yml

# make build-all-in-one
docker run -d \
-p 9080:9080 -p 9091:9091 -p 2379:2379 \
-v $PWD/all-in-one/apisix/config.yaml:/usr/local/apisix/conf/config.yaml \
apache/apisix:whole

# make build-dashboard
docker run -d \
-p 19080:9080 -p 19091:9091 -p 12379:2379 -p 19000:9000 \
-v $(pwd)/all-in-one/apisix/config.yaml:/usr/local/apisix/conf/config.yaml \
-v $(pwd)/all-in-one/apisix-dashboard/conf.yaml:/usr/local/apisix-dashboard/conf/conf.yaml \
apache/apisix-dashboard:whole
prometheus port 9091
plugin_attr:
prometheus:
export_addr:
ip: '0.0.0.0'
port: 9091

基本概念

  • Route/路由
    • =uri+plugins+upstream/service_id
    • 路由条件:Host、Path、Method、表达式、请求参数、Cookie、Auth 信息
    • Weight-based traffic split
    • 插件
    • 支持 TCP、UDP 路由
    • 注意
      • path 默认完整匹配, / 只会匹配 /
  • Service/服务 - 路由集合
    • plugins+upstream
  • Upstream/上游
    • nodes
    • LB、健康检查、重试、超时
  • PubSub - 只支持 Kafka
  • xRPC - stream_routes - 实现 L4 代理转发
    • 内置 redis 协议 - 可植入错误
    • 可用于实现自定义协议
  • 外部插件
  • 预定义变量

服务发现

  • 通过 upstream.service_name 匹配使用, 发现的内容作为 nodes
  • 管理接口 /v1/discovery/{discovery_type}/dump
  • consule_kv
  • dns
  • eureka
  • nacos
  • tars
  • apisix/discovery/
  • kubernetes
    • 发现服务的 endpoints
discovery:
kubernetes:
service:
schema: https # https,http
host: ${KUBERNETES_SERVICE_HOST}
port: ${KUBERNETES_SERVICE_PORT}

# 如果在集群外运行 需要配置客户端信息
client:
token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
#token: |- # 静态 Token

# 选择生效的 namespace
namespace_selector:
# 匹配方式 equal, not_equal, match, not_match
# 值可以是数组
equal: default

# 通过 label 选择生效服务
label_selector: |-
first="a",second="b"

路由

{
"plugins": {
"key-auth": {},
// 限定针对 consumer 生效
"consumer-restriction": {
"blacklist": ["jack"]
}
},
"upstream": {
// grpc,grpcs
"schema": "http",
"nodes": {
"127.0.0.1:1980": 1
},
"type": "roundrobin"
},
"uri": "/hello"
}

插件

  • 无服务器架构
  • 可观测性
  • 其它协议
    • mqtt-proxy - 基于 client_id 负载

通用

  • batch-requests
  • redirect - 重定向
    http_to_https: false # 与 url,regex_uri 三选一
    url: '' # 目标地址
    regex_uri: [] # 正则替换
    ret_code: 302 # HTTP 状态码
    encode_uri: false
    append_query_string: false
  • echo - 回显结果 - 用于测试
    before_body: ''
    body: ''
    after_body: ''
    headers: {}
  • gzip
  • real-ip
  • server-info - 向 ETCD 上报服务信息
    • 在配置中启用 conf/config.yaml
    • report_ttl=36 - 单位秒
    • 可请求 http://127.0.0.1:9090/v1/server_info
  • ext-plugin-pre-req - 前置外部插件
    # 插件和它的配置
    conf: [{ 'name': 'ext-plugin-A', 'value': '{"enable":"feature"}' }]
    allow_degradation: false
  • ext-plugin-post-req - 后置外部插件

传输

  • response-rewrite
  • proxy-rewrite
  • grpc-transcode
    # 上传 PB POST {"content":""} /admin/proto/{id}
    # 合并 proto protoc --include_imports --descriptor_set_out=proto.pb proto/helloworld.proto
    prodo_id: ''
    service: ''
    method: ''
    deadline: 0
    # enum_as_name, enum_as_value
    # int64_as_number, int64_as_string, int64_as_hexstring
    # auto_default_values, no_default_values, use_default_values, use_default_metatable
    # enable_hooks, disable_hooks
    pb_option: []
  • grpc-web
  • fault-injection
  • mocking

身份验证

  • key-auth
    • Consumer - 定义 消费者的 Token
      • key
    • Route - 定义获取 key 的方式
      • header = apikey
      • query = apikey
      • hide_credentials = false - 是否透传给 upstream
  • jwt-auth
    • Consumer
      • key
  • basic-auth
    • Consumer
      • username
      • password
    • Router
      • hide_credentials
  • authz-keycloak - OIDC 认证逻辑+Keycloak 授权判断逻辑
# 🌟 推荐设置自动发现,减少配置量
discovery: # <endpoint>/.well-known/uma2-configuration
token_endpoint:
resource_registration_endpoint:
client_id:
client_secret:
grant_type: urn:ietf:params:oauth:grant-type:uma-ticket
policy_enforcement_mode: ENFORCING # PERMISSIVE
# 一般格式为 资源名#scope名
permissions: []
# 加载针对资源的权限 - 需要让插件能请求 token - 因此需要 client_id 和 client_secret
lazy_load_paths: false
http_method_as_scope: false # 一般配合 lazy_load_paths 使用
timeout: 3000
access_token_expires_in: 300
access_token_expires_leeway: 0
refresh_token_expires_in: 3600
refresh_token_expires_leeway: 0
ssl_verify: true
cache_ttl_seconds: 86400 # 24h - 缓存 发现配置
keepalive: true
keepalive_timeout: 60000
keepalive_pool: 5
# 默认返回 {"error_description":"not_authorized"}
access_denied_redirect_uri:
# 配置后可使用 POST username+password 获取 token
password_grant_token_generation_incoming_uri:

opa

请求

{
"type": "http",
"request": {
"scheme": "http",
"path": "/get",
"headers": {
"user-agent": "curl/7.68.0",
"accept": "*/*",
"host": "127.0.0.1:9080"
},
"query": {},
"port": 9080,
"method": "GET",
"host": "127.0.0.1"
},
"var": {
"timestamp": 1701234567,
"server_addr": "127.0.0.1",
"server_port": "9080",
"remote_port": "port",
"remote_addr": "ip address"
},
"route": {},
"service": {},
"consumer": {}
}

响应

{
"result": {
"allow": true,
"reason": "test",
"headers": {
"an": "header"
},
"status_code": 401
}
}

安全防护

  • cors
  • uri-blocker
  • ip-restriction
  • ua-restriction
  • referer-restriction
  • consumer-restriction
  • csrf
  • public-api - 暴露插件里的接口
    • uri
  • consumer-restriction - 限定 consumer
    # consumer_name - 限定 consumer 访问 service, route
    # service_id,route_id - 在 consumer 上定义, 限定访问 service,route
    type: consumer_name
    whitelist: []
    blacklist: []
    rejected_code: 403
    rejected_code: ""
    allowed_by_methods: [] # 允许的 HTTP 方法

流量控制

  • limit-req - leaky bucket
  • limit-conn - 限定连接数
  • limit-count - 限定时间范围内请求
  • proxy-cache
    cache_strategy: disk # memory
    cache_zone: disk_cache_one
    cache_key: ['$host', '$request_uri'] # ["$host", "$uri", "-cache-id"]
    cache_bypass: []
    cache_method: ['GET', 'HEAD']
    cache_http_status: [200, 301, 404]
    hide_cache_headers: false
    cache_control: false
    no_cache: []
    cache_ttl: 300
  • request-validation - JSON Schema 校验
  • proxy-mirror
  • api-breaker - 熔断
  • traffic-split
  • request-id
    header_name: X-Request-Id
    include_in_response: true
    algorithm: uuid # snowflake, nanoid
  • proxy-control
  • client-control
    max_body_size: 0 # 控制最大 body

GraphQL

  • 暴露变量用于路由
    • graphql_operation - query, mutation
    • graphql_name, graphql_root_fields
{
"methods": ["POST"],
"uri": "/_graphql",
"vars": [
["graphql_operation", "==", "query"],
["graphql_name", "==", "getRepo"],
["graphql_root_fields", "has", "owner"]
],
"upstream": {
"type": "roundrobin",
"nodes": {
"192.168.1.200:4000": 1
}
}
}

global

- name: real-ip
enable: true
config:
source: http_x_forwarded_for
recursive: true
trusted_addresses:
- 192.168.0.0/16
- 10.0.0.0/8
- name: request-id
enable: true
# https://apisix.apache.org/docs/apisix/plugins/request-id/
config:
header_name: X-Request-Id
algorithm: uuid
- name: response-rewrite
enable: true
# https://apisix.apache.org/docs/apisix/plugins/response-rewrite/
config:
headers:
remove: [Server]
- name: elasticsearch-logger
enable: true
# https://apisix.apache.org/docs/apisix/plugins/elasticsearch-logger/
config:
endpoint_addr: http://openobserve.data-system/api/wener
field:
index: apisix-log
type: collector
auth:
username: X
password: X
ssl_verify: false
#
timeout: 60
batch_max_size: 500
buffer_duration: 60
retry_delay: 1
max_retry_count: 3
inactive_timeout: 5

batch-processor

# 每批发送日志的最大条数
# 聚合的日志数量 - 注意配合 timeout
batch_max_size: 1000
# 刷新缓冲区的最大时间
# 小于 buffer_duration
inactive_timeout: 5
# 必须先处理批次中最旧条目的最长期限
buffer_duration: 60
# 从处理管道中移除之前的最大重试次数
max_retry_count: 0
# 延迟执行流程的秒数
retry_delay: 1
# 默认
KEY=edd1c9f034335f136f87ad84b625c8f1
curl -H "X-API-KEY: $KEY" 127.0.0.1:9180/apisix/admin/global_rules

log_format

{ "host": "$host", "@timestamp": "$time_iso8601", "client_ip": "$remote_addr" }
{
"request-id": {
"algorithm": "uuid",
"header_name": "X-Request-Id"
},
"response-rewrite": {
"headers": {
"remove": ["Server"]
}
},
"real-ip": {
"recursive": true,
"source": "http_x_forwarded_for",
"trusted_addresses": ["192.168.0.0/16", "10.0.0.0/8"]
}
}

serverless

  • serverless-pre-function
  • serverless-post-function
  • phase - rewrite, access, header_filter, body_filter, log, before_proxy
  • functions
local count = 1
return function(conf,ctx)
count = count + 1
ngx.say(count)
end

FAQ

failed to process entries: 20: unable to get local issuer certificate, context: ngx.timer

ssl_verify: false

配置 CA

apisix:
ssl:
ssl_trusted_certificate: /path/to/certs/ca-certificates.crt

removing batch processor stale object

an upstream response is buffered to a temporary file

超出 proxy_buffer_size 后写入临时文件

没问题

proxy_buffer_size 4k; # 默认 4k or 8k
# proxy_buffers number size;
proxy_buffers 8 4k; # 默认 4k or 8k

proxy_max_temp_file_size 1024m

tls: failed to verify certificate: x509: certificate signed by unknown authority

2023-12-21T18:50:39Z	error	apisix/route.go:90	failed to list routes: Get "https://apisix-control-plane:9180/apisix/admin/routes": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "apisix-ca")
2023-12-21T18:50:39Z error apisix/cluster.go:298 failed to list routes in APISIX: Get "https://apisix-control-plane:9180/apisix/admin/routes": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "apisix-ca")
2023-12-21T18:50:39Z error apisix/plugin.go:46 failed to list plugins' names: Get "https://apisix-control-plane:9180/apisix/admin/plugins?all=true": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "apisix-ca")
2023-12-21T18:50:39Z error apisix/cluster.go:483 failed to list plugin names in APISIX: Get "https://apisix-control-plane:9180/apisix/admin/plugins?all=true": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "apisix-ca")
2023-12-21T18:50:39Z error apisix/cluster.go:446 failed to sync schema: Get "https://apisix-control-plane:9180/apisix/admin/plugins?all=true": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "apisix-ca")
2023-12-21T18:50:41Z error apisix/route.go:90 failed to list routes: Get "https://apisix-control-plane:9180/apisix/admin/routes": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "apisix-ca")