Prometheus

Prometheus

Tips

# 安装
brew install prometheus
# 从源码编译安装
go get -v -u github.com/prometheus/prometheus/cmd/...
# 启动 默认前端页面 localhost:9090
prometheus --config.file ~/.config/prometheus.yml
# docker 启动
docker run \
-p 9090:9090 \
-v /etc/prometheus:/etc/prometheus \
prom/prometheus
# 或通过 brew 启动服务
# 如果没有 --storage.tsdb.path 可能出现权限问题
echo "--config.file $HOME/.config/prometheus.yml --storage.tsdb.path $HOME/.data/prometheus" > /usr/local/etc/prometheus.args
brew services start prometheus
# 错误日志
cat /usr/local/var/log/prometheus.err.log
# 正常日志
cat /usr/local/var/log/prometheus.log
# 管理接口
# --web.enable-admin-api 启用 /api/*/admin/ 相关接口
# --web.enable-lifecycle 启用 reload 和 quite
# 参考 https://prometheus.io/docs/operating/security/
prometheus --config.file ~/.config/prometheus.yml --web.enable-admin-api --web.enable-lifecycle
# brew 参数
echo "--config.file $HOME/.config/prometheus.yml --storage.tsdb.path $HOME/.data/prometheus --web.enable-admin-api --web.enable-lifecycle" > /usr/local/etc/prometheus.args
# 重启服务
brew services restart prometheus
# 重载配置
curl -X POST http://localhost:9090/-/reload
# 主机节点监控
brew install node_exporter
# 通过 service 启动
echo --web.listen-address :9101 > /usr/local/etc/node_exporter.args
brew services start node_exporter
# 直接启动
node_exporter --web.listen-address :9101
# Docker
# ===================
docker pull prom/prometheus
docker pull prom/alertmanager
# 需要挂载 /proc
docker pull prom/node-exporter
docker pull prom/blackbox-exporter
docker pull prom/container-exporter
docker pull prom/mysqld-exporter
# node_exporter 在容器中运行的不太好
go get github.com/prometheus/node_exporter
# 默认账号密码为 admin/admin
docker pull grafana/grafana
docker network create --subnet=172.18.0.0/16 mon-net
docker run --net mon-net --ip 172.18.0.10 -i -p 12000:3000 grafana/grafana
docker run --net mon-net --ip 172.18.0.20 -i -p 12001:9090 prom/prometheus
# http://docs.grafana.org/installation/docker/
docker run -d --restart always -v /etc/localtime:/etc/localtime:ro \
-p 12000:3000 \
-e "GF_SERVER_ROOT_URL=http://grafana.server.name" \
-e "GF_SECURITY_ADMIN_PASSWORD=secret" \
grafana/grafana

Config

# 全局配置
global:
# 抓取间隔,默认 1m
scrape_interval: 15s
# 抓取超时,默认 10s
scrape_timeout: 10s
# 计算规则间隔,默认 1m
evaluation_interval: 15s
# 告警配置
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# 周期性计算的规则文件
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# 抓取配置
scrape_configs:
# 任务名字,会添加一个 job=$job_name 的标签
- job_name: 'prometheus'
# 指标路径,默认 '/metrics'
metrics_path: '/metrics'
# 请求 schema,默认 'http'
scheme: 'http'
# 静态配置
static_configs:
# 抓取目标
- targets: ['localhost:9090']

服务发现

  • 支持配置
    • azure
    • consul - 服务 catalog
    • digitalocean
    • dockerswarm
    • dns - SVR 记录
    • ec2
    • openstack
    • file - 检测文件变化
      • 格式与 static_config 相同
    • gce
    • kubernetes
      • node、service、pod、endpoints、ingress
    • marathon
    • nerve
    • serverset
    • triton
  • mDNS
# _prometheus-http._tcp
# _prometheus-https._tcp
go get github.com/msiebuhr/prometheus-mdns-sd
#
prometheus-mdns-sd -out /etc/prometheus/mdns-sd.json
cat <<XML > /etc/avahi/services/node-exporter.service
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_prometheus-http._tcp</type>
<port>9100</port>
</service>
</service-group>
XML
# macOS
dns-sd -R "node_exporter metrics" _prometheus-http._tcp. . 9100 path=/metrics

集成

  • INTEGRATIONS
  • 支持读写的存储
    • Azure Data Explorer
    • Cortex
    • CrateDB
    • Google BigQuery
    • Google Cloud Spanner
    • InfluxDB
    • IRONdb
    • M3DB
    • MetricFire
    • PostgreSQL/TimescaleDB
    • QuasarDB
    • Splunk
    • TiKV
    • Thanos

Pushing

Proxy

scrape_configs:
- job_name: node
# 代理
proxy_url: http://proxy:8080/
static_configs:
- targets: ['client:9100'] # Presuming the FQDN of the client is "client".

label

  • __ 开头的为内部 label
  • __meta 可能由 Service Discovery 添加
  • __tmp 可由用户使用
LabelDesc
__address__目标地址
__schema__http/https
__name__标签名字
__param_target?target=
__param_module?module=
- job_name: "printer"
static_configs:
- targets:
- 192.168.1.2
metrics_path: /snmp
params:
module: [printer_mib]
relabel_configs:
# __param_target=__address__
- source_labels: [__address__]
target_label: __param_target
# instance=__param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
# snmp exporter 地址
replacement: 192.168.1.3:9116

FAQ

Push vs Pull