Skip to main content

vector

  • vectordotdev/vector
    • MPL-2.0, Rust
    • Datadog 开源
    • observability data pipeline
    • logs, metrics, traces(WIP) - observability 采集处理全家桶
  • Sources
    • Prometheus RemoteWrite/Scrap
    • s3, fluent, logstash
    • Kubernetes_logs, docker_logs, nginx_metrics, postgresql_metrics, statsd
    • host_metrics, journald, exec, file, syslog, socket, stdin
    • nats
    • internal_logs, internal_metrics
    • vector
    • dnstap
  • Transforms
  • Sinks
    • S3, ClickHouse, Elastic, Loki, InfluxDB, Prometheus RW/Export, Statsd
    • Redis, Plusar, Nats, Kafka
    • Vector
  • 角色
    • agent - 采集
      • source -> sink
      • source 通常为 file, kubernetes_logs, host_metrics
    • aggregator - 聚合
      • source 通常为 vector, syslog, statsd, fluent
      • transform
    • sidecar - 应用
issues

  • stops watching logs from new pods #8616

  • opentelemetry source & sink #1444
# macOS
brew tap vectordotdev/brew
brew install vector

# https://github.com/vectordotdev/vector/releases

Sources & Sinks

  • kubernetes_logs
  • host_metrics
  • internal_metrics
  • prometheus_scrape
  • Metrics
    • prometheus_exporter - Prometheus 的 /metrics
    • prometheus_remote_write
    • statsd
  • console - 输出日志到 stdout - debug 用
  • 通用
    • vector
    • file
    • socket
    • redis
    • websocket
    • nats
    • kafka
    • pulsar

配置

# data_dir = "/var/lib/vector"

[sources.dummy_logs]
type = "generator"
format = "syslog"
interval = 1

# ector Remap Language https://vector.dev/docs/reference/vrl/
[transforms.parse_logs]
type = "remap"
inputs = ["dummy_logs"]
source = '
. = parse_syslog!(string!(.message))
'

# Print parsed logs to stdout
[sinks.print]
type = "console"
inputs = ["parse_logs"]
encoding.codec = "json"

# GraphQL API http://localhost:8686
# vector top
[api]
enabled = true
address = "127.0.0.1:8686"
  • /etc/vector/

部署

PortName
5044logstash
6000vector
8080splunk-hec
8125statsd
8282datadog-agent
9000syslog
9090prom-exporter
24224fluent

Agent

  • 收集 host 主机上的 k8s 日志
  • 收集 host metrics 暴露为 prometheus_exporter
  • 作为 DaemonSet
volumefrom hostto path
data/var/lib/vector/vector-data-dir
config/etc/vector/
var-log/var/log/var/log/
var-lib/var/lib/var/lib/
procfs/proc/host/proc
sysfs/sys/host/sys
  • /var/lib/vector
    • 用来存储状态
    • 例如 /var/lib/vector/kubernetes_logs/checkpoints.json
data_dir: /vector-data-dir
api:
enabled: true
address: 127.0.0.1:8686
playground: false
sources:
kubernetes_logs:
type: kubernetes_logs
host_metrics:
filesystem:
devices:
excludes: [binfmt_misc]
filesystems:
excludes: [binfmt_misc]
mountPoints:
excludes: ['*/proc/sys/fs/binfmt_misc']
type: host_metrics
internal_metrics:
type: internal_metrics
sinks:
prom_exporter:
type: prometheus_exporter
inputs: [host_metrics, internal_metrics]
address: 0.0.0.0:9090
stdout:
type: console
inputs: [kubernetes_logs]
encoding:
codec: json

file

kubernetes_logs

排除采集

vector.dev/exclude: 'true'

多行

out:
type: reduce
inputs:
- log
group_by:
- kubernetes_container_id
#- container_id
merge_strategies:
message: concat_newline
starts_when: match(string!(.message), r'^[^\s]')
#starts_when: match(string!(.message) , r'^[^}$]')

More than one file has the same fingerprint

syslog