Skip to main content

Fluentbit

  • fluentbit
  • fluent/fluent-bit
  • Build Options
  • 数据处理流程
    • input - -i, --input
    • parser - -R, --parser
    • fliter - -F, --fliter
    • buffer
      • -b --storage_path=PATH
      • -f, --flush=SECONDS
    • router
      • 匹配
        • -m, --match - 等同于 -p match=abc
        • -t, --tag - 设置 tag,等同于 -p tag=abc
        • -p, --prop="A=B" - 设置插件属性
    • output - 多个
caution
# macOS
brew install fluent-bit
# AlpineLinux
apk add fluent-bit -X https://mirrors.aliyun.com/alpine/edge/testing/

# 读取 kernel 消息,输出到 stdout
fluent-bit -i kmsg -t kernel -o stdout -m '*'
# Windows Event Log 输出到 stdout
fluent-bit -i winlog -p 'channels=Setup,Windows PowerShell' -o stdout

# 监听 MQTT - 0.0.0.0:1883
fluent-bit -i mqtt -t data -o stdout -m '*'
# 发送 MQTT
mosquitto_pub -m '{"key1": 123, "key2": 456}' -t some/topic
# Listen=0.0.0.0
# macOS /usr/local/opt/fluent-bit/etc/fluent-bit/parsers.conf
# alpinelinux
fluent-bit -R /usr/local/opt/fluent-bit/etc/fluent-bit/parsers.conf -i syslog -p path=/tmp/flb_syslog -o stdout
logger -u /tmp/flb_syslog my_ident my_message

# syslog
# rfc3164 - 废弃的 BSD syslog 协议
# rfc5424 - 新的格式
fluent-bit -R /etc/fluent-bit/parsers.conf \
-i syslog -p mode=tcp -p listen=0.0.0.0 -p port=5140 -o stdout
# 日志
logger -T -n 127.0.0.1 -P 5140 wener

配置

# 引入文件
@INCLUDE somefile.conf
@INCLUDE input_*.conf

[SERVICE]
# 一个段落缩进 4 个空格
# 可以存在相同名字 key

# seconds.nanoseconds
Flush 5
Daemon off
Log_Level debug

[INPUT]
Name cpu
Tag my_cpu

[FILTER]
Name stdout
Match *​

[OUTPUT]
Name stdout
Match my*cpu

[OUTPUT]
# 可以使用环境变量
Name ${MY_OUTPUT}
Match *

# 直接设置变量
@SET KEY=VAL

# 上游配置
[UPSTREAM]
name forward-balancing

[NODE]
name node-1
host 127.0.0.1
port 43000

[NODE]
name node-2
host 127.0.0.1
port 44000

[NODE]
name node-3
host 127.0.0.1
port 45000
tls on
tls.verify off
shared_key secret

Input

  • Input Plugins
  • collectd
  • dummy - 随机生成日志
  • exec - 执行外部命令
  • forwad
  • head - 读取文件前面几行
  • health - TCP 健康检查
  • 系统 - cpu、disk、mem、netif、proc、thermal
  • kmsg - 内核日志
  • mqtt - MQTT 服务
  • random - 生成随机样本
  • serial - 串口
  • stdin
  • syslog
  • systemd
  • tail
  • tcp
  • winlog - Windows Event Log

Fliter

  • Fliter Plugins
  • expect - 条件过滤
  • grep
  • kubernetes - 添加元信息
  • lua
  • parser - 解析记录
  • record_modifier - 修改记录
  • rewrite_tag - 标签重写
  • stdout
  • throttle
  • nest - 嵌套记录
  • modify

Output

  • Output Plugins
  • es - Elasticsearch
  • file
  • forward
  • http
  • influxdb
  • kafka
  • kafka-rest
  • nats
  • null
  • stdout
  • tcp
  • postgresql
  • cockroachdb - v1.6+

插件

Build

if [ "$CBUILD" != "$CHOST" ]; then
CMAKE_CROSSOPTS="-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_HOST_SYSTEM_NAME=Linux"
fi

# default CORE_STACK_SIZE=((3 * PTHREAD_STACK_MIN) / 2)=3072 is invalid
# based on https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file
# set default to 24576
cmake -B build \
-DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_BUILD_TYPE=None \
-DFLB_CORO_STACK_SIZE=24576 \
-DFLB_TESTS_INTERNAL=Yes \
-DFLB_TLS=On \
-DFLB_JEMALLOC=Off \
$CMAKE_CROSSOPTS .
make -C build

FAQ

Fluentd vs Fluentbit

-FluentdFluent Bit
ScopeServersEmbedded & IoT devices
LanguageC & RubyC
Memory~20MB~150KB
PerformanceHigh PerformanceHigh Performance
DependenciesRuby Gem, need gems.no dependencies, unless some special plugin requires them.
Plugins300+ plugins available~15 plugins available
LicenseApache License v2.0Apache License v2.0
# fluentd
resources:
requests:
cpu: 100m
memory: 200Mi
limits:
memory: 500Mi

# fluentbit
resources:
requests:
cpu: 5m
memory: 10Mi
limits:
cpu: 50m
memory: 60Mi

core stack size

  • PTHREAD_STACK_MIN
    • musl 下为 2048
  • FLB_CORO_STACK_SIZE=((3 * PTHREAD_STACK_MIN) / 2)
    • musl 下为 3072 - 是无效的 core stack size
    • 可以考虑设置为 24576
// 默认
#define FLB_CORO_STACK_SIZE ((3 * PTHREAD_STACK_MIN) / 2)