Skip to content

Fluent Bit

Overview

FluentBit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources, enrich them with filters and send them to multiple destinations. It’s the preferred choice for containerized environments like Kubernetes.

Big Bang Touch Points

graph TB
  subgraph "Fluent-Bit"
    fluentbit
  end

  subgraph "Elasticsearch"
    fluentbit --> elasticsearch
  end

Storage

Fluentbit itself does not use or require any persistent storage, however it does need hostPath mounts to the kubernetes nodes to tail and process log data. These hostPath volumes are for /var/log/containers to tail logs from containers running on the nodes, and /var/log/flb-storage which is a configurable storage buffer path in use for Big Bang production environments. This storage buffer is configurable via the following umbrella values. Keep storage.path aligned with the volume mount; config.service replaces the whole service configuration, so retain any other settings you use:

packages:
  fluentbit:
    values:
      upstream:
        config:
          service: |
            [SERVICE]
                Daemon Off
                Flush 1
                Log_Level info
                Parsers_File /fluent-bit/etc/parsers.conf
                Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
                HTTP_Server On
                HTTP_Listen 0.0.0.0
                HTTP_Port 2020
                storage.path /var/log/flb-storage/
                storage.sync normal
                storage.backlog.mem_limit 15M
                Health_Check On
        extraVolumes:
          - hostPath:
              path: /var/log/flb-storage/
              type: DirectoryOrCreate
            name: flb-storage

        extraVolumeMounts:
          - mountPath: /var/log/flb-storage/
            name: flb-storage

This storage buffer hostPath mount, in conjunction with the hostPath mount of /var/log/containers/ used to fetch logs requires a securityContext of privileged to be set if SELinux is set to Enforcing on the kubernetes nodes. To set this securityContext for the fluentbit pods, add the following values in Big Bang:

packages:
  fluentbit:
    values:
      upstream:
        securityContext:
          privileged: true

Logging

Since Fluentbit is the method for shipping cluster logs to the ECK stack, to reduce the amount of logs fluentbit and ECK has to process, fluentbit container logs are excluded from being processed and shipped to ECK. However, if you would like to enable fluentbit container logs being sent to ECK you just have to remove the “Excluded_Path” portion of this INPUT block (requires presence of entire block even when changing a single line):

packages:
  fluentbit:
    values:
      upstream:
        config:
          inputs: |
            [INPUT]
                Name tail
                Path /var/log/containers/*.log
                Exclude_Path /var/log/containers/*fluent*.log,/var/log/containers/*gatekeeper-audit*.log
                Parser containerd
                Tag kube.*
                Mem_Buf_Limit 50MB
                Skip_Long_Lines On
                storage.type filesystem

High Availability

Fluent-bit by default runs as a Kubernetes DaemonSet with a single pod on each node in the cluster. There is no need to run multiple pods per node as only one is required to maintain the state of logs that appear on that node. The Big Bang fluent-bit package also comes with default values to enable and configure a storage buffer to better index and process records on your Kubernetes nodes in the event of pod restarts or pods becoming unhealthy.

Health Checks

Fluentbit is able to be configured with a service port for the container, which is able to expose all kinds of metrics including metrics for Prometheus. Starting with Chart version 0.15.X fluentbit comes packaged (when monitoring is enabled) with a ServiceMonitor for the prometheus-operator also bundled with Big Bang so that metrics are available in the Prometheus and Grafana UIs, the latter thanks to this Grafana Dashboard.