K8S日志通过fluent-bit采集到ES并在Kibana上查询

一 背景和简介

背景：我们的K8S集群，之前虽然已经搭建起来并且服务也已经在上面跑了，但是我们的服务日志并没有做采集，导致的结果是只要pod重启，历史的日志就没法查看了，非常不利于定位问题。
fluent-bit官网：https://fluentbit.io/
fluent-bit官方简介：Fluent Bit 是一个开源日志处理器和转发器，它允许您从不同来源收集任何数据，如指标和日志，用过滤器丰富它们并将它们发送到多个目的地。它是 Kubernetes 等容器化环境的首选。

二 环境与版本

• K8S版本：1.15.12
• fluent-bit版本：1.7.9
• ES：6.2.3（原有单独部署-非k8s部署）
• Kibana：6.2.3（原有单独部署-非k8s部署）

三 fluent-bit部署过程

增加logging命名空间
logging-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: logging

配置外部的ES、Kibana到k8s的service
elasticsearch-logging.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: logging
  name: elasticsearch-logging
spec:
  ports:
    - port: 9200
      targetPort: 9200
      protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  namespace: logging
  name: elasticsearch-logging
subsets:
  - addresses:
      - ip: 10.0.43.248
    ports:
      - port: 9200
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: logging
  name: elasticsearch-logging
spec:
  rules:
    - host: elasticsearch-logging.xxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: elasticsearch-logging
              servicePort: 9200

kinbana-logging.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: logging
  name: kibana-logging
spec:
  ports:
    - port: 5601
      targetPort: 5601
      protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  namespace: logging
  name: kibana-logging
subsets:
  - addresses:
      - ip: 10.0.43.248
    ports:
      - port: 5601
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: logging
  name: kibana-logging
spec:
  rules:
    - host: kibana-logging.xxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: kibana-logging
              servicePort: 5601

fluent-bit的configmap与ds
fluent-bit-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

fluent-bit-ds.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.7.9
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch-logging"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      nodeSelector:
        beta.kubernetes.io/fluentd-ds-ready: "true"
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

至此，fluent-bit采集日志就算已经配置OK了，启动fluent-bit的ds之后，可以查看pod日志，看是否已经正常起来并且无报错。有报错则根据报错信息进一步定位调整。

我在安装过程中，一开始安装了较低版本的fluent-bit，采集会有报错，主要是因为一个index中有两种doc类型，原因是从ElasticSearch的6.0之后的版本，仅支持一种类型，而fluent-bit的老版本可能采集保存es的类型有两种，所以会看到很多保存ES失败的报错信息，但是升级版本并把之前的索引删掉之后，就完全正常了。

四 Kibana建立索引

直接访问进入kibana建立索引