Logging

Logging in our clusters uses a Elasticsearch / Fluentd / Kibana stack.

Save logging of application

Pod logs will not be picked up and stored in Elasticsearch Database unless the application has the label fluentd: "true"

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
    fluentd: "true"

Create a log output filter

Log output filters needs to be added to the configMap log-filters

apiVersion: v1
data:
  nginx.conf: |
    <filter kubernetes.**>
        @type parser
        key_name log
        reserve_data true
        emit_invalid_record_to_error false
        <parse>
            @type regexp
            expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/
            time_format %d/%b/%Y:%H:%M:%S %z
        </parse>
    </filter>
  apache2.conf: |
    <filter kubernetes.**>
        @type parser
        key_name log
        reserve_data true
        emit_invalid_record_to_error false
        <parse>
            @type regexp
            expression /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
            time_format %d/%b/%Y:%H:%M:%S %z
        </parse>
    </filter>
kind: ConfigMap
metadata:
  name: log-filters

Kibana Dashboard

Default username is elastic and password can be found with kubectl

$ kubectl get secret -n elastic-system es-logging-es-elastic-user \
-o go-template='{{.data.elastic | base64decode }}'