Logging

Clusters with our logging addon uses a Elasticsearch / Fluentd / Kibana stack. The stack is installed and ready to use, you only need to add filters for your application.

Save logging of application

Pod logs will not be picked up and stored in Elasticsearch Database unless the application has the appropriate label

ASERGO Kubernetes V2 and below

spec:
  metadata:
    labels:
      fluentd: "true"

ASERGO Kubernetes V3+

spec:
  metadata:
    labels:
      asergo.com/logs: "true"

Create a log output filter

ASERGO Kubernetes V2 and below

Log output filters needs to be added to the configMap fluentd-filters in the logging namespace

apiVersion: v1
data:
  filters.conf: |
    <filter kubernetes.**>
        @type parser
        key_name log
        reserve_data true
        emit_invalid_record_to_error false
        <parse>
            @type regexp
            expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/
            time_format %d/%b/%Y:%H:%M:%S %z
        </parse>
    </filter>
kind: ConfigMap
metadata:
  name: log-filters

ASERGO Kubernetes V3+

Log output is defined as parsers and needs to be added to the configMap parsers in the logging namespace

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Add an annotation with with name of the custom parser to use it with your application

annotations:
    fluentbit.io/parser: nginx

Note

All pods in logging namespace needs to restart if changes are made to the fluentd-filters or parsers configmap

Kibana Dashboard

Kibana Dashboard can be accessed through your ASERGO Dashboard.

Default username is elastic and password can be found with kubectl

ASERGO Kubernetes V2 and below

kubectl get secret -n default fluentd-es-elastic-user \
-o go-template='{{.data.elastic | base64decode }}'

ASERGO Kubernetes V3+

kubectl get secret -n default application-logs \
-o go-template='{{.data.elastic | base64decode }}'