Prometheus

In our Private Managed Clusters Prometheus will automatically scan all Deployments, DaemonSets and StatefulSets for a scraping port or ServiceMonitors.

If you want to use scraping, you need to add the following annotations, example:

apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  annotations:
    prometheus.io/port: 7472
    prometheus.io/scrape: true
...

For a ServiceMonitor you need to create a ServiceMonitor map in the Prometheus namespace:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: coredns
  name: coredns
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: http-metrics
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      component: metrics
      k8s-app: coredns

Grafana Dashboard

We supply a Grafana Dashboard that provides an overview of the Kubernetes cluster. The default username is admin and password can be found in the Grafana ConfigMap:

$ kubectl -n prometheus describe configmaps grafana-config

Persistent Storage

To enable persistent storage in Grafana should it be patched to use a storage volume. This will require Ceph or another form of network or local storage

Create a persistent volume claim for Grafana

$ kubectl -n prometheus create --edit -f \
  https://asergo.com/knowledge-base/kubernetes/deploy/grafana/grafana-storage-pvc.yml
# https://asergo.com/knowledge-base/kubernetes/deploy/grafana/grafana-storage-pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-storage
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi # Set the size to match your needs
  volumeMode: Filesystem

Patch Grafana Deployment to use the volume claim

$ kubectl -n prometheus apply -f \
  https://asergo.com/knowledge-base/kubernetes/deploy/grafana/grafana-ds-storage.yml
# https://asergo.com/knowledge-base/kubernetes/deploy/grafana/grafana-ds-storage.yml
      volumes:
      - persistentVolumeClaim:
          claimName: grafana-storage
        name: grafana-storage

Custom Grafana Settings

It is possible to add custom settings to Grafana, such as GitLab, Github or SSO user authentication. These settings should be added to the configmap grafana-config

Disable anonymous browsing

apiVersion: v1
data:
  grafana.ini: |-
  ...
    [auth.anonymous]
    enabled = false
    org_name = Main Org.
    org_role = Viewer
  ...
kind: ConfigMap
metadata:
  name: grafana-config