Monitoring Kubernetes Clusters (CloudPE)

1. Executive Summary

CloudPE Kubernetes clusters support integrated monitoring, giving you built-in metrics collection, alerting, and visualization without deploying a separate monitoring stack.

• Enabled by setting the monitoring_enabled=true label during cluster deployment.
• Provides cluster-level and node-level metrics out of the box.
• ✅ No manual installation of Prometheus/Grafana is required if enabled at creation time.

2. Prerequisites

Before starting, ensure the following:

☸️ Cluster Requirement

• A Kubernetes cluster with integrated monitoring enabled must already exist.
  • See “Creating and Deleting Kubernetes Clusters” for details.

💻 Local Tooling

• The kubectl CLI must be installed and configured on your client machine.
• You must be able to download the cluster’s .kubeconfig file from the self-service panel.

3. Monitoring Stack Overview

When integrated monitoring is enabled, the following components are installed automatically:

📊 Prometheus (Data Collection, Storage, Search)

• node-exporter — exposes server-level and OS-level metrics.
• kube-state-metrics — generates metrics on the state of Kubernetes objects.

🔔 Alertmanager

• Handles alarm aggregation, processing, and dispatch.

📈 Grafana

• Provides metrics visualization dashboards.
  • Accessible in-cluster at magnum-grafana.kube-system.svc.cluster.local:80.

🌐 Metrics Endpoints

• Kubernetes object-state metrics are exposed at:

  magnum-kube-state-metrics.kube-system.svc.cluster.local:8080/metrics

• These can be scraped by Prometheus itself or by any Prometheus-compatible scraper.

💡 For configuring custom Alertmanager/Prometheus instances, refer to the kube-prometheus documentation. For the full list of exposed metrics, refer to the kube-state-metrics documentation.

4. Accessing the Grafana Dashboards

🔹 Phase A: Get Cluster Access

1. On the Kubernetes clusters screen, click the target cluster.
2. Click Download kubeconfig to save the .kubeconfig file to your client machine.
3. Install and configure kubectl on your client machine (see official Kubernetes documentation).
4. Point kubectl at your cluster:

export KUBECONFIG=<path_to_kubeconfig>

🔹 Phase B: Verify the Monitoring Stack

5. Confirm the kube-prometheus stack is running:

kubectl --namespace kube-system get pods -l "release=magnum"

Expected output (pod names/counts may vary):

NAME                                                    READY   STATUS    RESTARTS   AGE
magnum-kube-prometheus-sta-operator-85f757c5dc-ckllb    1/1     Running   0          3d17h
magnum-kube-state-metrics-5cc46cbc5f-tclcv              1/1     Running   0          3d17h
magnum-prometheus-node-exporter-99kfc                   1/1     Running   0          3d3h
magnum-prometheus-node-exporter-gwgzr                   1/1     Running   0          3d17h
magnum-prometheus-node-exporter-q2pm2                   1/1     Running   0          3d17h
magnum-prometheus-node-exporter-sqsl7                   1/1     Running   0          2d22h

🔹 Phase C: Retrieve the Grafana Admin Password

6. Get the auto-generated admin password:

kubectl get secret --namespace kube-system magnum-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo

🔹 Phase D: Port-Forward and Log In

7. Forward the Grafana service port to your local machine:

kubectl --namespace kube-system port-forward service/magnum-grafana 3000:80

8. Open http://localhost:3000 in your browser and log in as admin using the password from step 6.
9. In the left menu, go to Dashboards → Browse, and select the dashboard you want to view.

5. Accessing the Prometheus UI

🔹 Phase A: Set Up Access

1. On the Kubernetes clusters screen, click the target cluster.
2. Click Download kubeconfig to save the .kubeconfig file.
3. Install and configure kubectl on your client machine.
4. Point kubectl at your cluster:

export KUBECONFIG=<path_to_kubeconfig>

🔹 Phase B: Port-Forward Prometheus

5. Forward the Prometheus service port:

kubectl --namespace kube-system port-forward service/magnum-kube-prometheus-sta-prometheus 9090

6. Use the following endpoints in your browser:

6. Accessing the Alertmanager UI

🔹 Phase A: Set Up Access

1. On the Kubernetes clusters screen, click the target cluster.
2. Click Download kubeconfig to save the .kubeconfig file.
3. Install and configure kubectl on your client machine.
4. Point kubectl at your cluster:

export KUBECONFIG=<path_to_kubeconfig>

🔹 Phase B: Port-Forward Alertmanager

5. Forward the Alertmanager service port:

kubectl --namespace kube-system port-forward service/magnum-kube-prometheus-sta-alertmanager 9093

6. Open http://localhost:9093 to access the Alertmanager UI.

7. Safety & Recovery

🔒 Experimental Feature Notice

• This monitoring stack is experimental — do not rely on it as the sole monitoring solution in production clusters.

⚠️ Port-Forward Sessions

• kubectl port-forward sessions are tied to your terminal session and stop when the terminal closes or the connection drops.
• Only one local port can be forwarded at a time per command; use different local ports if forwarding multiple services simultaneously.

🕒 Best Practice

• Rotate/change the default Grafana admin password after first login if the dashboard will be used regularly.
• Restrict access to the .kubeconfig file, since it grants full API access to the cluster.

✅ Summary

• Integrated monitoring (Prometheus, Alertmanager, Grafana) is available on clusters created with monitoring_enabled=true.
• Requires:
  • Downloaded .kubeconfig file
  • kubectl installed and configured locally
  • Port-forwarding to reach Grafana, Prometheus, or Alertmanager UIs
• Grafana: localhost:3000 · Prometheus: localhost:9090 · Alertmanager: localhost:9093
• Feature is experimental — not for production use.