Heapster monitors the kubernetes cluster, more information on it is available here.

Deploy Heapster

There are some good example of the deployment here:

First let’s clone their repo:

git clone https://github.com/kubernetes/heapster.git

Next, modify the files to use a NodePort:

# heapster/deploy/kube-config/influxdb/grafana.yaml
type: NodePort
  - port: 80
    targetPort: 3000
    nodePort: 30081

Also add local volumes:

# heapster/deploy/kube-config/influxdb/grafana.yaml
- name: grafana-storage
          path: /data/shared/heapster/grafana/data


# heapster/deploy/kube-config/influxdb/influxdb.yaml
- name: influxdb-storage
          path: /data/shared/heapster/influxdb/data

Then deploy the pod:

<> kubectl apply -f .
deployment "monitoring-grafana" created
service "monitoring-grafana" created
serviceaccount "heapster" created
deployment "heapster" created
service "heapster" created
deployment "monitoring-influxdb" created
service "monitoring-influxdb" created

Fix Permission issue

At first had permission errors:

<> kubectl logs --namespace=kube-system po/heapster-5d67855584-zq8k5
I1125 18:33:23.332631       1 heapster.go:72] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
I1125 18:33:23.332676       1 heapster.go:73] Heapster version v1.4.0
I1125 18:33:23.332869       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I1125 18:33:23.332884       1 configs.go:62] Using kubelet port 10255
E1125 18:33:23.342896       1 kubelet.go:334] Failed to load nodes: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list nodes at the cluster scope
E1125 18:33:23.343800       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:51: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list nodes at the cluster scope
E1125 18:33:23.388293       1 influxdb.go:264] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb.kube-system.svc:8086" - Get http://monitoring-influxdb.kube-system.svc:8086/ping: dial tcp: lookup monitoring-influxdb.kube-system.svc on no such host, will retry on use
I1125 18:33:23.388332       1 influxdb.go:278] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s
I1125 18:33:23.388368       1 heapster.go:196] Starting with InfluxDB Sink
I1125 18:33:23.388376       1 heapster.go:196] Starting with Metric Sink
E1125 18:33:23.390558       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:51: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list nodes at the cluster scope

The permission issue is mentioned at the readme, and we need to create the RBAC role:

cd heapster/deploy/kube-config/rbac/

Then just create it:

<> kubectl apply -f heapster-rbac.yaml
clusterrolebinding "heapster" configured

Then was able to go to the dashboard (http://{K8S_HOST}:{NodePort}) and see cluster information:


and also pod information:


And also checking out the kubernetes Dashboard, I now saw CPU information as well:


Confirm Kubernetes DNS is working

The easiest thing to do is to just attach to a container running in kubernetes and run the following:

<> docker exec -it b931cce32e8e /bin/sh
$ ping -c 1 monitoring-influxdb.kube-system.svc
PING monitoring-influxdb.kube-system.svc.cluster.local ( 56 data bytes
--- monitoring-influxdb.kube-system.svc.cluster.local ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

It matched the IP of the deployment:

<> kubectl get service monitoring-influxdb --namespace=kube-system
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
monitoring-influxdb   ClusterIP   <none>        8086/TCP   23m

If configured appropriately with iptables you can also do a host lookup from the docker host it self:

<> host kubernetes.default.svc.cluster.local
Using domain server:

kubernetes.default.svc.cluster.local has address

and also the service it self:

<> host monitoring-influxdb.kube-system.svc.cluster.local
Using domain server:

monitoring-influxdb.kube-system.svc.cluster.local has address

Lastly you can also do a simple busybox deployment for testing (this is covered in Troubleshooting Tips). Here is a simple deploy config:

<> cat busybox/deploy.yaml
apiVersion: v1
kind: Pod
  name: busybox
  namespace: default
  - image: busybox
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Never
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

Then deploy it:

<> kubectl apply -f deploy.yaml
pod "busybox" created

Then do a test run

<> kubectl exec -it busybox nslookup kubernetes.default
Address 1: kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: kubernetes.default.svc.cluster.local

You can go one step further and make sure the port is reachable:

<> kubectl exec -it busybox /bin/sh
/ # telnet monitoring-influxdb.kube-system.svc 8086

HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

400 Bad RequestConnection closed by foreign host