I am trying to get Heapster working on my Kubernetes cluster. I am using Kube-DNS for DNS resolution.
My Kube-DNS seems to be set up correctly:
kubectl describe pod kube-dns-v20-z2dd2 -n kube-system
Name: kube-dns-v20-z2dd2 Namespace: kube-system Node: 172.31.48.201/172.31.48.201 Start Time: Mon, 22 Jan 2018 09:21:49 +0000 Labels: k8s-app=kube-dns version=v20 Annotations: scheduler.alpha.kubernetes.io/critical-pod= scheduler.alpha.kubernetes.io/tolerations=[{"key":"CriticalAddonsOnly", "operator":"Exists"}] Status: Running IP: 172.17.29.4 Controlled By: ReplicationController/kube-dns-v20 Containers: kubedns: Container ID: docker://13f95bdf8dee273ca18a2eee1b99fe00e5fff41279776cdef5d7e567472a39dc Image: gcr.io/google_containers/kubedns-amd64:1.8 Image ID: docker-pullable://gcr.io/google_containers/kubedns-amd64@sha256:39264fd3c998798acdf4fe91c556a6b44f281b6c5797f464f92c3b561c8c808c Ports: 10053/UDP, 10053/TCP Args: --domain=cluster.local. --dns-port=10053 State: Running Started: Mon, 22 Jan 2018 09:22:05 +0000 Ready: True Restart Count: 0 Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:8080/healthz-kubedns delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-9zxzd (ro) dnsmasq: Container ID: docker://576ebc30e8f7aae13000a2d06541c165a3302376ad04c604b12803463380d9b5 Image: gcr.io/google_containers/kube-dnsmasq-amd64:1.4 Image ID: docker-pullable://gcr.io/google_containers/kube-dnsmasq-amd64@sha256:a722df15c0cf87779aad8ba2468cf072dd208cb5d7cfcaedd90e66b3da9ea9d2 Ports: 53/UDP, 53/TCP Args: --cache-size=1000 --no-resolv --server=127.0.0.1#10053 --log-facility=- State: Running Started: Mon, 22 Jan 2018 09:22:20 +0000 Ready: True Restart Count: 0 Liveness: http-get http://:8080/healthz-dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-9zxzd (ro) healthz: Container ID: docker://3367d05fb0e13c892243a4c86c74a170b0a9a2042387a70f6690ed946afda4d2 Image: gcr.io/google_containers/exechealthz-amd64:1.2 Image ID: docker-pullable://gcr.io/google_containers/exechealthz-amd64@sha256:503e158c3f65ed7399f54010571c7c977ade7fe59010695f48d9650d83488c0a Port: 8080/TCP Args: --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null --url=/healthz-dnsmasq --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null --url=/healthz-kubedns --port=8080 --quiet State: Running Started: Mon, 22 Jan 2018 09:22:32 +0000 Ready: True Restart Count: 0 Limits: memory: 50Mi Requests: cpu: 10m memory: 50Mi Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-9zxzd (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: default-token-9zxzd: Type: Secret (a volume populated by a Secret) SecretName: default-token-9zxzd Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43m default-scheduler Successfully assigned kube-dns-v20-z2dd2 to 172.31.48.201 Normal SuccessfulMountVolume 43m kubelet, 172.31.48.201 MountVolume.SetUp succeeded for volume "default-token-9zxzd" Normal Pulling 43m kubelet, 172.31.48.201 pulling image "gcr.io/google_containers/kubedns-amd64:1.8" Normal Pulled 43m kubelet, 172.31.48.201 Successfully pulled image "gcr.io/google_containers/kubedns-amd64:1.8" Normal Created 43m kubelet, 172.31.48.201 Created container Normal Started 43m kubelet, 172.31.48.201 Started container Normal Pulling 43m kubelet, 172.31.48.201 pulling image "gcr.io/google_containers/kube-dnsmasq-amd64:1.4" Normal Pulled 42m kubelet, 172.31.48.201 Successfully pulled image "gcr.io/google_containers/kube-dnsmasq-amd64:1.4" Normal Created 42m kubelet, 172.31.48.201 Created container Normal Started 42m kubelet, 172.31.48.201 Started container Normal Pulling 42m kubelet, 172.31.48.201 pulling image "gcr.io/google_containers/exechealthz-amd64:1.2" Normal Pulled 42m kubelet, 172.31.48.201 Successfully pulled image "gcr.io/google_containers/exechealthz-amd64:1.2" Normal Created 42m kubelet, 172.31.48.201 Created container Normal Started 42m kubelet, 172.31.48.201 Started container
kubectl describe svc kube-dns -n kube-system
Name: kube-dns Namespace: kube-system Labels: k8s-app=kube-dns kubernetes.io/cluster-service=true kubernetes.io/name=KubeDNS Annotations: <none> Selector: k8s-app=kube-dns Type: ClusterIP IP: 10.254.0.2 Port: dns 53/UDP TargetPort: 53/UDP Endpoints: 172.17.29.4:53 Port: dns-tcp 53/TCP TargetPort: 53/TCP Endpoints: 172.17.29.4:53 Session Affinity: None Events: <none>
kubectl describe ep kube-dns -n kube-system
Name: kube-dns Namespace: kube-system Labels: k8s-app=kube-dns kubernetes.io/cluster-service=true kubernetes.io/name=KubeDNS Annotations: <none> Subsets: Addresses: 172.17.29.4 NotReadyAddresses: <none> Ports: Name Port Protocol ---- ---- -------- dns 53 UDP dns-tcp 53 TCP Events: <none>
kubectl exec -it busybox1 — nslookup kubernetes.default
Server: 10.254.0.2 Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.254.0.1 kubernetes.default.svc.cluster.local
However if I am trying to resolve http://monitoring-influxdb on either the busybox container (outside the kube-system namespace) it can’t get resolved:
kubectl exec -it heapster-v1.2.0-7657f45c77-65w7w –container heapster -n kube-system — nslookup http://monitoring-influxdb
Server: (null) Address 1: 127.0.0.1 localhost Address 2: ::1 localhost nslookup: can't resolve 'http://monitoring-influxdb': Try again command terminated with exit code 1
kubectl exec -it heapster-v1.2.0-7657f45c77-65w7w –container heapster -n kube-system — cat /etc/resolv.conf
nameserver 10.254.0.2 search kube-system.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal options ndots:5
kubectl exec -it busybox1 — nslookup http://monitoring-influxdb
Server: 10.254.0.2 Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local nslookup: can't resolve 'http://monitoring-influxdb' command terminated with exit code 1
kubectl exec -it busybox1 — cat /etc/resolv.conf
nameserver 10.254.0.2 search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal options ndots:5
Finally here are the logs from the heapster pod. I could not find any error in the dns pod logs:
kubectl logs heapster-v1.2.0-7657f45c77-65w7w heapster -n kube-system
E0122 09:22:46.966896 1 influxdb.go:217] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.2:53: server misbehaving, will retry on use
Any pointers are highly appreciated.
EDIT:
The monitoring-influxdb is located in the same namespace as the heapster (kube-system).
kubectl exec -it heapster-v1.2.0-7657f45c77-65w7w –container heapster -n kube-system — nslookup monitoring-influxdb.kube-system
Server: (null) Address 1: 127.0.0.1 localhost Address 2: ::1 localhost nslookup: can't resolve 'monitoring-influxdb.kube-system': Name does not resolve command terminated with exit code 1
But for whatever reason busybox is able to resolve the server.
kubectl exec -it busybox1 — nslookup http://monitoring-influxdb.kube-system
Server: 10.254.0.2 Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local Name: monitoring-influxdb.kube-system Address 1: 10.254.48.109 monitoring-influxdb.kube-system.svc.cluster.local
kubectl -n kube-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE heapster ClusterIP 10.254.193.208 <none> 80/TCP 1h kube-dns ClusterIP 10.254.0.2 <none> 53/UDP,53/TCP 1h kubernetes-dashboard NodePort 10.254.89.241 <none> 80:32431/TCP 1h monitoring-grafana ClusterIP 10.254.176.96 <none> 80/TCP 1h monitoring-influxdb ClusterIP 10.254.48.109 <none> 8083/TCP,8086/TCP 1h
kubectl -n kube-system get ep
NAME ENDPOINTS AGE heapster 172.17.29.7:8082 1h kube-controller-manager <none> 1h kube-dns 172.17.29.6:53,172.17.29.6:53 1h kubernetes-dashboard 172.17.29.5:9090 1h monitoring-grafana 172.17.29.3:3000 1h monitoring-influxdb 172.17.29.3:8086,172.17.29.3:8083 1h
Advertisement
Answer
In kubernetes, you can resolve services by their name alone, but only if you are inside the same namespace.
Services are also reachable through a DNS name in the form:
<service name>.<namespace>
From your question is not clear in which namespace you deployed influxdb, but give the above suggestion a try.