Background 6 node Kafka Cluster 3 node Zookeeper Cluster 3 node Nimbus Cluster Apache Storm Worker hosts dynamically adjusted using amazon spot fleet Scenario For a particular topology for a given partition it subscribes to, the Zookeeper entry looks as follows {“topology”:{“id”:”Topology_Name-25-1520374231″,”name”:”Topology_Name”},”offset”:217233,”partition”:0,”broker”:{“host”:”Zk_host_name”,”port”:9092},”topic”:”topic1″} Now for worker hosts to access Zk_host_name, a mapping is added on each worker host in /etc/hosts file