Skip to content
Advertisement

How to sync configuration between hadoop worker machines

We have huge hadoop cluster and we installed one coordinator preso node and 850 presto workers nodes. now we want to change the values in the file – config.properties but this should be done on all the workers!

so under

  /opt/DBtasks/presto/presto-server-0.216/etc

the file is like this

[root@worker01 etc]# more config.properties
#
coordinator=false
http-server.http.port=8008
query.max-memory=50GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node=2GB
discovery.uri=http://master01.sys76.com:8008 

and we want to change it to

coordinator=false
http-server.http.port=8008
query.max-memory=500GB
query.max-memory-per-node=5GB
query.max-total-memory-per-node=20GB
discovery.uri=http://master01.sys76.com:8008 

but this was done only on the first node – worker01, but we need to do it also on all workers. well – we can copy this file by scp to all other workers , but not in case root is restricted but what I want to know , if presto already think about more elegant approach that sync the configuration on all workers node as all know after we set new values we need also to restart the presto louncer script

dose presto have solution to this ?

I must to tell that my cluster is restricted root , so we cant copy the files VIA ssh

Advertisement

Answer

Presto does not have the ability to sync the configurations. This is something you would need to manage outside e.g. using a tool like Ansible. There is also project command line tool presto-admin (https://github.com/prestosql/presto-admin) that can assist with deploying the configs across the cluster.

Additionally, if you are using public clouds such as AWS, there are commercial solutions from Starburst (https://www.starburstdata.com/) that can assist management of the configurations as well.

User contributions licensed under: CC BY-SA
8 People found this is helpful
Advertisement