I have a great amount of Linux servers to maintain. Frequently I need to run a script (script.sh) on all of them to get the health status, this script usually takes about 30-40 seconds to give an output. To facilitate maintenance tasks, I’m writing a shell script that uses SSH to loop through all remote hosts, run script.sh, collect output and write it to a log file in my local host. For the sake of this question, I have named this script MyScript.sh
The script works fine, however, it has to wait for the SSH output to continue to the next host. Because I have too many servers, and the commands runs in sequence, it take several minutes to finish. I would like to loop through all servers in parallel, without needing to wait for a response from each host.
Is there a way I can remotely run script.sh simultaneously on all host using MyScript.sh? Maybe run the ssh command in the background and somehow collect the output?
The output of script.sh is a single line separated by pipes. Such as the following
host1|49 days|10%|3.77%|27677/63997 MB|43% - /usr|38% - /usr|Optimal|No|40%|No
The output of Myscript.sh is the concatenation of the output from all host without pipes.
Date Hostname Uptime CPU I/O Free MEM File System INODES STATUS WWW YYY ZZZ XXX
===================================================================================================================================================================================================
01/31/20 host1 44 days 5% 10.33% 38083/64000 MB 57% - / 37% - /usr OPTIMAL No 40% No
01/31/20 host2 45 days 11% 1.79% 27915/63997 MB 43% - /usr 38% - /usr OPTIMAL UP 7% OK
01/31/20 host3 45 days 2% 1.89% 32145/63997 MB 43% - /usr 38% - /usr OPTIMAL UP NO OK
01/31/20 host4 45 days 11% 3.72% 52477/128637 MB 49% - /var 38% - /usr OPTIMAL UP 8% OK
01/31/20 host5 45 days 6% 3.21% 65264/128637 MB 46% - /var 38% - /usr OPTIMAL UP NO OK
01/31/20 host6 45 days 7% 5.79% 56369/63997 MB 43% - /usr 38% - /usr OPTIMAL UP NO No
01/31/20 host7 45 days 6% 1.66% 56391/63997 MB 43% - /var 38% - /usr OPTIMAL UP NO No
The core of MyScript.sh is the following:
(
for ip in $IP_LIST;
do
echo "Checking $ip"
ssh -q -t $user@$ip 'sudo /tmp/script.sh' > /tmp/$$
current_date=$(date +%D)
printf "%-10s " "$current_date" >> $logfile
while read line;
do
echo $line | awk -F '|' '{printf("%-10s %-10s %-7s %-8s %-18s %-25s %-25s %-15s %-15s %-25s %-10sn",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11); }' >> $logfile
done< /tmp/$$
done
)
In summary, I would like to optimize this script to run the above code simultaneously on multiple servers. Thanks!
Advertisement
Answer
The solution could be to deploy a monitoring software with custom checks.
For the parrallel ssh problem, without install any binaries you could use this script I wrote a while ago.
Put in a file mssh, run chmod u+x mssh and then :
./mssh -s SERVER1 -s SERVER2 -C script.sh
The mssh file :
#!/usr/bin/env bash
readonly prog_name="$(basename "$0")"
readonly date="$(date +%Y%m%d_%H%M%S)"
# print help
usage() {
cat <<- EOF
usage: $prog_name options
parallel ssh executions.
OPTIONS:
-c --cmd CMD execute command CMD
-s --host SRV execute cmd on server SRV
-C --cmd CMD_FILE execute command contained in CMD_FILE
-S --hosts-file SRV_FILE execute cmd on all servers contained in SRV_FILE
-h --help show this help
Examples:
Run CMD on SERVER1 and SERVER2:
./$prog_name -s SERVER1 -s SERVER2 -c "CMD"
EOF
}
# test if an element is in an array
is_element(){
local search=$1; shift;
for e in "$@"; do [[ "$e" == "$search" ]] && return 0; done
return 1
}
# parse arguments
for arg in "$@"; do
case "$arg" in
--help) args+=( -h );;
--host) args+=( -s );;
--hosts-file) args+=( -S );;
--cmd) args+=( -c );;
--cmd-file) args+=( -C );;
*) args+=("$arg");;
esac
done
set -- "${args[@]}"
while getopts "hs:S:c:C:" OPTION; do
case $OPTION in
h) usage; exit 0;;
s) servers_array+=("$OPTARG");;
S) while read -r L; do servers_array+=("$L"); done < <( grep -vE "^ *(#|$)" "$OPTARG");;
c) cmd="$OPTARG";;
C) cmd="$(< "$OPTARG")"; file=$OPTARG;;
*) :;;
esac
done
if [[ -z ${servers_array[0]} ]] || [[ -z $cmd ]]; then
usage; exit 1
fi
# clean up created files at exit
trap "rm -f /tmp/pssh*$date" EXIT
[[ -n $file ]] && echo "executing command file : $file" || echo "executing command : $cmd"
# run cmd on each server
for i in "${!servers_array[@]}"; do
# executing cmd in subshell
ssh -n "${servers_array[$i]}" "$cmd" > "/tmp/pssh_${i}_${servers_array[$i]}_${date}" 2>&1 &
pid=$!
pids_array+=("$pid")
echo "${servers_array[$i]} - $pid"
done
# for each pid, set state to running
ps_state_array=( $(for i in "${!servers_array[@]}"; do echo "running"; done) )
echo "waiting for results..."
echo
# begin finished verifications
continue=true; attempt=0
while $continue; do
# foreach ps
for i in "${!pids_array[@]}"; do
# if already finished skip
[[ ${ps_state_array[$i]} == "finished" ]] && continue
# else check if finished
ps -o pid "${pids_array[$i]}" > /dev/null 2>&1 && ps_finished=false || ps_finished=true
if $ps_finished; then
ps_state_array[$i]="finished"
echo -e "[ ${servers_array[$i]} @ $(date +%H:%M:%S) ]" | grep '.*' --color=always
cat "/tmp/pssh_${i}_${servers_array[$i]}_${date}"
rm -f "/tmp/pssh_${i}_${servers_array[$i]}_${date}"
echo
fi
done
is_element "running" "${ps_state_array[@]}" || continue=false
if $continue; then
(( attempt < 5 )) && attempt=$(( attempt + 1 ))
sleep $attempt
fi
done
exit 0