I have a data for 24 hours logs per each day. the data is huge to grep the required string, taking 3 hours to search the string in all 24 hours dir. to reducing the consuming time I’ve decided to separate the 24 hours in to three parts and wanted to execute each part parallely in background using &
cat test.sh log1="22 01 02 03 04 05 06 21" log2="07 08 09 10 11 12 13 22" log3="14 15 16 17 18 19 20 23" for me in $log1 do for me1 in $me do cd $me1 $(cat * | grep ASS-va1appus06-1480458532238-14424 | grep -i "am/agent/registered" &) >../log_$me1 & cd .. done done
The problem here is not working as expected.its not running background and not updating log files, script coming out with out processing further.
some of the script output is + cd 22 + cd .. ++ grep ASS-va1appus06-1480458532238-14424 ++ grep -i am/agent/registered ++ cat logfile.log1 logfile.log2 .......
if you see the commands are not executing as per the given grep sequence,
Advertisement
Answer
You designed your paralell call wrongly, and mix paralell bits of code along with calls made in your main script. You could try the following approach :
#!/bin/bash log1="22 01 02 03 04 05 06 21" log2="07 08 09 10 11 12 13 22" log3="14 15 16 17 18 19 20 23" for me in $log1 do for me1 in $me do # you could aslo get rid of the cd altogether # (grep "ASS-va1appus06-1480458532238-14424" $me1/* | grep -i "am/agent/registered" >log_$me1) & (cd $me1 && grep "ASS-va1appus06-1480458532238-14424" * | grep -i "am/agent/registered" >../log_$me1) & done done
(...)
means we execute our instructions in a subshell. Changes like variable declaration orcd
are effective in this context, but do not affect our main script environment.- We put the full operation that need to be paralelized in this subshell. We need to put all the relevant information in this context.
- We paralelize the full subshell.
A few notes regarding your script : I removed a useless use of cat, also beware of concurent writtings in the same file when using paralell computing.