Skip to content
Advertisement

how do i execute grep chain command in background using bash script

I have a data for 24 hours logs per each day. the data is huge to grep the required string, taking 3 hours to search the string in all 24 hours dir. to reducing the consuming time I’ve decided to separate the 24 hours in to three parts and wanted to execute each part parallely in background using &

cat test.sh
log1="22 01  02  03  04  05  06  21"
log2="07  08  09  10  11  12  13  22"
log3="14  15  16  17  18  19  20  23"

for me in $log1
do
    for me1 in $me
    do
            cd  $me1
            $(cat * | grep ASS-va1appus06-1480458532238-14424 |  grep -i "am/agent/registered" &) >../log_$me1 &
            cd ..
    done
done

The problem here is not working as expected.its not running background and not updating log files, script coming out with out processing further.

some of the script output is
+ cd 22
+ cd ..
++ grep ASS-va1appus06-1480458532238-14424
++ grep -i am/agent/registered
++ cat logfile.log1 logfile.log2 .......

if you see the commands are not executing as per the given grep sequence,

Advertisement

Answer

You designed your paralell call wrongly, and mix paralell bits of code along with calls made in your main script. You could try the following approach :

#!/bin/bash
log1="22 01  02  03  04  05  06  21"
log2="07  08  09  10  11  12  13  22"
log3="14  15  16  17  18  19  20  23"

for me in $log1
do
    for me1 in $me
    do
            # you could aslo get rid of the cd altogether 
            # (grep "ASS-va1appus06-1480458532238-14424" $me1/* |  grep -i "am/agent/registered" >log_$me1) &
            (cd $me1 && grep "ASS-va1appus06-1480458532238-14424" * |  grep -i "am/agent/registered" >../log_$me1) &
    done
done
  • (...) means we execute our instructions in a subshell. Changes like variable declaration or cd are effective in this context, but do not affect our main script environment.
    • We put the full operation that need to be paralelized in this subshell. We need to put all the relevant information in this context.
    • We paralelize the full subshell.

A few notes regarding your script : I removed a useless use of cat, also beware of concurent writtings in the same file when using paralell computing.

User contributions licensed under: CC BY-SA
9 People found this is helpful
Advertisement