Using variables on grep –q doesn’t produce founds

I need to extract entries from a log file and put them on an errors file.

I don’t want to duplicate the entries on the errors file every time that the script is run, so I create this:

grep $1 $2 | while read -r line ; do
    echo "$line"
    if [ ! -z "$line" ]
    then
            echo "Line is NOT empty"
            if grep -q "$line" $3; then
                    echo "Line NOT added"
            else
                    echo $line >> $3
                    echo "Line added"
            fi
    fi
done

And is run using:

./log_monitor.sh ERROR logfile.log errors.txt

The first time that the script runs it finds the entries, and create the errors file (there are no errors file before).

The next time, this line never found the recently added lines to the errors file,

if grep -q “$line” $3;

therefore, the script adds the same entries to the errors file.

Any ideas of why this is happening?

Answer

This most likely happens because you are not searching for the line itself, but treating the line as regex. Let’s say you have this file:

$ cat file
[ERROR] This is a test
O This is a test

and you try to find the first line:

$ grep "[ERROR] This is a test" file
O This is a test

As you can see, it does not match the line we’re looking for (causing duplicates) and does match a different line (causing dropped entries). You can instead use -F -x to search for literal strings matching the full line:

$ grep -F -x "[ERROR] This is a test" file
[ERROR] This is a test

Applying this to your script:

grep $1 $2 | while read -r line ; do
    echo "$line"
    if [ ! -z "$line" ]
    then
            echo "Line is NOT empty"
            if grep -F -x -q "$line" $3; then
                    echo "Line NOT added"
            else
                    echo $line >> $3
                    echo "Line added"
            fi
    fi
done

And here with additional fixes and cleanup:

grep -e "$1" -- "$2" | while IFS= read -r line ; do
    printf '%sn' "$line"
    if [ "$line" ]
    then
            echo "Line is NOT empty"
            if grep -Fxq -e "$line" -- "$3"; then
                    echo "Line NOT added"
            else
                    printf '%sn' "$line" >> "$3"
                    echo "Line added"
            fi
    fi
done

PS: this could be shorter, faster and have a better time complexity with a snippet of awk.