awk in non-interactive is leaving some data missing

I’m running an awk command in a shell script, and it’s breaking the datafile. All of the data doesn’t load, and there’s a newline character at the end of each line.

awk -v name="$filename" -v batch="$BATCHNAME" -F'|' 'BEGIN{OFS="|"}{$2=batch;print > (name".locked.concur")}' $filename.locked.concur

If I run this as a command, it works perfectly.

I’ve just found that i can use fflush() and it seems to have solved the issue.

So now i have:

awk -v name="$filename" -v batch="$BATCHNAME" -F'|' 'BEGIN{OFS="|"}{$2=batch;print > (name".locked.concur") fflush()}' $filename.locked.concur

Am I setting myself up for disaster?

Also, the file is renamed to $filename.locked.concur0 — I’m guessing this zero is the exit code that’s returned from fflush? How do I get rid of this 0?

full code:

for filename in `cat ${INFILEDIR}/file_list_concur`

do

rm -f $OHAP_DAT/bad/$filename.bad

##Create Batchname based on filename

BATCH_GROUP=`ls $filename.locked.concur |cut -d '_' -f 1`
BATCH_DATE=`ls $filename.locked.concur |cut -d '_' -f 2`
BATCHNAME=${BATCH_GROUP}_Concur_${BATCH_DATE}

echo 'Batch Date = '$BATCH_DATE
echo 'Batch Group = '$BATCH_GROUP
echo 'Batch Name = '$BATCHNAME
echo 'File Name = ' $filename

gawk -v name="$filename" -v batch="$BATCHNAME" -F'|' 'BEGIN{OFS="|"}{$2=batch;print > (name".locked.concur");}' $filename.locked.concur


##End Batchname

sqlldr $ORACLE_PW control=$XXOH_TOP/bin/XXOH_AP_CONCUR_IMPORT_CTL.ctl 
              data=$OHAP_DAT/in/$filename.locked.concur 
              log=$OHAP_DAT/log/$filename.log 
              bad=$OHAP_DAT/bad/$filename.bad 
                bindsize=512000 << end_of_sqlldr
end_of_sqlldr

if [ -f $OHAP_DAT/bad/$filename.bad ]
then
  echo !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  echo !! 'date'
  echo !! SQL*Loader bad file was found!!
  echo !! Script $filename Failed - BAD FILE!
  echo !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Answer

the problem is indeed that you are reading from and writing to the same file. this should work better:

gawk ... print > (name".locked.concur.tmp")}' $filename.locked.concur
mv -f $filename.locked.concur.tmp $filename.locked.concur

better still:

gawk ... print}' $filename.locked.concur > $filename.locked.concur.tmp
mv -f $filename.locked.concur.tmp $filename.locked.concur