Write background command output (stdout) to file during execution

I have a command that takes a long time that I like to run in the background like this:

python3 script.py -f input.key -o output >> logs/script.log 2>&1 &

This works perfectly in the sense that the command is indeed in the background and I can check the output and potential errors later.

The main problem is the output is only appended after the command is completely finished, whereas I would like to have up-to-date log messages so I check the progress.

So currently the log would be empty and than suddenly at 08:30 two lines would appear:

[08:00] Script starting...
[08:30] Script finished!

Instead, I would like to have output saved to file before the command is completely finished.

Answer

Since you are calling a Python script you would want to use the -u option, which forces the stdout and stderr streams to be unbuffered.

$ python3 -u script.py -f input.key -o output >> logs/script.log 2>&1 &

You can check the log periodically using cat or realtime in combination with watch:

$ watch cat logs/script.log

↳ https://docs.python.org/3.7/using/cmdline.html#cmdoption-u