How to redirect entire output of spark-submit to a file

So, I am trying to redirect the output of an apache spark-submit command to text file but some output fails to populate file. Here is the command I am using:

spark-submit something.py > results.txt

I can see the output in the terminal but I do not see it in the file. What am I forgetting or doing wrong here?

Edit:

If I use

spark-submit something.py | less

I can see all the output being piped into less

Answer

spark-submit prints most of it’s output to STDERR

To redirect the entire output to one file, you can use:

spark-submit something.py > results.txt 2>&1

Or

spark-submit something.py &> results.txt