AWS CLI S3 CP –recursive function works in console but not in .sh file

I have the following line inside of a .sh file:

aws s3 cp s3://bucket/folder/ /home/ec2-user/ --recursive

When I run this line in the console, it runs fine and completes as expected. When I run this line inside of a .sh file, the line returns the following error.

Unknown options: --recursive

Here is my full .sh script.

echo "Test"
aws s3 cp s3://bucket/folder/ /home/ec2-user/ --recursive
echo "Test"
python /home/ec2-user/Run.py

If I manually add the Run.py file (not having it copied over from S3 as desired) and run the script I get the following output.

Test

Unknown options: --recursive
Test
Hello World

If I remove the files which are expected to be transferred by S3 out of the Linux environment and rely on the AWS S3 command I get the following output:

Test

Unknown options: --recursive
Test
python: can't open file '/home/ec2-user/Run.py': [Errno 2] No such file or directory

Note that there are multiple files that can be transferred inside of the specified S3 location. All of these files are transferred as expected when running the AWS command from the console.

I initially thought this was a line ending error, as I am copying the .sh file over from Windows. I have made sure that my line endings are working and we see this by the rest of the script running as expected. As a result, this issue seems isolated to the actual AWS command. If I remove the --recursive from the call, it will transfer over a single file successfully.

Any ideas on why the –recursive option would be working in the console but not in the .sh file?

P.s.

$ aws --version
aws-cli/2.1.15 Python/3.7.3 Linux/4.14.209-160.339.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off

Answer

I think it is about the position of the option or you have somewhere the AWS CLI path which is pointing to older version because recursive option was added later. For me, it works both ways inside the shell script as well as the console.

aws s3 cp --recursive  s3://bucket/folder/ /home/ec2-user/ 

$ aws --version
aws-cli/2.1.6 Python/3.7.4 Darwin/20.2.0 exe/x86_64 prompt/off

Try printing the version inside the shell script.

cp –recursive method lists source path and copies (overwrites) all to the destination path.

Plus instead of doing recursive use sync.sync recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files.

aws s3 sync 3://bucket/folder/ /home/ec2-user/

sync method first lists both source and destination paths and copies only differences (name, size etc.).