I have the following line inside of a .sh file:
aws s3 cp s3://bucket/folder/ /home/ec2-user/ --recursive
When I run this line in the console, it runs fine and completes as expected. When I run this line inside of a .sh file, the line returns the following error.
Unknown options: --recursive
Here is my full .sh script.
echo "Test" aws s3 cp s3://bucket/folder/ /home/ec2-user/ --recursive echo "Test" python /home/ec2-user/Run.py
If I manually add the Run.py file (not having it copied over from S3 as desired) and run the script I get the following output.
Test Unknown options: --recursive Test Hello World
If I remove the files which are expected to be transferred by S3 out of the Linux environment and rely on the AWS S3 command I get the following output:
Test Unknown options: --recursive Test python: can't open file '/home/ec2-user/Run.py': [Errno 2] No such file or directory
Note that there are multiple files that can be transferred inside of the specified S3 location. All of these files are transferred as expected when running the AWS command from the console.
I initially thought this was a line ending error, as I am copying the .sh file over from Windows. I have made sure that my line endings are working and we see this by the rest of the script running as expected. As a result, this issue seems isolated to the actual AWS command. If I remove the --recursive from the call, it will transfer over a single file successfully.
Any ideas on why the –recursive option would be working in the console but not in the .sh file?
P.s.
$ aws --version aws-cli/2.1.15 Python/3.7.3 Linux/4.14.209-160.339.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off
Advertisement
Answer
I think it is about the position of the option or you have somewhere the AWS CLI path which is pointing to older version because recursive option was added later. For me, it works both ways inside the shell script as well as the console.
aws s3 cp --recursive s3://bucket/folder/ /home/ec2-user/ $ aws --version aws-cli/2.1.6 Python/3.7.4 Darwin/20.2.0 exe/x86_64 prompt/off
Try printing the version inside the shell script.
cp –recursive method lists source path and copies (overwrites) all to the destination path.
Plus instead of doing recursive use sync.sync recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files.
aws s3 sync 3://bucket/folder/ /home/ec2-user/
sync method first lists both source and destination paths and copies only differences (name, size etc.).