Docker awslogs gives error NoCredentialProviders

Problem

I’m trying to send docker logs to Aws Cloudwatch using my on premise server but it keeps failing on authentication. I’ve spent tons of hours searching through documentation and tutorials – and yet it does not work.

Approach

I’ve installed AWS-cli and configured it, so ~/.aws/config is filled with my credentials. I’ve also set temporarily the session-variables just to be safe:

EXPORT AWS_SECRET_ACCESS_KEY=...
EXPORT AWS_ACCESS_KEY_ID=...
EXPORT AWS_SESSION_TOKEN=...

I’ve verified I can connect to AWS using:

aws s3 ls

This is my run config:

docker run --log-driver=awslogs --log-opt awslogs-group=docker-logs --log-opt awslogs-region=eu-west-1 --log-opt awslogs-create-group=true alpine echo 'hi cloudwatch'

When tailing /var/log/daemon.log I see the following error:

Feb 1 01:12:07 XXXXXX dockerd[7389]: time="2021-02-01T01:12:07.670370559+01:00" level=error msg="Failed to create log stream" errorCode=NoCredentialProviders logGroupName=docker-logs logStreamName=61c82801d22d3db4c68cdc5b3d1dcba51f97c77dea5ce33e262b712c0e2a23a7 message="no valid providers in chain. Deprecated.ntFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" origError=""

I’ve also tried giving docker run the session-variables as –ENV-parameters. Still doesn’t work. I’ve also tried insert the EXPORT’ed variables into /etc/default/docker. Still no luck.

Docker has no problem uploading to cloudwatch if I run with log-driver=”json-file” and point to a file. So it’s only awslogs giving problems.

Versions

$ aws --version
aws-cli/2.1.22 Python/3.7.3 Linux/4.19.57-v7l+ source/armv7l.raspbian.10 prompt/off

$ docker --version
Docker version 20.10.2, build 2291f61

Answer

I was facing similar kind of issue on ubuntu 18 instance. Following are the steps which worked for me:

mkdir -p /etc/systemd/system/docker.service.d/
touch /etc/systemd/system/docker.service.d/aws-credentials.conf
vi /etc/systemd/system/docker.service.d/aws-credentials.conf content of file as follows: [Service] Environment=”AWS_ACCESS_KEY_ID=” Environment=”AWS_SECRET_ACCESS_KEY=”
sudo systemctl daemon-reload
sudo service docker restart

Advertisement

Answer