All of the similar questions I see are resolved by cleaning up the images or containers or orphaned volumes but I am not having any of those problems. I even completely deleted /var/lib/docker
and still nothing.
Relevant output:
[N] ⋊> ~/W/W/cocagne on master ⨯ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v /var/lib/docker:/var/lib/docker martin/docker-cleanup-vol umes docker: Error response from daemon: Container command '/usr/local/bin/docker-cleanup-volumes.sh' not found or does not exist.. [N] ⋊> ~/W/W/cocagne on master ⨯ docker-compose build 11:56:23 mysql uses an image, skipping Building vitess Traceback (most recent call last): File "/usr/bin/docker-compose", line 9, in <module> load_entry_point('docker-compose==1.7.1', 'console_scripts', 'docker-compose')() File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 58, in main command() File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 109, in perform_command handler(command, command_options) File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 213, in build force_rm=bool(options.get('--force-rm', False))) File "/usr/lib/python3.5/site-packages/compose/project.py", line 300, in build service.build(no_cache, pull, force_rm) File "/usr/lib/python3.5/site-packages/compose/service.py", line 718, in build buildargs=build_opts.get('args', None), File "/usr/lib/python3.5/site-packages/docker/api/build.py", line 54, in build path, exclude=exclude, dockerfile=dockerfile, gzip=gzip File "/usr/lib/python3.5/site-packages/docker/utils/utils.py", line 103, in tar t.add(os.path.join(root, path), arcname=path, recursive=False) File "/usr/lib/python3.5/tarfile.py", line 1938, in add self.addfile(tarinfo, f) File "/usr/lib/python3.5/tarfile.py", line 1966, in addfile copyfileobj(fileobj, self.fileobj, tarinfo.size) File "/usr/lib/python3.5/tarfile.py", line 244, in copyfileobj dst.write(buf) File "/usr/lib/python3.5/tempfile.py", line 483, in func_wrapper return func(*args, **kwargs) OSError: [Errno 28] No space left on device [I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -a 11:56:30 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -q 11:57:25 [I] ⋊> ~/W/W/cocagne on master ⨯ docker image -q 11:57:28 docker: 'image' is not a docker command. See 'docker --help'. [I] ⋊> ~/W/W/cocagne on master ⨯ docker images -a 11:57:39 REPOSITORY TAG IMAGE ID CREATED SIZE martin/docker-cleanup-volumes latest 8c41df286c03 12 weeks ago 22.12 MB [I] ⋊> ~/W/W/cocagne on master ⨯ df -h 11:57:41 Filesystem Size Used Avail Use% Mounted on dev 3.9G 0 3.9G 0% /dev run 3.9G 832K 3.9G 1% /run /dev/sda4 27G 9.1G 17G 36% / tmpfs 3.9G 64M 3.8G 2% /dev/shm tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs 3.9G 32K 3.9G 1% /tmp /dev/sda1 42G 16G 25G 39% /home /dev/sda2 42G 9.4G 30G 24% /var /dev/sda5 1.3G 32M 1.3G 3% /boot tmpfs 790M 12K 790M 1% /run/user/1000 [I] ⋊> ~/W/W/cocagne on master ⨯ 11:57:54
docker info
[I] ⋊> ~/W/W/cocagne on master ⨯ docker info 12:01:55 Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 1.11.2 Storage Driver: devicemapper Pool Name: docker-8:2-2359321-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 11.8 MB Data Space Total: 107.4 GB Data Space Available: 34.57 GB Metadata Space Used: 581.6 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.131 (2016-07-15) Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: null host bridge Kernel Version: 4.6.4-1-ARCH Operating System: Arch Linux OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 7.706 GiB Name: crockford ID: HO2U:ELWR:LDB3:PMEY:5YOJ:D7YJ:2HJA:PVYG:45K2:J6KI:D6WO:4RUE Docker Root Dir: /var/lib/docker Debug mode (client): false Debug mode (server): false Registry: https://index.docker.io/v1/
One thing that makes my issue a little different (Where I think the root of the issue comes from)
Before I created a separate partition for /var
, it was on my root partition, which eventually maxed out. Once it maxed out, I shrunk my home partition, create a /var partition, copied my root’s /var
to my new /var
, and removed my old /var
. But for some reason, docker still think’s it’s maxed out? I have no idea.
I also tried to resinstall docker with sudo pacman -S docker
but nothing.
Edit: I just tried it with a normal docker build .
and that works fine. Somehow docker-compose
thinks it’s out of memory though?
Advertisement
Answer
The python stack trace from docker-compose
indicates that it can’t seem to create a temporary file. This would indicate there’s no space left in /tmp
.
OP mentioned that his RAM is completely consumed when he runs docker-compose in the comments. Given that and the fact that /tmp
is mounted on tmpfs
it makes sense that there is no space left for Python/docker-compose
to create any temporary files in /tmp
.
The possible solutions are:
- Temporarily switch the default tempfile generation location by setting one of the following environment variables:
TMPDIR
,TEMP
,TMP
(ref: Python doc) - Change
/tmp
to not usetmpfs
and use disk instead. - Increase the amount of RAM/Swap space on your machine. (You can increase swap without messing with your partitions like so).
tmpfs
is backed by volatile storage, which means both RAM and Swap should theoretically work.
Note, most of these cases will result in a slowdown of your application, especially if the docker build process is I/O heavy.