I’m running a rails server with docker on EC2, it has 64G volume. The web sevice crashed yesterday, I logged in the server and kept getting message about running out of the disk space. I used df -h and du -sh /*/, the result was like:
I deleted some logs to free about 3G space, but it’s full again in around 30 mins. I executed the command du -sh /*/ again, I got the result as below.
I couldn’t see where space is increasing, the /var folder decreasing 3G was the only thing being changed.
Any tips would be appreciated.
Advertisement
Answer
One of the issues could be that processes may be removing some large files but the files may still be on disk, and would be removed when the process gets a SIGHUP or the process is restarted.
You can find such files by doing:
lsof -n | grep -i deleted
This will show you a list of deleted files and the process. You can restart that process to free up the disk or you can send a SIGHUP signal to the process.
To see what’s taking up disk space, you will have to keep a watch on a few things. You can create a cron job that runs every 5 minutes (or every 10 minutes or 30 minutes, you choose) that does:
date >> /tmp/deleted-files.txt && lsof -n | grep -i deleted >> /tmp/deleted-files.txt
Analyze the file and see if files are being created and deleted chronically.
If you have identified the directory that keeps on growing, you can also create a cron job that runs every few minutes to save the file listing in a temporary file like so
date >> /tmp/file-list.txt && ls -ltrh >> /tmp/file-list.txt
That way you can watch the files that are being generated and review their contents. It is possible that someone may be logging in debug mode.
If you are using Ruby on Rails (RoR), Ruby on Rails production log rotation thread can help you set up log rotation. You can be aggressive about log rotation to get a handle on the disk size.
One thing I can tell you is that if you attach an EBS volume worth 200 GB, the cost will be ~$200 for the year and you will have to spend less time urgently on the issue. If your time savings generates more revenue than $200 in the year, getting an EBS volume and storing logs on that would be a much cheaper proposition in the long run.