I work on Linux. How to know that a gzip file is ready? I have a server that polls files in directory /dir/. There is an another, independent process that gzip files to /dir/. How can my server know that file is ready?
Advertisement
Answer
There is no ready-made solution for this. Looking at the last modification timestamp of the file (mtime) is not reliable because writes could delayed if the system is overloaded (or the input to the gzip operation is not ready), or the generating process may stop writing because it has crashed.
Usually, when applications need to do something like this, they write the temporary file under a different name, following a specific pattern. The reading process recognizes the temporary files and skips them, assuming that they are still a work in process and incomplete. Once the writer is finished, it renames the file to its final name (which is an atomic operation), and only then, the reader picks it up. This approach became popular with Dan Bernstein’s maildir format:
In maildir, a different directory is used for staging, but the general principle is the same.
It is also possible to use lock files and POSIX advisory locking, but they lead to more complexity. However, in some cases, they can be employed in such a way that busy waiting/polling/periodic probing is not necessary.