Remove ext2 file with rootfs while it’s already mounted

What happens after mounting filesystem from file? Example: I have rootfs.ext2 file which is located in data directory and mounted under /mnt directory

mount rootfs.ext2 /mnt

After removing rootfs.ext2 I still can use files under /mnt directory, cat file, run binaries, etc.

rm -f rootfs.ext2

I was thinking that rootfs.ext2 file still exists in data directory however it was deleted. I filled whole data directory for test purposes with new data by filling file from /dev/urandom (for rewritting actual data that was before in data directory)

cat /dev/urandom > /data/Filling

Even after filling whole space in data directory I still can access /mnt and run binaries.

The question is what happens with file after mounting it and why I still can moderate throw it? Can I delete rootfs.ext2 (if it’s mounted under /) file without undefined behavior of system(binaries are running, full access to filesystem, etc) Links to documentation are appreciated.

Answer

Linux (and Unix) filesystems have several features that allow that.

Inodes

Data (the thing you get when you run cat) and metadata (what you get from stat and ls) is stored in inodes (“indexed nodes”) which are like a key-value type storage. Inodes are indexed in the sense that an inode is referred to by its ID, the inode number.

That means that the data in rootfs.ext2 is stored in an inode.

Hard Links

Files inside directories are represented as directory entries. A directory entry is a pair of name and inode number.
You can think of directories as hashtables, where the key is the name, and the value is the inode number.
The full path that a directory entry represents is called a hard link to that inode.

That means that multiple directory entries, in different directories or even in the same directory, can point to the same inode number.
You can create that by running:

$ echo hello > x1
$ cat x1
hello
$ ls -li x1
1956 -rw-r----- 1 root root 6 2022-09-03 21:26 x1
$ ln -v x1 x2
'x2' => 'x1'
$ cat x2
hello
$ ll -li x1 x2
1956 -rw-r----- 2 root root 6 2022-09-03 21:26 x1
1956 -rw-r----- 2 root root 6 2022-09-03 21:26 x2

ln, by default, creates a hard link. ls -i prints the inode number, and you can see that in the above example, x1 and x2 have the same inode number, and are therefore both hard links to that inode.
You can also see that the first ls prints 1 before root – that’s the number of hard links that inode 1956 has. You see it increasing to 2 after x2 is created.

What this means is that rootfs.ext2 is a hard link that points to the inode that actually holds the filesystem.

Reference Count

Every inode has a reference count.

When nothing is loaded, the inode’s reference count is equal to its hard link count.

But if the file is opened, the open file is another reference.
For example:

$ exec 8<>x2 # opens x2 for read & write as file descriptor 8
$ cat /proc/self/fd/8 
hello

Because this is reference counting, a file can has 0 hard links, but still have references. Continuing the above example, with the file still open:

$ rm -v x1 x2
removed 'x1'
removed 'x2'
$ ls -li
total 0
$ cat /proc/self/fd/8 
hello

The hard links that point to the inode are gone, but the open file still points to the inode, so the inode is not deleted.

(BTW if you check, you’ll see that /proc/self/fd/8 is actually not another hard link to that inode, but rather a symbolic link. However, the fact that you can still read the inode’s data indicates that the inode wasn’t deleted)

Internally Open Files

Opening a file from userspace, like we did above with exec 8<>x2, is just one way to open files. Many things in the Linux kernel internally open files. For example:

• The swap file is internally open
• When a program is executed, its executable file internally open while the program is running, as well as the dynamically linked libraries it uses.
• As long as a block device is mounted, the inode that represents it is internally open.
• When a socket is created, it is internally represented as an open file.
• When a block device is set to be a loop device, it keeps the backing file open.

Loop Mounts

When you run mount rootfs.ext2 /mnt, what actually happens is that mount creates a block device, e.g. /dev/loop9, then opens rootfs.ext2, and configures /dev/loop9, as a loop device backed by the open file descriptor for rootfs.ext2.

As noted above, that means that as long as the block device is configured as a loop device for that file descriptor, that rootfs.ext2 inode remains open, and therefore with a reference count > 0, and therefore not deleted.

In fact, even if you deleted the loop device itself, the data would still be available, because that block device is also internally open, meaning both the backing regular file (rootfs.ext2) and the block device (/dev/loop9) are kept open:

$ sudo mount rootfs.ext2 /mnt/test/
$ echo hello > /mnt/test/x
$ losetup --list
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                             DIO LOG-SEC
/dev/loop9         0      0         1  0 /tmp/rootfs.ext2                        0     512
$ rm -v rootfs.ext2 
removed 'rootfs.ext2'
$ sudo rm -v /dev/loop9
removed '/dev/loop9'
$ cat /mnt/test/x
hello
$ sudo umount /mnt/test
$ ls /mnt/test/
$

Extra Credit: Open Directories

Inodes contain whatever data and/or metadata is needed. Regular files, like rootfs.ext2, are represented as inodes. But directories are also inodes, as well as block devices, pipes, sockets, etc.

This means that directories have reference counts too, and that they too are opened. Famously via opendir(), but also internally:

• When you call something like open(/etc/passwd), the inode of the root directory (/) is briefly opened to look up etc, and the inode for /etc is briefly opened to loop up passwd.
• The working directory of every process is always internally open – if you delete it from another process, the first process could still run ls in it. However, it will not be able to create new files in it.
• When a directory is a mount point, it is internally open.

You can unmount a mount point that is still in use, because every such “use” is counted as a reference:

$ sudo mount rootfs.ext2 /mnt/test/
$ cd /mnt/test/
$ echo hello > x
$ sudo umount --lazy /mnt/test
$ cat x
hello
$ cd / # reference count of what was mounted on /mnt/test drops to 0
$ cd /mnt/test
$ cat x
cat: x: No such file or directory