JDK 1.8 -XX:+UseLargePages behavior when there’s not enough huge pages left on os

I am currently confusing how to optimize using HugePages with JVM applications with Netty, -XX:+UseLargePages option enabled, and using G1Gc.
Also, I didn’t forget to set the same max and min size of the heap and metaspace.

My application looks fine, but I was wondering what happens if there’s no remaining free huge pages on system since JVM uses additional native memory area to allocate direct memory buffer, etc.
(Assume that application started up normally, and consumes additional HugePages on off-heap memory area.)

I’ve read following page, but there’s no description of the behavior when JVM failed to allocate huge pages. https://www.oracle.com/java/technologies/javase/largememory-pages.html

I use CentOS 7 and OpenJDK 1.8.0_151-b12 for the testbed before deployment.

Answer

If allocating large pages fails, OpenJDK 8 or later falls back to allocating regular pages.

src/hotspot/share/memory/virtualspace.cpp:

    if (base != NULL) {
[...]
    } else {
      // failed; try to reserve regular memory below
      if (UseLargePages && (!FLAG_IS_DEFAULT(UseLargePages) ||
                            !FLAG_IS_DEFAULT(LargePageSizeInBytes))) {
        log_debug(gc, heap, coops)("Reserve regular memory without large pages");
      }
    }

All GC implementations use the ReservedSpace helper for allocating memory, so this is not GC-specific.

You can easily test that behavior on Linux by restricting available large pages:

$ echo 16 > /proc/sys/vm/nr_hugepages
$ cat /proc/meminfo  | grep HugePages
AnonHugePages:     40960 kB
HugePages_Total:      16
HugePages_Free:       16
HugePages_Rsvd:        0
HugePages_Surp:        0
$ java -XX:+UseLargePages Test
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 251658240 (errno = 12).
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000707c00000 bytes: 4164943872 (errno = 12).
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 67108864 (errno = 12).
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 67108864 (errno = 12).
$ echo $?
0

strace confirms the failed allocation attempt and the successful retry with the same size but without MAP_HUGETLB:

11631 mmap(NULL, 251658240, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0) = -1 ENOMEM (Cannot allocate memory)
11631 mmap(NULL, 251658240, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f35d489c000