is rmem_default size per socket or for entire stack?

Does setting the net.core.rmem_default effect each socket or all sockets opened in the system? What is the maximum value I can configure for the net.core.rmem_default parameter?

I understand it depends on RAM. Assume I have much RAM available.

Answer

net.core.rmem_default is the size of the incoming kernel socket buffer per one socket.

From man socket(7):

SO_RCVBUF

Sets or gets the maximum socket receive buffer in bytes. The kernel doubles this value (to allow space for bookkeeping overhead) when it is set using setsockopt(2), and this doubled value is returned by getsockopt(2). The default value is set by the /proc/sys/net/core/rmem_default file, and the maximum allowed value is set by the /proc/sys/net/core/rmem_max file. The minimum (doubled) value for this option is 256.

Optimum incoming socket buffer size is the bandwidth-delay product.

In order to minimize recv syscalls your user-space socket buffer size should be not less than that of the kernel buffer. When using non-blocking I/O, having to issue more than one recv syscall to read the kernel socket buffer may result in starvation of other sockets.