I am running MPI job in linux server. I got error:
-------------------------------------------------------------------------- The OpenFabrics (openib) BTL failed to initialize while trying to allocate some locked memory. This typically can indicate that the memlock limits are set too low. For most HPC installations, the memlock limits should be set to "unlimited". The failure occured here: Local host: yw0431 OMPI source: ../../../../../ompi/mca/btl/openib/btl_openib_component.c:1216 Function: ompi_free_list_init_ex_new() Device: mlx4_0 Memlock limit: 65536 You may need to consult with your system administrator to get this problem fixed. This FAQ entry on the Open MPI web site may also be helpful: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: yw0431 Local device: mlx4_0 -------------------------------------------------------------------------- [yw0431:20193] 11 more processes have sent help message help-mpi-btl-openib.txt / init-fail-no-mem [yw0431:20193] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [yw0431:20193] 11 more processes have sent help message help-mpi-btl-openib.txt / error in device init forrtl: error (78): process killed (SIGTERM)
it means that my linux server have locked memory with 65M, but my job needed more memory. I think 2G should be emough.
I have found a solution about ulimiting the memory:
ulimit -l unlimited
But i am worried that i will cause system crash or some bad things happen.
so can i set “ulimit -l umlimited”?
Advertisement
Answer
When you set ulimit as unlimited and your process starting using memory exhaustively then OOM killer will kill ur job for system stability,I would set the ulimit as 80 to 90% of RAM of instead of unlimited.