We have Ubuntu 18.04 installed machine with an RTX 2080 Ti GPU with about 3-4 users using it remotely. Is it possible to give a maximum threshold GPU usage per user (say 60%) so any other could use the rest?
We are running tensorflow deep learning models if it helps to suggest an alternative.
Advertisement
Answer
My apologies for taking so long to come back here to answer the question, even after figuring out a way to do this.
It is indeed possible to threshold the GPU usage with tensorflow’s per_process_gpu_memory_fraction. [Hence I edited the question]
Following snippet assigns 46% of GPU memory for the user.
init = tf.global_variables_initializer()
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.46)
with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
sess.run(init)
############
#Training happens here#
############
Currently, we have 2 users using the same GPU simultaneously without any issues. We have assigned 46% per each. Make sure you don’t make it 50-50% (aborted, core dumped error occurs if you do so). Try to keep around 300MB memory in idle.
And as a matter of fact, this GPU division does not slow down the training process. Surprisingly, it offers about the same speed as if the full memory is used, at least according to our experience. Although, this may change with high dimensional data.