Subprocess – Too many open Files in subprocess

On my system(using python3.6.9) I got the Too many open Files Error.

I got the error while executing a subprocess in python.

Traceback:

File "/opt/KIDICAP/docengine/Objects/Watcher.py", line 99, in watch, self.check_ulimit()
File "/opt/KIDICAP/docengine/Objects/Watcher.py", line 469, in check_ulimit
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__    restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1254, in _execute_child    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

I tried to look it up but permanently setting the ulimit didn’t work.

I even build a function, that sets the ulimit before executing a subprocess.

        process = subprocess.Popen(comand1, shell=True, stdout=subprocess.PIPE)
        output = process.stdout.readlines()
        process.stdout.close()
        process.terminate()
        # print(output)

        # logger.info(comand2)
        process = subprocess.Popen(comand2, shell=True, stdout=subprocess.PIPE)
        output = process.stdout.readlines()
        process.stdout.close()
        process.terminate()
        # print(output)

        # logger.info(comand3)
        process = subprocess.Popen(comand3, shell=True, stdout=subprocess.PIPE)
        output = process.stdout.readlines()
        process.stdout.close()
        process.terminate()

Answer

I didn’t find the right solution on the Internet.

The solution for me was to edit the fs.file max (/proc/sys/fs/file-max)

I put the value in the file on 1000000.

and to edit the bash.bashrc file to increase the ulimit in every session opened.

/etc/bash.bashrc

There i added:

ulimit -n 1000000
ulimit -s unlimited