Info: MySQL Community 5.7.12 64 bits
Programming Language: Python 3.5
mysqld.cnf: Mysqld.cnf
result of “show variables” show variables
Description: – I have 200 000 tasks. – I process these tasks with 500 threads, one per task, when a thread finishes, another task start. The point of this is not to have more than 500 threads running. – I am creating a connection per thread. (pymysql) – Every one of this thread does about 350 inserts into a table whit the query “insert ignore”. – This are very basic inserts in a table with 12 columns.
I have a server with the following specs: SO: Ubuntu 16.04 LTS RAM: 32 GB Procesor: 6 core 2.66 GHz Number of procesors: 2 HDD: 10K RPM SAS 2.5″ Virtual Cores: 24 cores.
The problem I have is that is taking a lot of time to do the inserts in database, with a average of 1000 seconds per thread.
How Can I improve this? with this question I put the links of mysqld.cnf and the result of show variables y csv format.
Please any advise?
Thanks!
Advertisement
Answer
Thanks to the comment of Jared Beekman, I was able to optmize this. I put together all the queries and the result was a 100 time faster process.