Design of multi-threaded server in c

When trying to implement a simple echo server with concurrent support on linux.

Following approaches are used:

• Use pthread functions to create a pool of thread, and maintained in a linked list. It’s created on process start, and destroy on process termination.
• Main thread will accept request, and use a POSIX message queue to store accepted socket file descriptor.
• Threads in pool loop to read from message queue, and handle request it gets, when there is no request, it will block.

The program seems working now.

The questions are:

• Is it suitable to use message queue in the middle, is it efficient enough?
• What is the general approach to accomplish a thread tool that needs to handle concurrent request from multiple clients?
• If it’s not proper to make threads in pool loop & block to retrieve msg from message queue, then how to deliver requests to threads?

Answer

This seems unneccesarily complicated to me. The usual approach for a multithreaded server is:

• Create a listen-socket in a thread process
• Accept the client-connections in a thread
• For each accepted client connection, create a new threads, which receives the corresponding file descriptor and does the work
• The worker thread closes the client connection, when it is fully handled

I do not see much benefit in prepopulating a thread-pool here.

If you really want a threadpool:

I would just use a linked list for accepted connections and a pthread_mutex to synchronize access to it:

• The listener-process enqueues client fds at the tail of the list.
• The clients dequeue it at the head.

If the queue is empty, the thread can wait on a variable (pthread_cond_wait) and are notified by the listener process (pthread_cond_signal) when connections are available.

Another alternative

Depending on the complexity of handling requests, it might be an option to make the server single-threaded, i.e. handle all connections in one thread. This eliminates context-switches altogether and can thus be very performant.

One drawback is, that only one CPU-core is used. To improve that, a hybrid-model can be used:

• Create one worker-thread per core.
• Each thread handles simultaneously n connections.

You would however have to implement mechanisms to distribute the work fairly amongst the workers.