How to find out which kernel spinlock eat up most of CPU?

I’m doing performance tuning of a crypto software, which is run on Linux and utilizes hardware crypto acceleration device.

When the load is given over some threshold, kernel _spn_lock begin to eat most of the CPU’s time. The following perf top screenshot shows ~30% of CPU is taken by _spin_lock, but it goes up over 50% if a load is increased and overall performance degrades.

I can not tune up the performance of the software without finding out which spinlock causes this situation.

I checked up similar questions on stackoverflow, but I cannot find a answer on how to discover which spinlock spins. Is there any way to locate the spinlock position? Any tips or suggestions would be deeply appreciated.

   PerfTop:      81 irqs/sec  kernel:45.7%  exact:  0.0% [1000Hz cycles],  (all, CPU: 53)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                DSO
             _______ _____ _______________________ __________________________

               44.00 27.2% _spin_lock              [kernel.kallsyms]
               30.00 18.5% sha512_block_data_order /usr/local/bin/cryptod
                6.00  3.7% _int_malloc             /lib64/libc-2.12.so
                5.00  3.1% __GI___libc_malloc      /lib64/libc-2.12.so



[root@qat ~]# uname -a
Linux qat 2.6.32-220.el6.tsx.65.x86_64 #68 SMP Fri Jan 26 17:45:32 KST 2018 x86_64 x86_64 x86_64 GNU/Linux

Answer

Do you want to find where _spin_lock is called?

Try it ftrace

#cd /sys/kernel/debug/tracing
#echo function_graph > current_tracer
#cat trace | grep -B 5 _spin_lock

For more information about ftrace, see here https://lwn.net/Articles/365835/