I’m doing performance tuning of a crypto software, which is run on Linux and utilizes hardware crypto acceleration device.
When the load is given over some threshold, kernel _spn_lock
begin to eat most of the CPU’s time. The following perf top
screenshot shows ~30% of CPU is taken by _spin_lock
, but it goes up over 50% if a load is increased and overall performance degrades.
I can not tune up the performance of the software without finding out which spinlock causes this situation.
I checked up similar questions on stackoverflow, but I cannot find a answer on how to discover which spinlock spins. Is there any way to locate the spinlock position? Any tips or suggestions would be deeply appreciated.
PerfTop: 81 irqs/sec kernel:45.7% exact: 0.0% [1000Hz cycles], (all, CPU: 53) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- samples pcnt function DSO _______ _____ _______________________ __________________________ 44.00 27.2% _spin_lock [kernel.kallsyms] 30.00 18.5% sha512_block_data_order /usr/local/bin/cryptod 6.00 3.7% _int_malloc /lib64/libc-2.12.so 5.00 3.1% __GI___libc_malloc /lib64/libc-2.12.so [root@qat ~]# uname -a Linux qat 2.6.32-220.el6.tsx.65.x86_64 #68 SMP Fri Jan 26 17:45:32 KST 2018 x86_64 x86_64 x86_64 GNU/Linux
Advertisement
Answer
Do you want to find where _spin_lock is called?
Try it ftrace
#cd /sys/kernel/debug/tracing #echo function_graph > current_tracer #cat trace | grep -B 5 _spin_lock
For more information about ftrace, see here https://lwn.net/Articles/365835/