I have a very strange problem with a board based on a MIPS processor and Linux 2.6. There are NIC interrupts for all incoming Ethernet packets. If I send 10.000 packets I could see that 10.000 NIC interrupts occurred.
START SYSTEM
SEND 10k PACKETS
/mnt/system # cat /proc/interrupts
CPU0
24: 10000 MIPS NIC
29: 7192 MIPS timer
30: 0 MIPS UART1
31: 3092 MIPS serial
ERR: 0
However, after I open and close a file (filled with zeros or regular) in the filesystem, there are much fewer NIC interrupts generated. For instance, just 2-7k interrupts for 10k packets. It has a detrimental effect on the system performance, but after rebooting everything with NIC interrupts is fine again.
START SYSTEM
std::fstream f;
f.open("/mnt/system/myfile");
f.close();
WAIT FOR SOME TIME
SEND 10k PACKETS
/mnt/system # cat /proc/interrupts
CPU0
24: 2045 MIPS NIC
29: 7192 MIPS timer
30: 0 MIPS UART1
31: 3092 MIPS serial
ERR: 0
The filesystem is jffs2 and the flash drive is 32M NOR serial device. Why reading a file kills NIC interrupts until reboot?
Advertisement
Answer
Caveat: This may not be a total solution, but I have some ideas. More information/testing may be necessary.
When you do [the system does] nothing else, the NIC driver is fast enough to respond to an interrupt and the ISR will complete processing on the single packet before the next packet arrives. (i.e.) There is a one-to-one relationship between incoming packets and NIC interrupts.
If you have other system activity, this may delay entry to the NIC driver’s ISR. Also, this other activity may slow down processing in the NIC driver due to resource competition (e.g. locks, kmalloc, etc.)
Meanwhile, more NIC packets arrive. When the NIC ISR is finally/ultimately entered, it sees that there are multiple packets pending. It will process all of them, without leaving the ISR. So, it might process (e.g.) 5 packets on each interrupt, so the interrupt count is reduced by 5x.
This is what most “smart” NIC cards and drivers do. It actually improves throughput with heavy NIC traffic. Ordinarily, fewer NIC interrupts is a good thing because it reduces the wasted overhead of entering/exiting the ISR repeatedly.
This really depends upon the packet arrival rate and [average] interval.
So, the real question is: “When the interrupt count is reduced, do you lose packets/data or just see performance loss?”
And, how are you measuring the system performance difference?
Is the filesystem in question used otherwise (e.g. it is a root fs)? Or, is your program opening the file the only access to the fs [other than mounting it]
I’m not familiar with jffs2 or your flash driver, so I’d ask if you suspect either of doing something that would disable interrupts for an extended period of time?
UPDATE:
I just noticed that you’re using linux version 2.6 That kernel is approximately 10 years old. It’s well past end-of-life, so it probably isn’t getting bug fixes for drivers unless the platform vendor is providing them.
So, another thing to consider is that any of the drivers may have a bug that causes the performance problem. There’s a fair to good chance that the drivers have been fixed in more modern kernels.
You might want to switch to a newer kernel if you can. If not, you may be left with the [unenviable] task of backporting the newer drivers to the older kernel [or at least cherry pick some of the bug fixes].