I’m using io_submit(..., nr, ...) with nr up to 128 but I usually get fewer requests submitted.
According to the manual IO_SUBMIT(2), this is legit but I wonder: why? Also, is there a way to know which request was submitted right away – without checking io_getevents()?
From the manual:
On success, io_submit() returns the number of iocbs submitted (which may be less than nr, or 0 if nr is zero). For the failure return, see NOTES.
Advertisement
Answer
One possible reason you’re getting fewer results submitted is that you’ve reached a limit of 128 requests outstanding. Maybe you called io_submit with 128, and then call it again before all 128 requests have completed? See the nr_requests entry in sysfs for your device for the limit on the number of requests its willing to handle:
# cat /sys/block/sda/queue/nr_requests 128
It’s up to your code to track how many requests are outstanding and to avoid trying to call io_submit with a number of requests that’d push it over this limit.
Now, that might not be why your call is submitting fewer requests than you asked. If you look at the kernel source, in fs/aio.c, you can see all the failure conditions in __io_submit_one that could trigger a short response. There could be argument problems with one of the requests, or maybe it can’t allocate slab memory for the kiocb, etc. But most likely, it’s aio_get_req that’s failing due to that nr_requests queue depth limit.
If you look at the SYSCALL_DEFINE3(io_submit... in that same file, you can see it’s straightforward to figure out which requests weren’t submitted. It’s simply the position in the array you handed to io_submit. For example, if io_submit returns 5, then you know that the requests that you submitted in array indices 0-4 have been submitted. Anything between 5 and nr - 1 has not been submitted.