mirror of https://github.com/torvalds/linux.git
f7bc22ca0d
When multiple io_uring rings poll on the same NVMe queue, one ring can find completions belonging to another ring. The current code always uses task_work to handle this, but this adds overhead for the common single-ring case. This patch passes the polling io_ring_ctx through io_comp_batch's new poll_ctx field. In io_do_iopoll(), the polling ring's context is stored in iob.poll_ctx before calling the iopoll callbacks. In nvme_uring_cmd_end_io(), we now compare iob->poll_ctx with the request's owning io_ring_ctx (via io_uring_cmd_ctx_handle()). If they match (local context), we complete inline with io_uring_cmd_done32(). If they differ (remote context) or iob is NULL (non-iopoll path), we use task_work as before. This optimization eliminates task_work scheduling overhead for the common case where a ring polls and finds its own completions. ~10% IOPS improvement is observed in the following benchmark: fio/t/io_uring -b512 -d128 -c32 -s32 -p1 -F1 -O0 -P1 -u1 -n1 /dev/ng0n1 Signed-off-by: Ming Lei <ming.lei@redhat.com> Reviewed-by: Kanchan Joshi <joshi.k@samsung.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> |
||
|---|---|---|
| .. | ||
| Kconfig | ||
| Makefile | ||
| advise.c | ||
| advise.h | ||
| alloc_cache.c | ||
| alloc_cache.h | ||
| cancel.c | ||
| cancel.h | ||
| cmd_net.c | ||
| epoll.c | ||
| epoll.h | ||
| eventfd.c | ||
| eventfd.h | ||
| fdinfo.c | ||
| fdinfo.h | ||
| filetable.c | ||
| filetable.h | ||
| fs.c | ||
| fs.h | ||
| futex.c | ||
| futex.h | ||
| io-wq.c | ||
| io-wq.h | ||
| io_uring.c | ||
| io_uring.h | ||
| kbuf.c | ||
| kbuf.h | ||
| memmap.c | ||
| memmap.h | ||
| mock_file.c | ||
| msg_ring.c | ||
| msg_ring.h | ||
| napi.c | ||
| napi.h | ||
| net.c | ||
| net.h | ||
| nop.c | ||
| nop.h | ||
| notif.c | ||
| notif.h | ||
| opdef.c | ||
| opdef.h | ||
| openclose.c | ||
| openclose.h | ||
| poll.c | ||
| poll.h | ||
| query.c | ||
| query.h | ||
| refs.h | ||
| register.c | ||
| register.h | ||
| rsrc.c | ||
| rsrc.h | ||
| rw.c | ||
| rw.h | ||
| slist.h | ||
| splice.c | ||
| splice.h | ||
| sqpoll.c | ||
| sqpoll.h | ||
| statx.c | ||
| statx.h | ||
| sync.c | ||
| sync.h | ||
| tctx.c | ||
| tctx.h | ||
| timeout.c | ||
| timeout.h | ||
| truncate.c | ||
| truncate.h | ||
| uring_cmd.c | ||
| uring_cmd.h | ||
| waitid.c | ||
| waitid.h | ||
| xattr.c | ||
| xattr.h | ||
| zcrx.c | ||
| zcrx.h | ||