Add a variable for the number of hash buckets in io_ring_ctx_alloc(),
makes it more readable.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/993926ed0d614ba9a76b2a85bebae2babcb13983.1655371007.git.asml.silence@gmail.comReviewed-by: NHao Xu <howeyxu@tencent.com>
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Don't allocate to many hash/cancellation buckets, there might be too
many, clamp it to 8 bits, or 256 * 64B = 16KB. We don't usually have too
many requests, and 256 buckets should be enough, especially since we
do hash search only in the cancellation path.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/b9620c8072ba61a2d50eba894b89bd93a94a9abd.1655371007.git.asml.silence@gmail.comReviewed-by: NHao Xu <howeyxu@tencent.com>
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Add a new io_hash_bucket structure so that each bucket in cancel_hash
has separate spinlock. Use per entry lock for cancel_hash, this removes
some completion lock invocation and remove contension between different
cancel_hash entries.
Signed-off-by: NHao Xu <howeyxu@tencent.com>
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/05d1e135b0c8bce9d1441e6346776589e5783e26.1655371007.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Clean up io_req_task_complete() and deduplicate io_put_kbuf() calls.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/ae3148ac7eb5cce3e06895cde306e9e959d6f6ae.1655371007.git.asml.silence@gmail.comReviewed-by: NHao Xu <howeyxu@tencent.com>
Signed-off-by: NJens Axboe <axboe@kernel.dk>

REQ_F_COMPLETE_INLINE is only needed to delay queueing into the
completion list to io_queue_sqe() as __io_req_complete() is inlined and
we don't want to bloat the kernel.

As now we complete in a more centralised fashion in io_issue_sqe() we
can get rid of the flag and queue to the list directly.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/600ba20a9338b8a39b249b23d3d177803613dde4.1655371007.git.asml.silence@gmail.comReviewed-by: NHao Xu <howeyxu@tencent.com>
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

io_req_task_complete() enqueues requests for state completion itself, no
need for REQ_F_COMPLETE_INLINE, which is only serve the purpose of not
bloating the kernel.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/aca80f71464ad02c06f1311d998a2d6ee0b31573.1655310733.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

All ctx->check_cq events are slow path, don't test every single flag one
by one in the hot path, but add a common guarding if.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/dff026585cea7ff3a172a7c83894a3b0111bbf6a.1655310733.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Luckily, nnobody completes multi-apoll requests outside the polling
functions, but don't set IO_URING_F_COMPLETE_DEFER in any case as
there is nobody who is catching REQ_F_COMPLETE_INLINE, and so will leak
requests if used.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/a65ed3f5effd9321ee06e6edea294a03be3e15a0.1655310733.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

There is a bunch of inline helpers that will be useful not only to the
core of io_uring, move them to headers.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/22df99c83723e44cba7e945e8519e64e3642c064.1655310733.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

We already have the declarations in opdef.h, move the rest into its own
file rather than in the main io_uring.c file.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Move both the opcodes related to it, and the internals code dealing with
it.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This also helps cleanup the io_uring.h cancel parts, as we can make
things static in the cancel.c file, mostly.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Add a io_poll_issue() rather than export the general task_work locking
and io_issue_sqe(), and put the io_op_defs definition and structure into
a separate header file so that poll can use it.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This kills the last per-op switch.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Remove some dead headers we no longer need, and get rid of the
io_ring_ctx and io_uring_fops forward declarations.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

This also means moving a bit more of the fixed file handling to the
filetable side, which makes sense separately too.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Convert the last spots that check for io_uring_fops to use the provided
helper instead.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

While at it, convert the handlers to just use io_eopnotsupp_prep()
if CONFIG_NET isn't set.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Would be nice to sort out Kconfig for this and don't even compile
epoll.c if we don't have epoll configured.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Add it and use it for the epoll handling, if epoll isn't configured.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

This splits out sync_file_range, fsync, and fallocate.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This splits out splice and tee support.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This splits out renameat, unlinkat, mkdirat, symlinkat, and linkat.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Normally request handlers complete requests themselves, if they don't
return an error. For the latter case, the core will complete it for
them.

This is unhandy for pushing opcode handlers further out, as we don't
want a bunch of inline completion code and we don't want to make the
completion path slower than it is now.

Let the core handle any completion, unless the handler explicitly
asks us not to.
Signed-off-by: NJens Axboe <axboe@kernel.dk>