Normally request handlers complete requests themselves, if they don't
return an error. For the latter case, the core will complete it for
them.

This is unhandy for pushing opcode handlers further out, as we don't
want a bunch of inline completion code and we don't want to make the
completion path slower than it is now.

Let the core handle any completion, unless the handler explicitly
asks us not to.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

This adds definitions of structs that both the core and the various
opcode handlers need to know about.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This can move request type specific cleanup into a private handler,
removing the need for the core io_uring parts to know what types
they are dealing with.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

They are really just a subset of each other, just use the one type.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

This converts statx, rename, unlink, mkdir, symlink, and hardlink to
use io_cmd_type.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

They all share the same struct io_sync, convert them to use the
io_cmd_type approach instead.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This converts accept, connect, send/recv, sendmsg/recvmsg, shutdown, and
socket to use io_cmd_type.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

There's a special case for recvmsg with MSG_ERRQUEUE set. This is
problematic as it means the core needs to know about this special
request type.

For now, just add a generic flag for it.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Remove struct io_poll_update from io_kiocb, and convert the poll path to
use the io_cmd_type approach instead.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Remove struct io_poll_iocb from io_kiocb, and convert the poll path to
use the io_cmd_type approach instead.

While at it, rename io_poll_iocb to io_poll which is consistent with the
other request type private structures.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Remove struct io_rw from io_kiocb, and convert the read/write path to
use the io_cmd_type approach instead.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Each opcode generally has a command structure in io_kiocb which it can
use to store data associated with that request.

In preparation for having the core layer not know about what's inside
these fields, add a generic io_cmd_data type and put in the union as
well.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Define an io_op_def->prep_async() handler and push the async preparation
to there. Since we now have that, we can drop ->needs_async_setup, as
they mean the same thing.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

In preparation for splitting io_uring up a bit, move it into its own
top level directory. It didn't really belong in fs/ anyway, as it's
not a file system only API.

This adds io_uring/ and moves the core files in there, and updates the
MAINTAINERS file for the new location.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Rather than have two giant switches for doing request preparation and
then for doing request issue, add a prep and issue handler for each
of them in the io_op_defs[] request definition.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

READV cannot recycle buffers as it would lose some of the data required to
reimport that buffer.
Reported-by: NAmmar Faizi <ammarfaizi2@gnuweeb.org>
Fixes: b66e65f4 ("io_uring: never call io_buffer_select() for a buffer re-select")
Signed-off-by: NDylan Yudaken <dylany@fb.com>
Link: https://lore.kernel.org/r/20220721131325.624788-1-dylany@fb.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

in the error path of io_register_pbuf_ring, only free bl if it was
allocated.
Reported-by: NDipanjan Das <mail.dipanjan.das@gmail.com>
Fixes: c7fb1942 ("io_uring: add support for ring mapped supplied buffers")
Signed-off-by: NDylan Yudaken <dylany@fb.com>
Reviewed-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/all/CANX2M5bXKw1NaHdHNVqssUUaBCs8aBpmzRNVEYEvV0n44P7ioA@mail.gmail.com/
Link: https://lore.kernel.org/all/CANX2M5YiZBXU3L6iwnaLs-HHJXRvrxM8mhPDiMDF9Y9sAvOHUA@mail.gmail.com/Signed-off-by: NJens Axboe <axboe@kernel.dk>

If IORING_FILE_INDEX_ALLOC is set asking for an allocated slot, the
helper doesn't check if we actually have a file table or not. The non
alloc path does do that correctly, and returns -ENXIO if we haven't set
one up.

Do the same for the allocated path, avoiding a NULL pointer dereference
when trying to find a free bit.

Fixes: a7c41b46 ("io_uring: let IORING_OP_FILES_UPDATE support choosing fixed file slots")
Signed-off-by: NJens Axboe <axboe@kernel.dk>

32 bit sqe->cmd_op is an union with 64 bit values. It's always a good
idea to do padding explicitly. Also zero check it in prep, so it can be
used in the future if needed without compatibility concerns.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/e6b95a05e970af79000435166185e85b196b2ba2.1657202417.git.asml.silence@gmail.com
[axboe: turn bitwise OR into logical variant]
Signed-off-by: NJens Axboe <axboe@kernel.dk>

io_import_iovec uses the s pointer, but this was changed immediately
after the iovec was re-imported and so it was imported into the wrong
place.

Change the ordering.

Fixes: 2be2eb02 ("io_uring: ensure reads re-import for selected buffers")
Signed-off-by: NDylan Yudaken <dylany@fb.com>
Link: https://lore.kernel.org/r/20220630132006.2825668-1-dylany@fb.com
[axboe: ensure we don't half-import as well]
Signed-off-by: NJens Axboe <axboe@kernel.dk>

We waste a u64 SQE field for flags even though we don't need as many
bits and it can be used for something more useful later. Store io_uring
specific send/recv flags in sqe->ioprio instead of ->addr2.
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Fixes: 0455d4cc ("io_uring: add POLL_FIRST support for send/sendmsg and recv/recvmsg")
[axboe: change comment in io_uring.h as well]
Signed-off-by: NJens Axboe <axboe@kernel.dk>

In prior kernels, we did file assignment always at prep time. This meant
that req->task == current. But after deferring that assignment and then
pushing the inflight tracking back in, we've got the inflight tracking
using current when it should in fact now be using req->task.

Fixup that error introduced by adding the inflight tracking back after
file assignments got modifed.

Fixes: 9cae36a0 ("io_uring: reinstate the inflight tracking")
Signed-off-by: NJens Axboe <axboe@kernel.dk>

We have re-polling for partial IO, so a request can be polled twice. If
it used two poll entries the first time then on the second
io_arm_poll_handler() it will find the old apoll entry and NULL
kmalloc()'ed second entry, i.e. apoll->double_poll, so leaking it.

Fixes: 10c87333 ("io_uring: allow re-poll if we made progress")
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/fee2452494222ecc7f1f88c8fb659baef971414a.1655852245.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Leaving ip.error set when a request was punted to task_work execution is
problematic, don't forget to clear it.

Fixes: aa43477b ("io_uring: poll rework")
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/a6c84ef4182c6962380aebe11b35bdcb25b0ccfb.1655852245.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Don't forget to cancel all linked requests of poll request when
__io_arm_poll_handler() failed.

Fixes: aa43477b ("io_uring: poll rework")
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/a78aad962460f9fdfe4aa4c0b62425c88f9415bc.1655852245.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

apoll_events should be set once in the beginning of poll arming just as
poll->events and not change after. However, currently io_uring resets it
on each __io_poll_execute() for no clear reason. There is also a place
in __io_arm_poll_handler() where we add EPOLLONESHOT to downgrade a
multishot, but forget to do the same thing with ->apoll_events, which is
buggy.

Fixes: 81459350 ("io_uring: cache req->apoll->events in req->cflags")
Signed-off-by: NPavel Begunkov <asml.silence@gmail.com>
Reviewed-by: NHao Xu <howeyxu@tencent.com>
Link: https://lore.kernel.org/r/0aef40399ba75b1a4d2c2e85e6e8fd93c02fc6e4.1655814213.git.asml.silence@gmail.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

With the dropping of the IOPOLL checking in the per-opcode handlers,
we inadvertently left two checks in the recv/recvmsg and send/sendmsg
prep handlers for the same thing, and one of them includes addr2 which
holds the flags for these opcodes.

Fix it up and kill the redundant checks.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

If we mark for reissue, we assume that the buffer will remain stable.
Hence if are using a provided buffer, we need to ensure that we stick
with it for the duration of that request.

This only affects block devices that use provided buffers, as those are
the only ones that get marked with REQ_F_REISSUE.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

io_arm_poll_handler() will recycle the buffer appropriately if we end
up arming poll (or if we're ready to retry), but not for the io-wq case
if we have attempted poll first.

Explicitly recycle the buffer to avoid both hanging on to it too long,
but also to avoid multiple reads grabbing the same one. This can happen
for ring mapped buffers, since it hasn't necessarily been committed.

Fixes: c7fb1942 ("io_uring: add support for ring mapped supplied buffers")
Link: https://github.com/axboe/liburing/issues/605Signed-off-by: NJens Axboe <axboe@kernel.dk>