Would be nice to sort out Kconfig for this and don't even compile
epoll.c if we don't have epoll configured.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Add it and use it for the epoll handling, if epoll isn't configured.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

This splits out sync_file_range, fsync, and fallocate.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This splits out splice and tee support.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This splits out renameat, unlinkat, mkdirat, symlinkat, and linkat.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Normally request handlers complete requests themselves, if they don't
return an error. For the latter case, the core will complete it for
them.

This is unhandy for pushing opcode handlers further out, as we don't
want a bunch of inline completion code and we don't want to make the
completion path slower than it is now.

Let the core handle any completion, unless the handler explicitly
asks us not to.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

This adds definitions of structs that both the core and the various
opcode handlers need to know about.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This can move request type specific cleanup into a private handler,
removing the need for the core io_uring parts to know what types
they are dealing with.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

They are really just a subset of each other, just use the one type.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

This converts statx, rename, unlink, mkdir, symlink, and hardlink to
use io_cmd_type.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NJens Axboe <axboe@kernel.dk>

They all share the same struct io_sync, convert them to use the
io_cmd_type approach instead.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

This converts accept, connect, send/recv, sendmsg/recvmsg, shutdown, and
socket to use io_cmd_type.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

There's a special case for recvmsg with MSG_ERRQUEUE set. This is
problematic as it means the core needs to know about this special
request type.

For now, just add a generic flag for it.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Remove struct io_poll_update from io_kiocb, and convert the poll path to
use the io_cmd_type approach instead.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Remove struct io_poll_iocb from io_kiocb, and convert the poll path to
use the io_cmd_type approach instead.

While at it, rename io_poll_iocb to io_poll which is consistent with the
other request type private structures.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Remove struct io_rw from io_kiocb, and convert the read/write path to
use the io_cmd_type approach instead.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Each opcode generally has a command structure in io_kiocb which it can
use to store data associated with that request.

In preparation for having the core layer not know about what's inside
these fields, add a generic io_cmd_data type and put in the union as
well.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Define an io_op_def->prep_async() handler and push the async preparation
to there. Since we now have that, we can drop ->needs_async_setup, as
they mean the same thing.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

In preparation for splitting io_uring up a bit, move it into its own
top level directory. It didn't really belong in fs/ anyway, as it's
not a file system only API.

This adds io_uring/ and moves the core files in there, and updates the
MAINTAINERS file for the new location.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Rather than have two giant switches for doing request preparation and
then for doing request issue, add a prep and issue handler for each
of them in the io_op_defs[] request definition.
Signed-off-by: NJens Axboe <axboe@kernel.dk>

READV cannot recycle buffers as it would lose some of the data required to
reimport that buffer.
Reported-by: NAmmar Faizi <ammarfaizi2@gnuweeb.org>
Fixes: b66e65f4 ("io_uring: never call io_buffer_select() for a buffer re-select")
Signed-off-by: NDylan Yudaken <dylany@fb.com>
Link: https://lore.kernel.org/r/20220721131325.624788-1-dylany@fb.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

in the error path of io_register_pbuf_ring, only free bl if it was
allocated.
Reported-by: NDipanjan Das <mail.dipanjan.das@gmail.com>
Fixes: c7fb1942 ("io_uring: add support for ring mapped supplied buffers")
Signed-off-by: NDylan Yudaken <dylany@fb.com>
Reviewed-by: NPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/all/CANX2M5bXKw1NaHdHNVqssUUaBCs8aBpmzRNVEYEvV0n44P7ioA@mail.gmail.com/
Link: https://lore.kernel.org/all/CANX2M5YiZBXU3L6iwnaLs-HHJXRvrxM8mhPDiMDF9Y9sAvOHUA@mail.gmail.com/Signed-off-by: NJens Axboe <axboe@kernel.dk>

If IORING_FILE_INDEX_ALLOC is set asking for an allocated slot, the
helper doesn't check if we actually have a file table or not. The non
alloc path does do that correctly, and returns -ENXIO if we haven't set
one up.

Do the same for the allocated path, avoiding a NULL pointer dereference
when trying to find a free bit.

Fixes: a7c41b46 ("io_uring: let IORING_OP_FILES_UPDATE support choosing fixed file slots")
Signed-off-by: NJens Axboe <axboe@kernel.dk>