The BFQ scheduler and ioprio_check_cap() both assume that the RT
priority class (IOPRIO_CLASS_RT) can have up to 8 different priority
levels, similarly to the BE class (IOPRIO_CLASS_iBE). This is
controlled using the IOPRIO_BE_NR macro , which is badly named as the
number of levels also applies to the RT class.

Introduce the class independent IOPRIO_NR_LEVELS macro, defined to 8,
to make things clear. Keep the old IOPRIO_BE_NR macro definition as an
alias for IOPRIO_NR_LEVELS.
Signed-off-by: NDamien Le Moal <damien.lemoal@wdc.com>
Link: https://lore.kernel.org/r/20210811033702.368488-6-damien.lemoal@wdc.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

The ki_ioprio field of struct kiocb is 16-bits (u16) but often handled
as an int in the block layer. E.g. ioprio_check_cap() takes an int as
argument.

With such implicit int casting function calls, the upper 16-bits of the
int argument may be left uninitialized by the compiler, resulting in
invalid values for the IOPRIO_PRIO_CLASS() macro (garbage upper bits)
and in an error return for functions such as ioprio_check_cap().

Fix this by masking the result of the shift by IOPRIO_CLASS_SHIFT bits
in the IOPRIO_PRIO_CLASS() macro. The new macro IOPRIO_CLASS_MASK
defines the 3-bits mask for the priority class.
Similarly, apply the IOPRIO_PRIO_MASK mask to the data argument of the
IOPRIO_PRIO_VALUE() macro to ignore the upper bits of the data value.
The IOPRIO_CLASS_MASK mask is also applied to the class argument of this
macro before shifting the result by IOPRIO_CLASS_SHIFT bits.

While at it, also change the argument name of the IOPRIO_PRIO_CLASS()
and IOPRIO_PRIO_DATA() macros from "mask" to "ioprio" to reflect the
fact that a priority value should be passed rather than a mask.
Signed-off-by: NDamien Le Moal <damien.lemoal@wdc.com>
Link: https://lore.kernel.org/r/20210811033702.368488-5-damien.lemoal@wdc.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Change the ioprio_valid() macro in include/usapi/linux/ioprio.h to an
inline function declared on the kernel side in include/linux/ioprio.h.
Also improve checks on the class value by checking the upper bound
value.
Signed-off-by: NDamien Le Moal <damien.lemoal@wdc.com>
Link: https://lore.kernel.org/r/20210811033702.368488-4-damien.lemoal@wdc.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

In include/usapi/linux/ioprio.h, change the ioprio class enum comment
to remove the outdated reference to CFQ and mention BFQ and mq-deadline
instead. Also document the high priority NCQ command use for RT class
IOs directed at ATA drives that support NCQ priority.
Signed-off-by: NDamien Le Moal <damien.lemoal@wdc.com>
Link: https://lore.kernel.org/r/20210811033702.368488-3-damien.lemoal@wdc.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

For a request that has a priority level equal to or larger than
IOPRIO_BE_NR, bfq_set_next_ioprio_data() prints a critical warning but
defaults to setting the request new_ioprio field to IOPRIO_BE_NR. This
is not consistent with the warning and the allowed values for priority
levels. Fix this by setting the request new_ioprio field to
IOPRIO_BE_NR - 1, the lowest priority level allowed.

Cc: <stable@vger.kernel.org>
Fixes: aee69d78 ("block, bfq: introduce the BFQ-v0 I/O scheduler as an extra scheduler")
Signed-off-by: NDamien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Link: https://lore.kernel.org/r/20210811033702.368488-2-damien.lemoal@wdc.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Not actually used in any modular code.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210816123649.601591-1-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

seq_get_buf is a crutch that undoes all the memory safety of the
seq_file interface.  Use the normal seq_printf interfaces instead.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NTejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/20210810152623.1796144-2-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Factor out a helper to deal with a single blkcg_gq to make the code a
little bit easier to follow.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NTejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/20210810152623.1796144-1-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NKeith Busch <kbusch@kernel.org>
Link: https://lore.kernel.org/r/20210804095634.460779-16-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-15-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NStefan Haberland <sth@linux.ibm.com>
Link: https://lore.kernel.org/r/20210804095634.460779-14-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-13-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-By: NAnton Ivanov <anton.ivanov@cambridgegreys.com>
Link: https://lore.kernel.org/r/20210804095634.460779-12-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NMartin K. Petersen <martin.petersen@oracle.com>
Link: https://lore.kernel.org/r/20210804095634.460779-11-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.  Note that the existing code is
fine despite ignoring bv_offset as the bio is known to contain exactly
one page from the page allocator per bio_vec.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NColy Li <colyli@suse.de>
Link: https://lore.kernel.org/r/20210804095634.460779-10-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>
Link: https://lore.kernel.org/r/20210804095634.460779-9-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJeff Layton <jlayton@kernel.org>
Link: https://lore.kernel.org/r/20210804095634.460779-8-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-7-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-6-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-5-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

__ebs_rw_bvec use page_address on the submitted bios data, and thus
can't deal with highmem.  Disable the target on highmem configs.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-4-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use the bvec_virt helper to clean up the bio integrity processing a
little bit.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NChaitanya Kulkarni <kch@kernel.org>
Acked-by: NMartin K. Petersen <martin.petersen@oracle.com>
Link: https://lore.kernel.org/r/20210804095634.460779-3-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Add a helper to get the virtual address for a bvec.  This avoids that
all callers need to know about the page + offset representation.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NChaitanya Kulkarni <kch@kernel.org>
Reviewed-by: NMartin K. Petersen <martin.petersen@oracle.com>
Link: https://lore.kernel.org/r/20210804095634.460779-2-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

inode_detach_wb references the "main" bdi of the inode.  With the
recent change to move the bdi from the request_queue to the gendisk
this causes a guaranteed use after free when using certain cgroup
configurations.  The big itself is older through as any non-default
inode reference (e.g. an open file descriptor) could have injected
this use after free even before that.

Fixes: 52ebea74 ("writeback: make backing_dev_info host cgroup-specific bdi_writebacks")
Reported-by: NQian Cai <quic_qiancai@quicinc.com>
Reported-by: Nsyzbot <syzbot+1fb38bb7d3ce0fa3e1c4@syzkaller.appspotmail.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210816122614.601358-3-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

The dev_t is used as the inode hash, so we should only released it
once then block device inode is gone from the inode cache.  Move it
to bdev_free_inode to ensure that.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210816122614.601358-2-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

After patch 54efd50b (block: make generic_make_request handle
arbitrarily sized bios), the IO through io-throttle may be larger,
and these IOs may be further split into more small IOs. However,
IOPS throttle does not seem to be aware of this change, which
makes the calculation of IOPS of large IOs incomplete, resulting
in disk-side IOPS that does not meet expectations. Maybe we should
fix this problem.

We can reproduce it by set max_sectors_kb of disk to 128, set
blkio.write_iops_throttle to 100, run a dd instance inside blkio
and use iostat to watch IOPS:

dd if=/dev/zero of=/dev/sdb bs=1M count=1000 oflag=direct

As a result, without this change the average IOPS is 1995, with
this change the IOPS is 98.
Signed-off-by: NChunguang Xu <brookxu@tencent.com>
Acked-by: NTejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/65869aaad05475797d63b4c3fed4f529febe3c26.1627876014.git.brookxu@tencent.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

bdev_resize_partition can only operate on the whole device.  Make that clear
by passing a gendisk instead of a block_device.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210810154512.1809898-5-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

bdev_del_partition can only operate on the whole device.  Make that clear
by passing a gendisk instead of a block_device.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210810154512.1809898-4-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

bdev_add_partition can only operate on the whole device.  Make that clear
by passing a gendisk instead of a block_device.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210810154512.1809898-3-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Partition scanning only happens on the whole device, so pass a
struct gendisk instead of the whole device block_device to the scanners.
This allows to simplify printing the device name in various places as the
disk name is available in disk->name.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NStefan Haberland <sth@linux.ibm.com>
Link: https://lore.kernel.org/r/20210810154512.1809898-2-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Just check inode_unhashed on the whole device bdev inode instead,
and provide a helper to check for that information.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210809064028.1198327-9-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Let the callers call del_gendisk so that we can check if add_disk
has been called properly for the cached device case instead of relying
on the block layer internal GENHD_FL_UP flag.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NColy Li <colyli@suse.de>
Link: https://lore.kernel.org/r/20210809064028.1198327-8-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Except for the IDA none of the allocations in bcache_device_init is
unwound on error, fix that.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NColy Li <colyli@suse.de>
Link: https://lore.kernel.org/r/20210809064028.1198327-7-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Remove usage of the block layer internal GENHD_FL_UP by just looking
at the host state.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210809064028.1198327-6-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Use the nvme-internal NVME_NSHEAD_DISK_LIVE flag instead of abusing
the block layer state.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210809064028.1198327-5-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Early probe failure never reaches nvme_ns_remove, so GENHD_FL_UP must
be set at this point.  Remove the check.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210809064028.1198327-4-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Restructure mmc_blk_probe to avoid a failure path with a half created
disk.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NUlf Hansson <ulf.hansson@linaro.org>
Link: https://lore.kernel.org/r/20210809064028.1198327-3-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Pass the attribute group for the attributes on the gendisk to
device_add_disk so that they are created atomically with the
disk creation.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NUlf Hansson <ulf.hansson@linaro.org>
Link: https://lore.kernel.org/r/20210809064028.1198327-2-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Move them (PAGE_SECTORS_SHIFT, PAGE_SECTORS and SECTOR_MASK) to the
generic header file to remove redundancy.
Signed-off-by: NGuoqing Jiang <jiangguoqing@kylinos.cn>
Link: https://lore.kernel.org/r/20210721025315.1729118-1-guoqing.jiang@linux.devSigned-off-by: NJens Axboe <axboe@kernel.dk>

Fix the !CONFIG_BLOCK build after the recent cleanup.

Fixes: 5ed964f8 ("mm: hide laptop_mode_wb_timer entirely behind the BDI API")
Reported-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJens Axboe <axboe@kernel.dk>