It is a bit confusing that there is BLKDEV_MAX_RQ and MAX_SCHED_RQ, as
the name BLKDEV_MAX_RQ would imply the max requests always, which it is
not.

Rename to BLKDEV_MAX_RQ to BLKDEV_DEFAULT_RQ, matching its usage - that being
the default number of requests assigned when allocating a request queue.
Signed-off-by: NJohn Garry <john.garry@huawei.com>
Reviewed-by: NMing Lei <ming.lei@redhat.com>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Link: https://lore.kernel.org/r/1633429419-228500-3-git-send-email-john.garry@huawei.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

The original code in commit 24d2f903 ("blk-mq: split out tag
initialization, support shared tags") would check tags->rqs is non-NULL and
then dereference tags->rqs[].

Then in commit 2af8cbe3 ("blk-mq: split tag ->rqs[] into two"), we
started to dereference tags->static_rqs[], but continued to check non-NULL
tags->rqs.

Check tags->static_rqs as non-NULL instead, which is more logical.
Signed-off-by: NJohn Garry <john.garry@huawei.com>
Reviewed-by: NMing Lei <ming.lei@redhat.com>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Link: https://lore.kernel.org/r/1633429419-228500-2-git-send-email-john.garry@huawei.comSigned-off-by: NJens Axboe <axboe@kernel.dk>

Make the bad sector information a little more useful by printing
current->comm to identify the caller.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NDamien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: NChaitanya Kulkarni <kch@nvidia.com>
Link: https://lore.kernel.org/r/20210928052755.113016-1-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

In addition to reverting commit 7b05bf77 ("Revert "block/mq-deadline:
Prioritize high-priority requests""), this patch uses 'jiffies' instead
of ktime_get() in the code for aging lower priority requests.

This patch has been tested as follows:

Measured QD=1/jobs=1 IOPS for nullb with the mq-deadline scheduler.
Result without and with this patch: 555 K IOPS.

Measured QD=1/jobs=8 IOPS for nullb with the mq-deadline scheduler.
Result without and with this patch: about 380 K IOPS.

Ran the following script:

set -e
scriptdir=$(dirname "$0")
if [ -e /sys/module/scsi_debug ]; then modprobe -r scsi_debug; fi
modprobe scsi_debug ndelay=1000000 max_queue=16
sd=''
while [ -z "$sd" ]; do
  sd=$(basename /sys/bus/pseudo/drivers/scsi_debug/adapter*/host*/target*/*/block/*)
done
echo $((100*1000)) > "/sys/block/$sd/queue/iosched/prio_aging_expire"
if [ -e /sys/fs/cgroup/io.prio.class ]; then
  cd /sys/fs/cgroup
  echo restrict-to-be >io.prio.class
  echo +io > cgroup.subtree_control
else
  cd /sys/fs/cgroup/blkio/
  echo restrict-to-be >blkio.prio.class
fi
echo $$ >cgroup.procs
mkdir -p hipri
cd hipri
if [ -e io.prio.class ]; then
  echo none-to-rt >io.prio.class
else
  echo none-to-rt >blkio.prio.class
fi
{ "${scriptdir}/max-iops" -a1 -d32 -j1 -e mq-deadline "/dev/$sd" >& ~/low-pri.txt & }
echo $$ >cgroup.procs
"${scriptdir}/max-iops" -a1 -d32 -j1 -e mq-deadline "/dev/$sd" >& ~/hi-pri.txt

Result:
* 11000 IOPS for the high-priority job
*    40 IOPS for the low-priority job

If the prio aging expiry time is changed from 100s into 0, the IOPS results
change into 6712 and 6796 IOPS.

The max-iops script is a script that runs fio with the following arguments:
--bs=4K --gtod_reduce=1 --ioengine=libaio --ioscheduler=${arg_e} --runtime=60
--norandommap --rw=read --thread --buffered=0 --numjobs=${arg_j}
--iodepth=${arg_d} --iodepth_batch_submit=${arg_a}
--iodepth_batch_complete=$((arg_d / 2)) --name=${positional_argument_1}
--filename=${positional_argument_1}

Cc: Damien Le Moal <damien.lemoal@wdc.com>
Cc: Niklas Cassel <Niklas.Cassel@wdc.com>
Cc: Hannes Reinecke <hare@suse.de>
Signed-off-by: NBart Van Assche <bvanassche@acm.org>
Reviewed-by: NDamien Le Moal <damien.lemoal@opensource.wdc.com>
Link: https://lore.kernel.org/r/20210927220328.1410161-5-bvanassche@acm.org
[axboe: @latest -> @latest_start]
Signed-off-by: NJens Axboe <axboe@kernel.dk>

Calculating the sum over all CPUs of per-CPU counters frequently is
inefficient. Hence switch from per-CPU to individual counters. Three
counters are protected by the mq-deadline spinlock since these are
only accessed from contexts that already hold that spinlock. The fourth
counter is atomic because protecting it with the mq-deadline spinlock
would trigger lock contention.
Reviewed-by: NDamien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: NNiklas Cassel <Niklas.Cassel@wdc.com>
Cc: Hannes Reinecke <hare@suse.de>
Signed-off-by: NBart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20210927220328.1410161-4-bvanassche@acm.orgSigned-off-by: NJens Axboe <axboe@kernel.dk>

Check a statistics invariant at module unload time. When running
blktests, the invariant is verified every time a request queue is
removed and hence is verified at least once per test.
Reviewed-by: NDamien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: NNiklas Cassel <Niklas.Cassel@wdc.com>
Cc: Hannes Reinecke <hare@suse.de>
Signed-off-by: NBart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20210927220328.1410161-3-bvanassche@acm.orgSigned-off-by: NJens Axboe <axboe@kernel.dk>

The scheduler .insert_requests() callback is called when a request is
queued for the first time and also when it is requeued. Only count a
request the first time it is queued. Additionally, since the mq-deadline
scheduler only performs zone locking for requests that have been
inserted, skip the zone unlock code for requests that have not been
inserted into the mq-deadline scheduler.

Fixes: 38ba64d1 ("block/mq-deadline: Track I/O statistics")
Reviewed-by: NDamien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: NNiklas Cassel <Niklas.Cassel@wdc.com>
Cc: Hannes Reinecke <hare@suse.de>
Signed-off-by: NBart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20210927220328.1410161-2-bvanassche@acm.orgSigned-off-by: NJens Axboe <axboe@kernel.dk>

struct request is only used by blk-mq drivers, so move it and all
related declarations to blk-mq.h.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-18-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Split the integrity/metadata handling definitions out into a new header.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-17-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

These are block-layer internal helpers, so move them to block/blk.h and
block/blk-merge.c.  Also update a comment a bit to use better grammar.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-16-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Drop various include not actually used in genhd.h itself, and
move the remaning includes closer together.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-15-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Drop various include not actually used in blkdev.h itself.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-14-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Except for the features passed to blk_queue_required_elevator_features,
elevator.h is only needed internally to the block layer.  Move the
ELEVATOR_F_* definitions to blkdev.h, and the move elevator.h to
block/, dropping all the spurious includes outside of that.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-13-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

This type doesn't exist at all, so no need to forward declare it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-12-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Entirely unused.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-11-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-10-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-9-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Only core.c needs blkdev.h, so move the #include statement there.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-8-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Various files have acquired spurious includes of <linux/blkdev.h> over
time.  Remove them.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-7-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Various files have acquired spurious includes of <linux/blkdev.h> over
time.  Remove them.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-6-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Various files have acquired spurious includes of <linux/blkdev.h> over
time.  Remove them.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-5-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

Move inode_to_bdi out of line to avoid having to include blkdev.h.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-4-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

There is no need to pull blk-cgroup.h and thus blkdev.h in here, so
break the include chain.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-3-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

blk-cgroup.h pulls in blkdev.h and thus pretty much all the block
headers.  Break this dependency chain by turning wbc_blkcg_css into a
macro and dropping the blk-cgroup.h include.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://lore.kernel.org/r/20210920123328.1399408-2-hch@lst.deSigned-off-by: NJens Axboe <axboe@kernel.dk>

c3df5fb5 ("cgroup: rstat: fix A-A deadlock on 32bit around
u64_stats_sync") made u64_stats updates irq-safe to avoid A-A deadlocks.
Unfortunately, the conversion missed one in blk_cgroup_bio_start(). Fix it.

Fixes: 2d146aa3 ("mm: memcontrol: switch to rstat")
Cc: stable@vger.kernel.org # v5.13+
Reported-by: syzbot+9738c8815b375ce482a1@syzkaller.appspotmail.com
Signed-off-by: NTejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/YWi7NrQdVlxD6J9W@slm.duckdns.orgSigned-off-by: NJens Axboe <axboe@kernel.dk>

Pull libata fixes from Damien Le Moal:
 "Two fixes for this cycle:

   - Fix a null pointer dereference in ahci-platform driver (from Hai)

   - Fix uninitialized variables in pata_legacy driver (from Dan)"

* tag 'libata-5.15-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/libata:
  ata: ahci_platform: fix null-ptr-deref in ahci_platform_enable_regulators()
  pata_legacy: fix a couple uninitialized variable bugs

Pull block fixes from Jens Axboe:
 "Bigger than usual for this point in time, the majority is fixing some
  issues around BDI lifetimes with the move from the request_queue to
  the disk in this release. In detail:

   - Series on draining fs IO for del_gendisk() (Christoph)

   - NVMe pull request via Christoph:
        - fix the abort command id (Keith Busch)
        - nvme: fix per-namespace chardev deletion (Adam Manzanares)

   - brd locking scope fix (Tetsuo)

   - BFQ fix (Paolo)"

* tag 'block-5.15-2021-10-17' of git://git.kernel.dk/linux-block:
  block, bfq: reset last_bfqq_created on group change
  block: warn when putting the final reference on a registered disk
  brd: reduce the brd_devices_mutex scope
  kyber: avoid q->disk dereferences in trace points
  block: keep q_usage_counter in atomic mode after del_gendisk
  block: drain file system I/O on del_gendisk
  block: split bio_queue_enter from blk_queue_enter
  block: factor out a blk_try_enter_queue helper
  block: call submit_bio_checks under q_usage_counter
  nvme: fix per-namespace chardev deletion
  block/rnbd-clt-sysfs: fix a couple uninitialized variable bugs
  nvme-pci: Fix abort command id

Pull io_uring fix from Jens Axboe:
 "Just a single fix for a wrong condition for grabbing a lock, a
  regression in this merge window"

* tag 'io_uring-5.15-2021-10-17' of git://git.kernel.dk/linux-block:
  io_uring: fix wrong condition to grab uring lock

Pull virtio fixes from Michael Tsirkin:
 "Fixes up some issues in rc5"

* tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost:
  vhost-vdpa: Fix the wrong input in config_cb
  VDUSE: fix documentation underline warning
  Revert "virtio-blk: Add validation for block size in config space"
  vhost_vdpa: unset vq irq before freeing irq
  virtio: write back F_VERSION_1 before validate

Pull powerpc fixes from Michael Ellerman:

 - Fix a bug where guests on P9 with interrupts passed through could get
   stuck in synchronize_irq().

 - Fix a bug in KVM on P8 where secondary threads entering a guest would
   write outside their allocated stack.

 - Fix a bug in KVM on P8 where secondary threads could confuse the host
   offline code and cause the guest or host to crash.

Thanks to Cédric Le Goater.

* tag 'powerpc-5.15-4' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux:
  KVM: PPC: Book3S HV: Make idle_kvm_start_guest() return 0 if it went to guest
  KVM: PPC: Book3S HV: Fix stack handling in idle_kvm_start_guest()
  powerpc/xive: Discard disabled interrupts in get_irqchip_state()

Pull objtool fixes from Borislav Petkov:

 - Update section headers before the respective relocations to not
   trigger a safety check in elftoolchain's implementation of libelf

 - Do not add garbage data to the .rela.orc_unwind_ip section

* tag 'objtool_urgent_for_v5.15_rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  objtool: Update section header before relocations
  objtool: Check for gelf_update_rel[a] failures

Pull EDAC fix from Borislav Petkov:

 - Log the "correct" uncorrectable error count in the armada_xp driver

* tag 'edac_urgent_for_v5.15_rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras:
  EDAC/armada-xp: Fix output of uncorrectable error counter

Pull perf fix from Borislav Petkov:

 - Add Sapphire Rapids to the list of CPUs supporting the SMI count MSR

* tag 'perf_urgent_for_v5.15_rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  perf/x86/msr: Add Sapphire Rapids CPU support

Pull EFI fixes from Borislav Petkov:
 "Forwarded from Ard Biesheuvel through the tip tree. Ard will send
  stuff directly in the near future.

  Low priority fixes but fixes nonetheless:

   - update stub diagnostic print that is no longer accurate

   - avoid statically allocated buffer for CPER error record decoding

   - avoid sleeping on the efi_runtime semaphore when calling the
     ResetSystem EFI runtime service"

* tag 'efi-urgent-for-v5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  efi: Change down_interruptible() in virt_efi_reset_system() to down_trylock()
  efi/cper: use stack buffer for error record decoding
  efi/libstub: Simplify "Exiting bootservices" message

Pull x86 fixes from Borislav Petkov:

 - Do not enable AMD memory encryption in Kconfig by default due to
   shortcomings of some platforms, leading to boot failures.

 - Mask out invalid bits in the MXCSR for 32-bit kernels again because
   Thomas and I don't know how to mask out bits properly. Third time's
   the charm.

* tag 'x86_urgent_for_v5.15_rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/fpu: Mask out the invalid MXCSR bits properly
  x86/Kconfig: Do not enable AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT automatically

Merge tag 'driver-core-5.15-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core

Pull driver core fixes from Greg KH:
 "Here are some small driver core fixes for 5.15-rc6, all of which have
  been in linux-next for a while with no reported issues.

  They include:

   - kernfs negative dentry bugfix

   - simple pm bus fixes to resolve reported issues"

* tag 'driver-core-5.15-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core:
  drivers: bus: Delete CONFIG_SIMPLE_PM_BUS
  drivers: bus: simple-pm-bus: Add support for probing simple bus only devices
  driver core: Reject pointless SYNC_STATE_ONLY device links
  kernfs: don't create a negative dentry if inactive node exists

Pull char/misc driver fixes from Greg KH:
 "Here are some small char/misc driver fixes for 5.15-rc6 for reported
  issues that include:

   - habanalabs driver fixes

   - mei driver fixes and new ids

   - fpga new device ids

   - MAINTAINER file updates for fpga subsystem

   - spi module id table additions and fixes

   - fastrpc locking fixes

   - nvmem driver fix

  All of these have been in linux-next for a while with no reported
  issues"

* tag 'char-misc-5.15-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc:
  eeprom: 93xx46: fix MODULE_DEVICE_TABLE
  nvmem: Fix shift-out-of-bound (UBSAN) with byte size cells
  mei: hbm: drop hbm responses on early shutdown
  mei: me: add Ice Lake-N device id.
  eeprom: 93xx46: Add SPI device ID table
  eeprom: at25: Add SPI ID table
  misc: HI6421V600_IRQ should depend on HAS_IOMEM
  misc: fastrpc: Add missing lock before accessing find_vma()
  cb710: avoid NULL pointer subtraction
  misc: gehc: Add SPI ID table
  MAINTAINERS: Drop outdated FPGA Manager website
  MAINTAINERS: Add Hao and Yilun as maintainers
  habanalabs: fix resetting args in wait for CS IOCTL
  fpga: ice40-spi: Add SPI device ID table

Pull staging and IIO driver fixes from Greg KH:
 "Here are a number of small IIO and staging driver fixes for 5.15-rc6.

  They include:

   - vc04_services bugfix for reported problem

   - r8188eu array underflow fix

   - iio driver fixes for a lot of tiny reported issues.

  All of these have been in linux-next for a while with no reported
  issues"

* tag 'staging-5.15-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging:
  staging: r8188eu: prevent array underflow in rtw_hal_update_ra_mask()
  staging: vc04_services: shut up out-of-range warning
  iio: light: opt3001: Fixed timeout error when 0 lux
  iio: adis16480: fix devices that do not support sleep mode
  iio: mtk-auxadc: fix case IIO_CHAN_INFO_PROCESSED
  iio: adis16475: fix deadlock on frequency set
  iio: ssp_sensors: add more range checking in ssp_parse_dataframe()
  iio: ssp_sensors: fix error code in ssp_print_mcu_debug()
  iio: adc: ad7793: Fix IRQ flag
  iio: adc: ad7780: Fix IRQ flag
  iio: adc: ad7192: Add IRQ flag
  iio: adc: aspeed: set driver data when adc probe.
  iio: adc: rzg2l_adc: add missing clk_disable_unprepare() in rzg2l_adc_pm_runtime_resume()
  iio: adc: max1027: Fix the number of max1X31 channels
  iio: adc: max1027: Fix wrong shift with 12-bit devices
  iio: adc128s052: Fix the error handling path of 'adc128_probe()'
  iio: adc: rzg2l_adc: Fix -EBUSY timeout error return
  iio: accel: fxls8962af: return IRQ_HANDLED when fifo is flushed
  iio: dac: ti-dac5571: fix an error code in probe()

Pull serial driver fix from Greg KH:
 "Here is a single 8250 Kconfig fix for 5.15-rc6 that resolves a
  regression that showed up in 5.15-rc1. It has been in linux-next for a
  while with no reported issues"

* tag 'tty-5.15-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty:
  serial: 8250: allow disabling of Freescale 16550 compile test