Add CQE support to the block driver, including:
    - optionally using DCMD for flush requests
    - "manually" issuing discard requests
    - issuing read / write requests to the CQE
    - supporting block-layer timeouts
    - handling recovery
    - supporting re-tuning

CQE offers 25% - 50% better random multi-threaded I/O.  There is a slight
(e.g. 2%) drop in sequential read speed but no observable change to sequential
write.

CQE automatically sends the commands to complete requests.  However it only
supports reads / writes and so-called "direct commands" (DCMD).  Furthermore
DCMD is limited to one command at a time, but discards require 3 commands.
That makes issuing discards through CQE very awkward, but some CQE's don't
support DCMD anyway.  So for discards, the existing non-CQE approach is
taken, where the mmc core code issues the 3 commands one at a time i.e.
mmc_erase(). Where DCMD is used, is for issuing flushes.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>

Define and use a blk-mq queue. Discards and flushes are processed
synchronously, but reads and writes asynchronously. In order to support
slow DMA unmapping, DMA unmapping is not done until after the next request
is started. That means the request is not completed until then. If there is
no next request then the completion is done by queued work.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>

Add error-handling comments to explain what would also be done for blk-mq
if it used the legacy error-handling.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>

Use blk_cleanup_queue() to shutdown the queue when the driver is removed,
and instead get an extra reference to the queue to prevent the queue being
freed before the final mmc_blk_put().
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>

The card is not necessarily being removed, but the debugfs files must be
removed when the driver is removed, otherwise they will continue to exist
after unbinding the card from the driver. e.g.

  # echo "mmc1:0001" > /sys/bus/mmc/drivers/mmcblk/unbind
  # cat /sys/kernel/debug/mmc1/mmc1\:0001/ext_csd
  [  173.634584] BUG: unable to handle kernel NULL pointer dereference at 0000000000000050
  [  173.643356] IP: mmc_ext_csd_open+0x5e/0x170

A complication is that the debugfs_root may have already been removed, so
check for that too.

Fixes: 627c3ccf ("mmc: debugfs: Move block debugfs into block module")
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Cc: stable@vger.kernel.org # 4.14+
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

blk_get_request() can fail, always check the return value.

Fixes: 0493f6fe ("mmc: block: Move boot partition locking into a driver op")
Fixes: 3ecd8cf2 ("mmc: block: move multi-ioctl() to use block layer")
Fixes: 614f0388 ("mmc: block: move single ioctl() commands to block requests")
Fixes: 627c3ccf ("mmc: debugfs: Move block debugfs into block module")
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Cc: stable@vger.kernel.org # 4.13+
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Ensure blk_get_request() is paired with blk_put_request().

Fixes: 0493f6fe ("mmc: block: Move boot partition locking into a driver op")
Fixes: 627c3ccf ("mmc: debugfs: Move block debugfs into block module")
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Cc: stable@vger.kernel.org # 4.13+
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Enhance mmc_blk_data_prep() to support CQE requests. That means adding
some things that for non-CQE requests would be encoded into the command
arguments - such as the block address, reliable-write flag, and data tag
flag. Also the request tag is needed to provide the command queue task id,
and a comment is added to explain the future possibility of defining a
priority.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Use local variables in mmc_blk_data_prep() in preparation for adding CQE
support which doesn't use the output variables.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Currently the host can be claimed by a task.  Change this so that the host
can be claimed by a context that may or may not be a task.  This provides
for the host to be claimed by a block driver queue to support blk-mq, while
maintaining compatibility with the existing use of mmc_claim_host().
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

I forgot to account for the fact that the device core holds a
reference to a device added with device_initialize() that need
to be released with a corresponding put_device() to reach a 0
refcount at the end of the lifecycle.

This led to a NULL pointer reference when freeing the device
when e.g. unbidning the host device in sysfs.

Fix this and use the device .release() callback to free the
IDA and free:ing the memory used by the RPMB device.

Before this patch:

/sys/bus/amba/drivers/mmci-pl18x$ echo 80114000.sdi4_per2 > unbind
[   29.797332] mmc3: card 0001 removed
[   29.810791] Unable to handle kernel NULL pointer dereference at
               virtual address 00000050
[   29.818878] pgd = de70c000
[   29.821624] [00000050] *pgd=1e70a831, *pte=00000000, *ppte=00000000
[   29.827911] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
[   29.833282] Modules linked in:
[   29.836334] CPU: 1 PID: 154 Comm: sh Not tainted
               4.14.0-rc3-00039-g83318e309566-dirty #736
[   29.844604] Hardware name: ST-Ericsson Ux5x0 platform (Device Tree Support)
[   29.851562] task: de572700 task.stack: de742000
[   29.856079] PC is at kernfs_find_ns+0x8/0x100
[   29.860443] LR is at kernfs_find_and_get_ns+0x30/0x48

After this patch:

/sys/bus/amba/drivers/mmci-pl18x$ echo 80005000.sdi4_per2 > unbind
[   20.623382] mmc3: card 0001 removed

Fixes: 97548575 ("mmc: block: Convert RPMB to a character device")
Reported-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Acked-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

This function is used by the block layer queue to bail out of
requests if the current request is towards an RPMB
"block device".

This was done to avoid boot time scanning of this "block
device" which was never really a block device, thus duct-taping
over the fact that it was badly engineered.

This problem is now gone as we removed the offending RPMB block
device in another patch and replaced it with a character
device.

Cc: Tomas Winkler <tomas.winkler@intel.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The RPMB partition on the eMMC devices is a special area used
for storing cryptographically safe information signed by a
special secret key. To write and read records from this special
area, authentication is needed.

The RPMB area is *only* and *exclusively* accessed using
ioctl():s from userspace. It is not really a block device,
as blocks cannot be read or written from the device, also
the signed chunks that can be stored on the RPMB are actually
256 bytes, not 512 making a block device a real bad fit.

Currently the RPMB partition spawns a separate block device
named /dev/mmcblkNrpmb for each device with an RPMB partition,
including the creation of a block queue with its own kernel
thread and all overhead associated with this. On the Ux500
HREFv60 platform, for example, the two eMMCs means that two
block queues with separate threads are created for no use
whatsoever.

I have concluded that this block device design for RPMB is
actually pretty wrong. The RPMB area should have been designed
to be accessed from /dev/mmcblkN directly, using ioctl()s on
the main block device. It is however way too late to change
that, since userspace expects to open an RPMB device in
/dev/mmcblkNrpmb and we cannot break userspace.

This patch tries to amend the situation using the following
strategy:

- Stop creating a block device for the RPMB partition/area

- Instead create a custom, dynamic character device with
  the same name.

- Make this new character device support exactly the same
  set of ioctl()s as the old block device.

- Wrap the requests back to the same ioctl() handlers, but
  issue them on the block queue of the main partition/area,
  i.e. /dev/mmcblkN

We need to create a special "rpmb" bus type in order to get
udev and/or busybox hot/coldplug to instantiate the device
node properly.

Before the patch, this appears in 'ps aux':

101 root       0:00 [mmcqd/2rpmb]
123 root       0:00 [mmcqd/3rpmb]

After applying the patch these surplus block queue threads
are gone, but RPMB is as usable as ever using the userspace
MMC tools, such as 'mmc rpmb read-counter'.

We get instead those dynamice devices in /dev:

brw-rw----    1 root     root      179,   0 Jan  1  2000 mmcblk0
brw-rw----    1 root     root      179,   1 Jan  1  2000 mmcblk0p1
brw-rw----    1 root     root      179,   2 Jan  1  2000 mmcblk0p2
brw-rw----    1 root     root      179,   5 Jan  1  2000 mmcblk0p5
brw-rw----    1 root     root      179,   8 Jan  1  2000 mmcblk2
brw-rw----    1 root     root      179,  16 Jan  1  2000 mmcblk2boot0
brw-rw----    1 root     root      179,  24 Jan  1  2000 mmcblk2boot1
crw-rw----    1 root     root      248,   0 Jan  1  2000 mmcblk2rpmb
brw-rw----    1 root     root      179,  32 Jan  1  2000 mmcblk3
brw-rw----    1 root     root      179,  40 Jan  1  2000 mmcblk3boot0
brw-rw----    1 root     root      179,  48 Jan  1  2000 mmcblk3boot1
brw-rw----    1 root     root      179,  33 Jan  1  2000 mmcblk3p1
crw-rw----    1 root     root      248,   1 Jan  1  2000 mmcblk3rpmb

Notice the (248,0) and (248,1) character devices for RPMB.

Cc: Tomas Winkler <tomas.winkler@intel.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

In may, Steven sent a patch deleting the bounce buffer handling
and the CONFIG_MMC_BLOCK_BOUNCE option.

I chose the less invasive path of making it a runtime config
option, and we merged that successfully for kernel v4.12.

The code is however just standing in the way and taking up
space for seemingly no gain on any systems in wide use today.

Pierre says the code was there to improve speed on TI SDHCI
controllers on certain HP laptops and possibly some Ricoh
controllers as well. Early SDHCI controllers lacked the
scatter-gather feature, which made software bounce buffers
a significant speed boost.

We are clearly talking about the list of SDHCI PCI-based
MMC/SD card readers found in the pci_ids[] list in
drivers/mmc/host/sdhci-pci-core.c.

The TI SDHCI derivative is not supported by the upstream
kernel. This leaves the Ricoh.

What we can however notice is that the x86 defconfigs in the
kernel did not enable CONFIG_MMC_BLOCK_BOUNCE option, which
means that any such laptop would have to have a custom
configured kernel to actually take advantage of this
bounce buffer speed-up. It simply seems like there was
a speed optimization for the Ricoh controllers that noone
was using. (I have not checked the distro defconfigs but
I am pretty sure the situation is the same there.)

Bounce buffers increased performance on the OMAP HSMMC
at one point, and was part of the original submission in
commit a45c6cb8 ("[ARM] 5369/1: omap mmc: Add new
   omap hsmmc controller for 2430 and 34xx, v3")

This optimization was removed in
commit 0ccd76d4 ("omap_hsmmc: Implement scatter-gather
   emulation")
which found that scatter-gather emulation provided even
better performance.

The same was introduced for SDHCI in
commit 2134a922 ("sdhci: scatter-gather (ADMA) support")

I am pretty positively convinced that software
scatter-gather emulation will do for any host controller what
the bounce buffers were doing. Essentially, the bounce buffer
was a reimplementation of software scatter-gather-emulation in
the MMC subsystem, and it should be done away with.

Cc: Pierre Ossman <pierre@ossman.eu>
Cc: Juha Yrjola <juha.yrjola@solidboot.com>
Cc: Steven J. Hill <Steven.Hill@cavium.com>
Cc: Shawn Lin <shawn.lin@rock-chips.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Suggested-by: NSteven J. Hill <Steven.Hill@cavium.com>
Suggested-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The intention for this patch is to help folks debug the failure
like this:

dwmmc_rockchip fe320000.dwmmc: IDMAC supports 32-bit address mode.
dwmmc_rockchip fe320000.dwmmc: Using internal DMA controller.
dwmmc_rockchip fe320000.dwmmc: Version ID is 270a
dwmmc_rockchip fe320000.dwmmc: DW MMC controller at irq 28,32 bit
host data width,256 deep fifo
dwmmc_rockchip fe320000.dwmmc: Got CD GPIO
mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual
400000HZ div = 0)
mmc_host mmc0: Bus speed (slot 0) = 50000000Hz (slot req 50000000Hz,
actual 50000000HZ div = 0)
mmc0: new high speed SDHC card at address 0007
mmcblk: probe of mmc0:0007 failed with error -28

The reason may be some buggy userspace daemon miss the disk remove
uevent sometimes so it would finally make the SD card not work.
So from the dmesg it only shows a errno of -28 but still don't understand
what happened.

For quick reproduce this, we could set max_devices to 8 and run

for i in $(seq 1 9); do
  echo "========================" $i
  echo fe320000.dwmmc > /sys/bus/platform/drivers/dwmmc_rockchip/unbind
  sleep .5
  echo fe320000.dwmmc > /sys/bus/platform/drivers/dwmmc_rockchip/bind
  sleep .5
  mount -t vfat /dev/mmcblk0 /mnt
  sleep .5
done

Another possible reason would be the device has more partitions than
what we support, so that they have to increase their max_devices.
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Instead of passing a block device to
mmc_blk_ioctl[_multi]_cmd(), let's pass struct mmc_blk_data()
so we operate ioctl()s on the MMC block device representation
rather than the vanilla block device.

This saves a little duplicated code and makes it possible to
issue ioctl()s not targeted for a specific block device but
rather for a specific partition/area.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Instead of passing a struct mmc_blk_data * to mmc_blk_part_switch()
let's pass the actual partition type we want to switch to. This
is necessary in order not to have a block device with a backing
mmc_blk_data and request queue and all for every hardware partition,
such as RPMB.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

mmc_blk_ioctl() calls either mmc_blk_ioctl_cmd() or
mmc_blk_ioctl_multi_cmd() and each of these make the same
check. Factor it into a new helper function, call it on
both branches of the switch() statement and save a chunk
of duplicate code.

Cc: Shawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

If we don't have the block layer enabled, we do not present card
status and extcsd in the debugfs.

Debugfs is not ABI, and maintaining files of no relevance for
non-block devices comes at a high maintenance cost if we shall
support it with the block layer compiled out.

The debugfs entries suffer from all the same starvation
issues as the other userspace things, under e.g. a heavy
dd operation.

The expected number of debugfs users utilizing these two
debugfs files is already low as there is an ioctl() to get the
same information using the mmc-tools, and of these few users
the expected number of people using it on SDIO or combo cards
are expected to be zero.

It is therefore logical to move this over to the block layer
when it is enabled, using the new custom requests and issue
it using the block request queue.

On the other hand it moves some debugfs code from debugfs.c
and into block.c.

Tested during heavy dd load by cat:in the status file.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

We have a data pointer for the ioctl() data, but we need to
pass other data along with the DRV_OP:s, so make this a
void * so it can be reused.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

It was never used and introduced a long standing compile
warning:

drivers/mmc/core/block.c: In function 'power_ro_lock_store':
drivers/mmc/core/block.c:191:19: warning: variable 'card' set but not
used [-Wunused-but-set-variable]

Remove it to fix the warning.
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Commit 2a842aca ("block: introduce new block status code type") changed
the error type but not in patches merged through the mmc tree, like
commit 0493f6fe ("mmc: block: Move boot partition locking into a driver
op"). Fix one error code that is incorrect and also use BLK_STS_OK in
preference to 0.

Fixes: 17ece345 ("Merge tag 'mmc-v4.13' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc")
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

We to some extent should tolerate R1_OUT_OF_RANGE for open-ending
mode as it is expected behaviour and most of the backup partition
tables should be located near some of the last blocks which will
always make open-ending read exceed the capacity of cards.

Fixes: 9820a5b1 ("mmc: core: for data errors, take response of stop cmd into account")
Fixes: a04e6bae ("mmc: core: check also R1 response for stop commands")
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Reviewed-by: NWolfram Sang <wsa+renesas@sang-engineering.com>
Tested-by: NShawn Guo <shawnguo@kernel.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Fix lockdep splat introduced in v4.13-rc4.

[  266.297226] ------------[ cut here ]------------
[  266.300078] WARNING: CPU: 2 PID: 176 at /mnt/src/jaja/git/tf300t/include/linux/blkdev.h:657 mmc_blk_remove_req+0xd0/0xe8 [mmc_block]
[  266.302937] Modules linked in: mmc_block(-) sdhci_tegra sdhci_pltfm sdhci pwrseq_simple pwrseq_emmc mmc_core
[  266.305941] CPU: 2 PID: 176 Comm: rmmod Tainted: G        W       4.13.0-rc4mq-00208-gb691e67724b8-dirty #694
[  266.308852] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  266.311719] [<b011144c>] (unwind_backtrace) from [<b010ca54>] (show_stack+0x18/0x1c)
[  266.314664] [<b010ca54>] (show_stack) from [<b062e3f4>] (dump_stack+0x84/0x98)
[  266.317644] [<b062e3f4>] (dump_stack) from [<b01214f4>] (__warn+0xf4/0x10c)
[  266.320542] [<b01214f4>] (__warn) from [<b01215d4>] (warn_slowpath_null+0x28/0x30)
[  266.323534] [<b01215d4>] (warn_slowpath_null) from [<af067858>] (mmc_blk_remove_req+0xd0/0xe8 [mmc_block])
[  266.326568] [<af067858>] (mmc_blk_remove_req [mmc_block]) from [<af068f40>] (mmc_blk_remove_parts.constprop.6+0x50/0x64 [mmc_block])
[  266.329678] [<af068f40>] (mmc_blk_remove_parts.constprop.6 [mmc_block]) from [<af0693b8>] (mmc_blk_remove+0x24/0x140 [mmc_block])
[  266.332894] [<af0693b8>] (mmc_blk_remove [mmc_block]) from [<af0052ec>] (mmc_bus_remove+0x20/0x28 [mmc_core])
[  266.336198] [<af0052ec>] (mmc_bus_remove [mmc_core]) from [<b046aa64>] (device_release_driver_internal+0x164/0x200)
[  266.339367] [<b046aa64>] (device_release_driver_internal) from [<b046ab54>] (driver_detach+0x40/0x74)
[  266.342537] [<b046ab54>] (driver_detach) from [<b046982c>] (bus_remove_driver+0x68/0xdc)
[  266.345660] [<b046982c>] (bus_remove_driver) from [<af06ad40>] (mmc_blk_exit+0xc/0x2cc [mmc_block])
[  266.348875] [<af06ad40>] (mmc_blk_exit [mmc_block]) from [<b01aee30>] (SyS_delete_module+0x1c4/0x254)
[  266.352068] [<b01aee30>] (SyS_delete_module) from [<b0108480>] (ret_fast_syscall+0x0/0x34)
[  266.355308] ---[ end trace f68728a0d3053b72 ]---

Fixes: 7c84b8b4 ("mmc: block: bypass the queue even if usage is present for hotplug")
Signed-off-by: NMichał Mirosław <mirq-linux@rere.qmqm.pl>
Reviewed-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The commit 304419d8 ("mmc: core: Allocate per-request data using the
block layer core") refactored mechanism of queue handling caused
mmc_init_request() can be called just after mmc_cleanup_queue() caused null
pointer dereference.

Another commit bbdc74dc ("mmc: block: Prevent new req entering queue
after its cleanup") tried to fix the problem. However it actually miss one
corner case.

We could still reproduce the issue mentioned with these steps:
(1) insert a SD card and mount it
(2) hotplug it, so it will leave md->usage still be counted
(3) reboot the system which will sync data and umount the card

[Unable to handle kernel NULL pointer dereference at virtual address
00000000
[user pgtable: 4k pages, 48-bit VAs, pgd = ffff80007bab3000
[[0000000000000000] *pgd=000000007a828003, *pud=0000000078dce003,
*pmd=000000007aab6003, *pte=0000000000000000
[Internal error: Oops: 96000007 [#1] PREEMPT SMP
[Modules linked in:
[CPU: 3 PID: 3507 Comm: umount Tainted: G        W
4.13.0-rc1-next-20170720-00012-g9d9bf45 #33
[Hardware name: Firefly-RK3399 Board (DT)
[task: ffff80007a1de200 task.stack: ffff80007a01c000
[PC is at mmc_init_request+0x14/0xc4
[LR is at alloc_request_size+0x4c/0x74
[pc : [<ffff0000087d7150>] lr : [<ffff000008378fe0>] pstate: 600001c5
[sp : ffff80007a01f8f0

....

[[<ffff0000087d7150>] mmc_init_request+0x14/0xc4
[[<ffff000008378fe0>] alloc_request_size+0x4c/0x74
[[<ffff00000817ac28>] mempool_create_node+0xb8/0x17c
[[<ffff00000837aadc>] blk_init_rl+0x9c/0x120
[[<ffff000008396580>] blkg_alloc+0x110/0x234
[[<ffff000008396ac8>] blkg_create+0x424/0x468
[[<ffff00000839877c>] blkg_lookup_create+0xd8/0x14c
[[<ffff0000083796bc>] generic_make_request_checks+0x368/0x3b0
[[<ffff00000837b050>] generic_make_request+0x1c/0x240

So mmc_blk_put wouldn't calling blk_cleanup_queue which actually the
QUEUE_FLAG_DYING and QUEUE_FLAG_BYPASS should stay. Block core expect
blk_queue_bypass_{start, end} internally to bypass/drain the queue before
actually dying the queue, so it didn't expose API to set the queue bypass.
I think we should set QUEUE_FLAG_BYPASS whenever queue is removed, although
the md->usage is still counted, as no dispatch queue could be found then.

Fixes: 304419d8 ("mmc: core: Allocate per-request data using the block layer core")
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The commit 304419d8 ("mmc: core: Allocate per-request data using the
block layer core"), refactored the mechanism of queue handling, but also
made mmc_init_request() to be called after mmc_cleanup_queue(). This
triggers a null pointer dereference:

[  683.123791] BUG: unable to handle kernel NULL pointer dereference at (null)
[  683.123801] IP: mmc_init_request+0x2c/0xf0 [mmc_block]
...
[  683.123905] Call Trace:
[  683.123913]  alloc_request_size+0x4f/0x70
[  683.123919]  mempool_alloc+0x5f/0x150
[  683.123925]  ? __enqueue_entity+0x6c/0x70
[  683.123928]  get_request+0x3ad/0x720
[  683.123933]  ? prepare_to_wait_event+0x110/0x110
[  683.123937]  blk_queue_bio+0xc1/0x3a0
[  683.123940]  generic_make_request+0xf8/0x2a0
[  683.123942]  submit_bio+0x75/0x150
[  683.123947]  submit_bio_wait+0x51/0x70
[  683.123951]  blkdev_issue_flush+0x5c/0x90
[  683.123956]  ext4_sync_fs+0x171/0x1b0
[  683.123961]  sync_filesystem+0x73/0x90
[  683.123965]  fsync_bdev+0x24/0x50
[  683.123971]  invalidate_partition+0x24/0x50
[  683.123973]  del_gendisk+0xb2/0x2a0
[  683.123977]  mmc_blk_remove_req.part.38+0x71/0xa0 [mmc_block]
[  683.123980]  mmc_blk_remove+0xba/0x190 [mmc_block]
[  683.123990]  mmc_bus_remove+0x1a/0x20 [mmc_core]
[  683.123995]  device_release_driver_internal+0x141/0x200
[  683.123999]  device_release_driver+0x12/0x20
[  683.124001]  bus_remove_device+0xfd/0x170
[  683.124004]  device_del+0x1e8/0x330
[  683.124012]  mmc_remove_card+0x60/0xc0 [mmc_core]
[  683.124019]  mmc_remove+0x19/0x30 [mmc_core]
[  683.124025]  mmc_stop_host+0xfb/0x1a0 [mmc_core]
[  683.124032]  mmc_remove_host+0x1a/0x40 [mmc_core]
[  683.124037]  sdhci_remove_host+0x2e/0x1c0 [mmc_sdhci]
[  683.124042]  sdhci_pci_remove_slot+0x3f/0x80 [sdhci_pci]
[  683.124045]  sdhci_pci_remove+0x39/0x70 [sdhci_pci]
[  683.124049]  pci_device_remove+0x39/0xc0
[  683.124052]  device_release_driver_internal+0x141/0x200
[  683.124056]  driver_detach+0x3f/0x80
[  683.124059]  bus_remove_driver+0x55/0xd0
[  683.124062]  driver_unregister+0x2c/0x50
[  683.124065]  pci_unregister_driver+0x29/0x90
[  683.124069]  sdhci_driver_exit+0x10/0x4f3 [sdhci_pci]
[  683.124073]  SyS_delete_module+0x171/0x250
[  683.124078]  entry_SYSCALL_64_fastpath+0x1e/0xa9

Fix this by setting the queue DYING flag before cleanup the queue, as it
prevents new reqs from entering the queue.
Signed-off-by: NGrzegorz Sluja <grzegorzx.sluja@intel.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Fixes: 304419d8 ("mmc: core: Allocate per-request data using the...")
[Ulf: Updated the changelog]
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

With gcc 4.1.2:

    drivers/mmc/core/block.c: In function ‘mmc_blk_ioctl_cmd_issue’:
    drivers/mmc/core/block.c:630: warning: ‘ioc_err’ may be used uninitialized in this function

Indeed, if mq_rq->ioc_count is zero, an uninitialized value will be
stored in mq_rq->drv_op_result and passed to blk_end_request_all().

Can mq_rq->ioc_count be zero?
  - mmc_blk_ioctl_cmd() sets ioc_count to 1, so this is safe,
  - mmc_blk_ioctl_multi_cmd() obtains ioc_count from user space in
    response to the MMC_IOC_MULTI_CMD ioctl, and does allow zero.

To avoid returning an uninitialized value, and as it is pointless to do
all this work when the MMC_IOC_MULTI_CMD ioctl is used with zero
entries, check for this early in mmc_blk_ioctl_multi_cmd(), and return
zero, like was returned before.

Fixes: 3ecd8cf2 ("mmc: block: move multi-ioctl() to use block layer")
Signed-off-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

With gcc 4.1.2:

    drivers/mmc/core/block.c: In function ‘mmc_blk_issue_drv_op’:
    drivers/mmc/core/block.c:1178: warning: ‘ret’ may be used uninitialized in this function

Indeed, for MMC_DRV_OP_IOCTL, if mq_rq->ioc_count is zero, an
uninitialized value will be stored in mq_rq->drv_op_result and passed to
blk_end_request_all().

Can mq_rq->ioc_count be zero?
  - mmc_blk_ioctl_cmd() sets ioc_count to 1, so this is safe,
  - mmc_blk_ioctl_multi_cmd() obtains ioc_count from user space in
    response to the MMC_IOC_MULTI_CMD ioctl, and does allow zero.

Initialize ret to zero to fix this for current and future callers.

Fixes: 0493f6fe ("mmc: block: Move boot partition locking into a driver op")
Signed-off-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Some errors are flagged only with the next command after a multiblock
transfer, e.g. ECC error. So, when checking for data transfer errors,
we check the result from the stop command as well.
Signed-off-by: NWolfram Sang <wsa+renesas@sang-engineering.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

To detect errors like ECC errors, we must parse the R1 response bits. Introduce
a helper function to also set the error value of a command when R1 error bits
are set. Add ECC error to list of flags checked. Use the new helper for the
stop command to call mmc_blk_recovery when detecting ECC errors which are only
flagged on the next command after multiblock.
Signed-off-by: NWolfram Sang <wsa+renesas@sang-engineering.com>
Tested-by: NYoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

drivers/mmc/core/block.c:1929:3-4: Unneeded semicolon

 Remove unneeded semicolon.

Generated by: scripts/coccinelle/misc/semicolon.cocci

CC: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: NFengguang Wu <fengguang.wu@intel.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The only reason to why the mmc block device driver needs to implements its
own version of how to get the status of the card, is that it needs to
specify a different amount of retries.

Therefore add a new exported function which allows the caller to specify
the number of retries and convert everybody to use it, as this simplifies
the code.
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>

This moves the boot partition lock command (issued from sysfs)
into a custom block layer request, just like the ioctl()s,
getting rid of yet another instance of mmc_get_card().

Since we now have two operations issuing special DRV_OP's, we
rename the result variable ->drv_op_result.

Tested by locking the boot partition from userspace:
> cd /sys/devices/platform/soc/80114000.sdi4_per2/mmc_host/mmc3/
     mmc3:0001/block/mmcblk3/mmcblk3boot0
> echo 1 > ro_lock_until_next_power_on
[  178.645324] mmcblk3boot1: Locking boot partition ro until next power on
[  178.652221] mmcblk3boot0: Locking boot partition ro until next power on

Also tested this with a huge dd job in the background: it
is now possible to lock the boot partitions on the card even
under heavy I/O.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

We will need to access static functions above the pure block layer
operations in the file, so move the driver operations issue
function down so we can see all non-blocklayer symbols.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

We will expand the DRV_OP usage, so we need to know which
operation we're performing. Tag the operations with an
enum:ed type and rename the function so it is clear that
it deals with any command and put a switch statement in
it. Currently only ioctls are supported.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Just as we can use blk_mq_rq_from_pdu() to get the per-request
tag we can use blk_mq_rq_to_pdu() to get a request from a tag.
Introduce a static inline helper so we are on the clear what
is happening.
Suggested-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

This switches also the multiple-command ioctl() call to issue
all ioctl()s through the block layer instead of going directly
to the device.

We extend the passed argument with an argument count and loop
over all passed commands in the ioctl() issue function called
from the block layer.

By doing this we are again loosening the grip on the big host
lock, since two calls to mmc_get_card()/mmc_put_card() are
removed.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NAvri Altman <Avri.Altman@sandisk.com>

This wraps single ioctl() commands into block requests using
the custom block layer request types REQ_OP_DRV_IN and
REQ_OP_DRV_OUT.

By doing this we are loosening the grip on the big host lock,
since two calls to mmc_get_card()/mmc_put_card() are removed.

We are storing the ioctl() in/out argument as a pointer in
the per-request struct mmc_blk_request container. Since we
now let the block layer allocate this data, blk_get_request()
will allocate it for us and we can immediately dereference
it and use it to pass the argument into the block layer.

We refactor the if/else/if/else ladder in mmc_blk_issue_rq()
as part of the job, keeping some extra attention to the
case when a NULL req is passed into this function and
making that pipeline flush more explicit.

Tested on the ux500 with the userspace:
mmc extcsd read /dev/mmcblk3
resulting in a successful EXTCSD info dump back to the
console.

This commit fixes a starvation issue in the MMC/SD stack
that can be easily provoked in the following way by
issueing the following commands in sequence:

> dd if=/dev/mmcblk3 of=/dev/null bs=1M &
> mmc extcs read /dev/mmcblk3

Before this patch, the extcsd read command would hang
(starve) while waiting for the dd command to finish since
the block layer was holding the card/host lock.

After this patch, the extcsd ioctl() command is nicely
interpersed with the rest of the block commands and we
can issue a bunch of ioctl()s from userspace while there
is some busy block IO going on without any problems.

Conversely userspace ioctl()s can no longer starve
the block layer by holding the card/host lock.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NAvri Altman <Avri.Altman@sandisk.com>

The variable is_rpmb is clearly a bool and even assigned true
and false, yet declared as an int.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The mmc_queue_req is a per-request state container the MMC core uses
to carry bounce buffers, pointers to asynchronous requests and so on.
Currently allocated as a static array of objects, then as a request
comes in, a mmc_queue_req is assigned to it, and used during the
lifetime of the request.

This is backwards compared to how other block layer drivers work:
they usally let the block core provide a per-request struct that get
allocated right beind the struct request, and which can be obtained
using the blk_mq_rq_to_pdu() helper. (The _mq_ infix in this function
name is misleading: it is used by both the old and the MQ block
layer.)

The per-request struct gets allocated to the size stored in the queue
variable .cmd_size initialized using the .init_rq_fn() and
cleaned up using .exit_rq_fn().

The block layer code makes the MMC core rely on this mechanism to
allocate the per-request mmc_queue_req state container.

Doing this make a lot of complicated queue handling go away. We only
need to keep the .qnct that keeps count of how many request are
currently being processed by the MMC layer. The MQ block layer will
replace also this once we transition to it.

Doing this refactoring is necessary to move the ioctl() operations
into custom block layer requests tagged with REQ_OP_DRV_[IN|OUT]
instead of the custom code using the BigMMCHostLock that we have
today: those require that per-request data be obtainable easily from
a request after creating a custom request with e.g.:

struct request *rq = blk_get_request(q, REQ_OP_DRV_IN, __GFP_RECLAIM);
struct mmc_queue_req *mq_rq = req_to_mq_rq(rq);

And this is not possible with the current construction, as the request
is not immediately assigned the per-request state container, but
instead it gets assigned when the request finally enters the MMC
queue, which is way too late for custom requests.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
[Ulf: Folded in the fix to drop a call to blk_cleanup_queue()]
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NHeiner Kallweit <hkallweit1@gmail.com>