Signed-off-by: NDan Williams <dan.j.williams@intel.com>
[hch: split from a larger patch by Dan]
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJens Axboe <axboe@fb.com>

Some drivers use it now, others just set the limits field manually.
But in preparation for splitting this into a hard and soft limit,
ensure that they all call the proper function for setting the hw
limit for discards.
Reviewed-by: NJeff Moyer <jmoyer@redhat.com>
Signed-off-by: NJens Axboe <axboe@fb.com>

On systems with CONFIG_PREEMPT=n, under certain circumstances, mmcqd
can continuously process requests for several seconds without blocking,
triggering the soft lockup watchdog.  For example, this can happen if
mmcqd runs on the CPU which services the controller's interrupt, and
a process on a different CPU continuously writes to the MMC block
device.

 NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [mmcqd/0:664]
 CPU: 0 PID: 664 Comm: mmcqd/0 Not tainted 4.1.0-rc7+ #4
 PC is at _raw_spin_unlock_irqrestore+0x24/0x28
 LR is at mmc_start_request+0x104/0x134
 ...
 [<805112a8>] (_raw_spin_unlock_irqrestore) from [<803db664>] (mmc_start_request+0x104/0x134)
 [<803db664>] (mmc_start_request) from [<803dc008>] (mmc_start_req+0x274/0x394)
 [<803dc008>] (mmc_start_req) from [<803eb2c4>] (mmc_blk_issue_rw_rq+0xd0/0xb98)
 [<803eb2c4>] (mmc_blk_issue_rw_rq) from [<803ebe8c>] (mmc_blk_issue_rq+0x100/0x470)
 [<803ebe8c>] (mmc_blk_issue_rq) from [<803ecab8>] (mmc_queue_thread+0xd0/0x170)
 [<803ecab8>] (mmc_queue_thread) from [<8003fd14>] (kthread+0xe0/0xfc)
 [<8003fd14>] (kthread) from [<8000f768>] (ret_from_fork+0x14/0x2c)

Fix it by adding a cond_resched() in the request handling loop so that
other processes get a chance to run.
Signed-off-by: NRabin Vincent <rabin.vincent@axis.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Use kernel.h macro definition.

Thanks to Julia Lawall for Coccinelle scripting support.
Signed-off-by: NFabian Frederick <fabf@skynet.be>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

During kernel boot, it will try to read some logical sectors
of each block device node for the possible partition table.

But since RPMB partition is special and can not be accessed
by normal eMMC read / write CMDs, it will cause below error
messages during kernel boot:
...
 mmc0: Got data interrupt 0x00000002 even though no data operation was in progress.
 mmcblk0rpmb: error -110 transferring data, sector 0, nr 32, cmd response 0x900, card status 0xb00
 mmcblk0rpmb: retrying using single block read
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 end_request: I/O error, dev mmcblk0rpmb, sector 0
 Buffer I/O error on device mmcblk0rpmb, logical block 0
 end_request: I/O error, dev mmcblk0rpmb, sector 8
 Buffer I/O error on device mmcblk0rpmb, logical block 1
 end_request: I/O error, dev mmcblk0rpmb, sector 16
 Buffer I/O error on device mmcblk0rpmb, logical block 2
 end_request: I/O error, dev mmcblk0rpmb, sector 24
 Buffer I/O error on device mmcblk0rpmb, logical block 3
...

This patch will discard the access request in eMMC queue if
it is RPMB partition access request. By this way, it avoids
trigger above error messages.

Fixes: 090d25fe ("mmc: core: Expose access to RPMB partition")
Signed-off-by: NYunpeng Gao <yunpeng.gao@intel.com>
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Tested-by: NMichael Shigorin <mike@altlinux.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Allocation of previous bounce buffer in mmc_init_queue when the current
bounce buffer allocation fails was leading to a crash later in
__blk_segment_map_sg. Error handling is improved by allocating previous
bounce buffer only if the current bounce buffer allocation succeeds.
Signed-off-by: NBhuvanesh Surachari <bhuvanesh_surachari@mentor.com>
Signed-off-by: NHarish Jenny K N <harish_kandiga@mentor.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Clear QUEUE_FLAG_ADD_RANDOM in all block drivers that set
QUEUE_FLAG_NONROT.

Historically, all block devices have automatically made entropy
contributions.  But as previously stated in commit e2e1a148 ("block: add
sysfs knob for turning off disk entropy contributions"):
    - On SSD disks, the completion times aren't as random as they
      are for rotational drives. So it's questionable whether they
      should contribute to the random pool in the first place.
    - Calling add_disk_randomness() has a lot of overhead.

There are more reliable sources for randomness than non-rotational block
devices.  From a security perspective it is better to err on the side of
caution than to allow entropy contributions from unreliable "random"
sources.
Signed-off-by: NMike Snitzer <snitzer@redhat.com>
Signed-off-by: NJens Axboe <axboe@fb.com>

Use the much more common pr_warn instead of pr_warning.

Other miscellanea:

o Coalesce formats
o Realign arguments
o Remove extra spaces when coalescing formats
Signed-off-by: NJoe Perches <joe@perches.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

We must use a 64-bit for this, otherwise overflowed bits get lost, and
that can result in a lower than intended value set.

Fixes: 8e0cb8a1 ("ARM: 7797/1: mmc: Use dma_max_pfn(dev) helper for bounce_limit calculations")
Fixes: 7d35496d ("ARM: 7796/1: scsi: Use dma_max_pfn(dev) helper for bounce_limit calculations")
Tested-Acked-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

DMA bounce limit is the maximum direct DMA'able memory beyond which
bounce buffers has to be used to perform dma operations. MMC queue layr
relies on dma_mask but its calculation is based on max_*pfn which
don't have uniform meaning across architectures. So make use of
dma_max_pfn() which is expected to return the DMAable maximum pfn
value across architectures.

Cc: Chris Ball <cjb@laptop.org>
Signed-off-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The sanitize support is added as a user-app ioctl call, and
was removed from the block-device request, since its purpose is
to be invoked not via File-System but by a user.

This feature deletes the unmap memory region of the eMMC card,
by writing to a specific register in the EXT_CSD.

unmap region is the memory region that was previously deleted
(by erase, trim or discard operation).

In order to avoid timeout when sanitizing large-scale cards,
the timeout for sanitize operation is 240 seconds.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NMaya Erez <merez@codeaurora.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

For normal request mmc_blk_issue_rq is called twice with asynchronous
transfer(cur and prev). Host's claim and release can be done in each
mmc_blk_issue_rq. However, Special request is currently excluded in
asynchronous transfer. After special request is finished, if there is
no new request, mmc_release_host won't be called in mmc_blk_issue_rq.
The problem is founded during mmc_suspend.

[<c0541124>] (__schedule+0x0/0x78c) from [<c05419e8>] (schedule+0x38/0x78)
[<c05419b0>] (schedule+0x0/0x78) from [<c03a843c>] (__mmc_claim_host+0xac/0x1b4)
[<c03a8390>] (__mmc_claim_host+0x0/0x1b4) from [<c03ac98c>] (mmc_suspend+0x28/0x9c)
[<c03ac964>] (mmc_suspend+0x0/0x9c) from [<c03aad24>] (mmc_suspend_host+0xb4/0x194)
...
Reported-by: NJohan Rudholm <jrudholm@gmail.com>
Signed-off-by: NSeungwon Jeon <tgih.jun@samsung.com>
Tested-by: NJohan Rudholm <johan.rudholm@stericsson.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

This patch supports packed write command of eMMC4.5 devices.  Several
writes can be grouped in packed command and all data of the individual
commands can be sent in a single transfer on the bus. Large amounts of
data in one transfer rather than several data of small size are
effective for eMMC write internally.  As a result, packed command help
write throughput be improved.  The following tables show the results
of packed write.

Type A:
test     none |  packed
iozone   25.8 |  31
tiotest  27.6 |  31.2
lmdd     31.2 |  35.4

Type B:
test     none |  packed
iozone   44.1 |  51.1
tiotest  47.9 |  52.5
lmdd     51.6 |  59.2

Type C:
test     none |  packed
iozone   19.5 |  32
tiotest  19.9 |  34.5
lmdd     22.8 |  40.7
Signed-off-by: NSeungwon Jeon <tgih.jun@samsung.com>
Reviewed-by: NMaya Erez <merez@codeaurora.org>
Reviewed-by: NNamjae Jeon <linkinjeon@gmail.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

When current request is running on the bus and if next request fetched
by mmcqd is NULL, mmc context (mmcqd thread) gets blocked until the
current request completes. This means that if new request comes in while
the mmcqd thread is blocked, this new request can not be prepared in
parallel to current ongoing request. This may result in delaying the new
request execution and increase it's latency.

This change allows to wake up the MMC thread on new request arrival.
Now once the MMC thread is woken up, a new request can be fetched and
prepared in parallel to the current running request which means this new
request can be started immediately after the current running request
completes.

With this change read throughput is improved by 16%.
Signed-off-by: NKonstantin Dorfman <kdorfman@codeaurora.org>
Reviewed-by: NSeungwon Jeon <tgih.jun@samsung.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Unlike normal r/w request, special requests(discard, flush)
is finished with a one-time issue_fn. Request change to
mqrq_prev makes unnecessary call.
Signed-off-by: NSeungwon Jeon <tgih.jun@samsung.com>
Reviewed-by: NKonstantin Dorfman <kdorfman@codeaurora.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

When current request is running on the bus and if next request fetched
by mmcqd is NULL, mmc context (mmcqd thread) gets blocked until the
current request completes. This means that if new request comes in while
the mmcqd thread is blocked, this new request can not be prepared in
parallel to current ongoing request. This may result in delaying the new
request execution and increase it's latency.

This change allows to wake up the MMC thread on new request arrival.
Now once the MMC thread is woken up, a new request can be fetched and
prepared in parallel to the current running request which means this new
request can be started immediately after the current running request
completes.

With this change read throughput is improved by 16%.
Signed-off-by: NKonstantin Dorfman <kdorfman@codeaurora.org>
Reviewed-by: NSeungwon Jeon <tgih.jun@samsung.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Unlike normal r/w request, special requests(discard, flush)
is finished with a one-time issue_fn. Request change to
mqrq_prev makes unnecessary call.
Signed-off-by: NSeungwon Jeon <tgih.jun@samsung.com>
Reviewed-by: NKonstantin Dorfman <kdorfman@codeaurora.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

In case both 'req' and 'mq->mqrq_prev->req' are null, there is no request
to be processed. That means there is no need to switch buffer.
Switching buffer is required only after finishing 'issue_fn'.
Signed-off-by: NSeungwon Jeon <tgih.jun@samsung.com>
Reviewed-by: NPer Forlin <per.forlin@stericsson.com>
Tested-by: NJohan Rudholm <johan.rudholm@stericsson.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Not needed to memset, as they are pointers and are assigned
to proper values in the next line anyway.
Signed-off-by: NVenkatraman S <svenkatr@ti.com>
Reviewed-by: NNamjae Jeon <linkinjeon@gmail.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

The name mmc_request is used for both the issue function
and a data structure, which creates conflicts in symbol lookups
in editors. Rename the function to mmc_request_fn.
Signed-off-by: NVenkatraman S <svenkatr@ti.com>
Reviewed-by: NNamjae Jeon <linkinjeon@gmail.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

eMMC v4.5 discard operation is significantly different from the
existing trim operation because it is not guaranteed to work with
the new sanitize operation.  Consequently mmc_can_trim() is
separated from mmc_can_discard().

Also the new discard operation does not result in the sectors being
set to all-zeros, so discard_zeroes_data must not be set.

In addition, the new discard has the same timeout as trim, but from
v4.5 trim is defined to use the hc timeout.  The timeout calculation
is adjusted accordingly.

Fixes apply to linux 3.2 on.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Cc: <stable@vger.kernel.org>
Acked-by: NJaehoon Chung <jh80.chung@samsung.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

Kill block requests when the host realizes that the card is
removed from the slot and is sure that subsequent requests
are bound to fail. Do this silently so that the block
layer doesn't output unnecessary error messages.
Signed-off-by: NSujit Reddy Thumma <sthumma@codeaurora.org>
Acked-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

In the v4.5, there's no secure erase & trim support.
Instead it supports the sanitize feature.
Signed-off-by: NKyungmin Park <kyungmin.park@samsung.com>
Signed-off-by: NJaehoon Chung <jh80.chung@samsung.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

All the files using printk function for displaying kernel messages
in the mmc driver have been replaced with corresponding macro.
Signed-off-by: NGirish K S <girish.shivananjappa@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

Fix the sparse warning "drivers/mmc/card/queue.c:111:20: warning:
symbol 'mmc_alloc_sg' was not declared. Should it be static?"
Signed-off-by: NVenkatraman S <svenkatr@ti.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Change mmc_blk_issue_rw_rq() to become asynchronous.
The execution flow looks like this:

* The mmc-queue calls issue_rw_rq(), which sends the request
  to the host and returns back to the mmc-queue.
* The mmc-queue calls issue_rw_rq() again with a new request.
* This new request is prepared in issue_rw_rq(), then it waits for
  the active request to complete before pushing it to the host.
* When the mmc-queue is empty it will call issue_rw_rq() with a NULL
  req to finish off the active request without starting a new request.
Signed-off-by: NPer Forlin <per.forlin@linaro.org>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NVenkatraman S <svenkatr@ti.com>
Tested-by: NSourav Poddar <sourav.poddar@ti.com>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

Add an additional mmc queue request instance to make way for two active
block requests. One request may be active while the other request is
being prepared.
Signed-off-by: NPer Forlin <per.forlin@linaro.org>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NVenkatraman S <svenkatr@ti.com>
Tested-by: NSourav Poddar <sourav.poddar@ti.com>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

The way the request data is organized in the mmc queue struct, it only
allows processing of one request at a time.  This patch adds a new struct
to hold mmc queue request data such as sg list, request, blk request and
bounce buffers, and updates any functions depending on the mmc queue
struct. This prepares for using multiple active requests in one mmc queue.
Signed-off-by: NPer Forlin <per.forlin@linaro.org>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NVenkatraman S <svenkatr@ti.com>
Tested-by: NSourav Poddar <sourav.poddar@ti.com>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

Some host controllers will not operate without a hardware
timeout that is limited in value.  However large discards
require large timeouts, so there needs to be a way to
specify the maximum discard size.

A host controller driver may now specify the maximum discard
timeout possible so that max_discard_sectors can be calculated.

However, for eMMC when the High Capacity Erase Group Size
is not in use, the timeout calculation depends on clock
rate which may change.  For that case Preferred Erase Size
is used instead.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

SCSI defines discard alignment as the offset to the first
optimal discard.  In the case of SD/MMC, that is always zero
which is the default.

SCSI defines discard granularity as a hint of a optimal
discard size.  That is much better expressed by the MMC
"preferred erase size" (pref_erase) field.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

For example, an eMMC with 2 boot partitions will have 3 threads.
The names change from:

   40 ?        00:00:00 mmcqd/0
   41 ?        00:00:00 mmcqd/0
   42 ?        00:00:00 mmcqd/0

to:

   40 ?        00:00:00 mmcqd/0
   41 ?        00:00:00 mmcqd/0boot0
   42 ?        00:00:00 mmcqd/0boot1
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Cc: Andrei Warkentin <andreiw@motorola.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

There is no need to disable irq's when using the sg_copy_*_buffer()
functions because those functions do that already. There are also
no races for the mm_queue struct here that would require the irq's
to be disabled before calling sg_copy_*_buffer().
Signed-off-by: NJohn Ogness <john.ogness@linutronix.de>
Signed-off-by: NChris Ball <cjb@laptop.org>

Code has been converted over to the new explicit on-stack plugging,
and delay users have been converted to use the new API for that.
So lets kill off the old plugging along with aops->sync_page().
Signed-off-by: NJens Axboe <jaxboe@fusionio.com>

Usually there are multiple mmc host controllers; rename mmc queue thread
by host index so we can easily identify which controller it belongs to.
Signed-off-by: NEthan Du <ethan.too@gmail.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Get rid of init_MUTEX[_LOCKED]() and use sema_init() instead.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: linux-mmc@vger.kernel.org
Signed-off-by: NChris Ball <cjb@laptop.org>

We have deprecated the distinction between hardware and physical
segments in the block layer.  Consolidate the two limits into one in
drivers/mmc/.
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Barrier is deemed too heavy and will soon be replaced by FLUSH/FUA
requests.  Deprecate barrier.  All REQ_HARDBARRIERs are failed with
-EOPNOTSUPP and blk_queue_ordered() is replaced with simpler
blk_queue_flush().

blk_queue_flush() takes combinations of REQ_FLUSH and FUA.  If a
device has write cache and can flush it, it should set REQ_FLUSH.  If
the device can handle FUA writes, it should also set REQ_FUA.

All blk_queue_ordered() users are converted.

* ORDERED_DRAIN is mapped to 0 which is the default value.
* ORDERED_DRAIN_FLUSH is mapped to REQ_FLUSH.
* ORDERED_DRAIN_FLUSH_FUA is mapped to REQ_FLUSH | REQ_FUA.
Signed-off-by: NTejun Heo <tj@kernel.org>
Acked-by: NBoaz Harrosh <bharrosh@panasas.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Nick Piggin <npiggin@kernel.dk>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Jeremy Fitzhardinge <jeremy@xensource.com>
Cc: Chris Wright <chrisw@sous-sol.org>
Cc: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Cc: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Alasdair G Kergon <agk@redhat.com>
Cc: Pierre Ossman <drzeus@drzeus.cx>
Cc: Stefan Weinhuber <wein@de.ibm.com>
Signed-off-by: NJens Axboe <jaxboe@fusionio.com>

Secure discard is implemented by Secure Trim if the discard is unaligned
or Secure Erase otherwise.
Signed-off-by: NAdrian Hunter <adrian.hunter@nokia.com>
Acked-by: NJens Axboe <axboe@kernel.dk>
Cc: Kyungmin Park <kmpark@infradead.org>
Cc: Madhusudhan Chikkature <madhu.cr@ti.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Ben Gardiner <bengardiner@nanometrics.ca>
Cc: <linux-mmc@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Enable MMC to service discard requests.  In the case of SD and MMC cards
that do not support trim, discards become erases.  In the case of cards
(MMC) that only allow erases in multiples of erase group size, round to
the nearest completely discarded erase group.
Signed-off-by: NAdrian Hunter <adrian.hunter@nokia.com>
Acked-by: NJens Axboe <axboe@kernel.dk>
Cc: Kyungmin Park <kmpark@infradead.org>
Cc: Madhusudhan Chikkature <madhu.cr@ti.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Ben Gardiner <bengardiner@nanometrics.ca>
Cc: <linux-mmc@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This removes q->prepare_flush_fn completely (changes the
blk_queue_ordered API).
Signed-off-by: NFUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJens Axboe <jaxboe@fusionio.com>