The mmc_queue_req is a per-request state container the MMC core uses
to carry bounce buffers, pointers to asynchronous requests and so on.
Currently allocated as a static array of objects, then as a request
comes in, a mmc_queue_req is assigned to it, and used during the
lifetime of the request.

This is backwards compared to how other block layer drivers work:
they usally let the block core provide a per-request struct that get
allocated right beind the struct request, and which can be obtained
using the blk_mq_rq_to_pdu() helper. (The _mq_ infix in this function
name is misleading: it is used by both the old and the MQ block
layer.)

The per-request struct gets allocated to the size stored in the queue
variable .cmd_size initialized using the .init_rq_fn() and
cleaned up using .exit_rq_fn().

The block layer code makes the MMC core rely on this mechanism to
allocate the per-request mmc_queue_req state container.

Doing this make a lot of complicated queue handling go away. We only
need to keep the .qnct that keeps count of how many request are
currently being processed by the MMC layer. The MQ block layer will
replace also this once we transition to it.

Doing this refactoring is necessary to move the ioctl() operations
into custom block layer requests tagged with REQ_OP_DRV_[IN|OUT]
instead of the custom code using the BigMMCHostLock that we have
today: those require that per-request data be obtainable easily from
a request after creating a custom request with e.g.:

struct request *rq = blk_get_request(q, REQ_OP_DRV_IN, __GFP_RECLAIM);
struct mmc_queue_req *mq_rq = req_to_mq_rq(rq);

And this is not possible with the current construction, as the request
is not immediately assigned the per-request state container, but
instead it gets assigned when the request finally enters the MMC
queue, which is way too late for custom requests.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
[Ulf: Folded in the fix to drop a call to blk_cleanup_queue()]
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Tested-by: NHeiner Kallweit <hkallweit1@gmail.com>

eMMC can have multiple internal partitions that are represented as separate
disks / queues. However switching between partitions is only done when the
queue is empty. Consequently the array of mmc requests that are queued can
be shared between partitions saving memory.

Keep a pointer to the mmc request queue on the card, and use that instead
of allocating a new one for each partition.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Change from viewing the requests in progress as 'current' and 'previous',
to viewing them as a queue. The current request is allocated to the first
free slot. The presence of incomplete requests is determined from the
count (mq->qcnt) of entries in the queue. Non-read-write requests (i.e.
discards and flushes) are not added to the queue at all and require no
special handling. Also no special handling is needed for the
MMC_BLK_NEW_REQUEST case.

As well as allowing an arbitrarily sized queue, the queue thread function
is significantly simpler.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Instead of masking and setting two bits in the "flags" field
for the mmc_queue, just use two bools named "suspended" and
"new_request".

The masking and setting would likely have race conditions
anyways, it is better to use a simple member like this.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The mmc_active member of struct mmc_queue_req has a very
confusing name: this is certainly not always "active", it is
the asynchronous request associated by the mmc_queue_req
but it is not guaranteed to be "active" in any sense, such
as being running on the host.

Simply rename this member to "areq".
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

This is the first step in cleaning up the private mmc header files. In this
change we makes sure each header file builds standalone, as that helps to
resolve dependencies.

While changing this, it also seems reasonable to stop including other
headers from inside a header itself which it don't depend upon.
Additionally, in some cases such dependencies are better resolved by
forward declaring the needed struct.
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Reviewed-by: NShawn Lin <shawn.lin@rock-chips.com>

Once upon a time it made sense to keep the mmc block device driver and its
related code, in its own directory called card. Over time, more an more
functions/structures have become shared through generic mmc header files,
between the core and the card directory. In other words, the relationship
between them has become closer.

By sharing functions/structures via generic header files, it becomes easy
for outside users to abuse them. In a way to avoid that from happen, let's
move the files from card directory into the core directory, as it enables
us to move definitions of functions/structures into mmc core specific
header files.

Note, this is only the first step in providing a cleaner mmc interface for
outside users. Following changes will do the actual cleanup, as that is not
part of this change.
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>

Add a mmc_queue member to record the size of the queue, which currently
supports 2 requests on-the-go at a time. Instead of allocating resources
for 2 slots in the queue, allow for an arbitrary number.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The only time the driver sleeps expecting to be woken upon the arrival of
a new request, is when the dispatch queue is empty. The only time that it
is known whether the dispatch queue is empty is after NULL is returned
from blk_fetch_request() while under the queue lock.

Recognizing those facts, simplify the synchronization between the queue
thread and the request function. A couple of flags tell the request
function what to do, and the queue lock and barriers associated with
wake-ups ensure synchronization.

The result is simpler and allows the removal of the context_info lock.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: NLinus Walleij <linus.walleij@linaro.org>
Reviewed-by: NHarjani Ritesh <riteshh@codeaurora.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

I've had it with this code now.

The packed command support is a complex hurdle in the MMC/SD block
layer, around 500+ lines of code which was introduced in 2013 in

commit ce39f9d1 ("mmc: support packed write command for eMMC4.5
devices")
commit abd9ac14 ("mmc: add packed command feature of eMMC4.5")

...and since then it has been rotting. The original author of the
code has disappeared from the community and the mail address is
bouncing.

For the code to be exercised the host must flag that it supports
packed commands, so in mmc_blk_prep_packed_list() which is called for
every single request, the following construction appears:

u8 max_packed_rw = 0;

if ((rq_data_dir(cur) == WRITE) &&
    mmc_host_packed_wr(card->host))
        max_packed_rw = card->ext_csd.max_packed_writes;

if (max_packed_rw == 0)
    goto no_packed;

This has the following logical deductions:

- Only WRITE commands can really be packed, so the solution is
  only half-done: we support packed WRITE but not packed READ.
  The packed command support has not been finalized by supporting
  reads in three years!

- mmc_host_packed_wr() is just a static inline that checks
  host->caps2 & MMC_CAP2_PACKED_WR. The problem with this is
  that NO upstream host sets this capability flag! No driver
  in the kernel is using it, and we can't test it. Packed
  command may be supported in out-of-tree code, but I doubt
  it. I doubt that the code is even working anymore due to
  other refactorings in the MMC block layer, who would
  notice if patches affecting it broke packed commands?
  No one.

- There is no Device Tree binding or code to mark a host as
  supporting packed read or write commands, just this flag
  in caps2, so for sure there are not any DT systems using
  it either.

It has other problems as well: mmc_blk_prep_packed_list() is
speculatively picking requests out of the request queue with
blk_fetch_request() making the MMC/SD stack harder to convert
to the multiqueue block layer. By this we get rid of an
obstacle.

The way I see it this is just cruft littering the MMC/SD
stack.

Cc: Namjae Jeon <namjae.jeon@samsung.com>
Cc: Maya Erez <qca_merez@qca.qualcomm.com>
Acked-by: NJaehoon Chung <jh80.chung@samsung.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

By moving the mmc_packed_init() and mmc_packed_clean() into the
only file in the kernel where they are used, we save two exported
functions and can staticize those to the block.c file.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

The struct mmc_blk_request contains an opaque void *data that
is actually only used to store a pointer to a per-request
struct mmc_blk_data. This is confusing, so rename the member
to blkdata and forward-declare the block.c local struct.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Commit f68381a7 (mmc: block: fix packed command header endianness)
correctly fixed endianness handling of packed_cmd_hdr in
mmc_blk_packed_hdr_wrq_prep.

But now, sparse complains about incorrect types:
drivers/mmc/card/block.c:1613:27: sparse: incorrect type in assignment (different base types)
drivers/mmc/card/block.c:1613:27:    expected unsigned int [unsigned] [usertype] <noident>
drivers/mmc/card/block.c:1613:27:    got restricted __le32 [usertype] <noident>
...

So annotate cmd_hdr properly using __le32 to make everyone happy.
Signed-off-by: NJiri Slaby <jslaby@suse.cz>
Fixes: f68381a7 (mmc: block: fix packed command header endianness)
Cc: stable@vger.kernel.org
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

We have enough vtables in the kernel as it is, we don't need
this one to create even more artificial separation of concerns.

As is proved by the Makefile:

obj-$(CONFIG_MMC_BLOCK)         += mmc_block.o
mmc_block-objs                  := block.o queue.o

block.c and queue.c are baked into the same mmc_block.o object.
So why would one of these objects access a function in the
other object by dereferencing a pointer?

Create a new block.h header file for the single shared function
from block to queue and remove the function pointer and just
call the queue request function.

Apart from making the code more readable, this also makes link
optimizations possible and probably speeds up the call as well.
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

Commit 288dab8a ("block: add a separate operation type for secure
erase") split REQ_OP_SECURE_ERASE from REQ_OP_DISCARD without considering
all the places REQ_OP_DISCARD was being used to mean either. Fix those.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Fixes: 288dab8a ("block: add a separate operation type for secure erase")
Signed-off-by: NJens Axboe <axboe@fb.com>

This adds a REQ_OP_FLUSH operation that is sent to request_fn
based drivers by the block layer's flush code, instead of
sending requests with the request->cmd_flags REQ_FLUSH bit set.
Signed-off-by: NMike Christie <mchristi@redhat.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NHannes Reinecke <hare@suse.com>
Signed-off-by: NJens Axboe <axboe@fb.com>

The req operation REQ_OP is separated from the rq_flag_bits
definition. This converts the block layer drivers to
use req_op to get the op from the request struct.
Signed-off-by: NMike Christie <mchristi@redhat.com>
Reviewed-by: NHannes Reinecke <hare@suse.com>
Signed-off-by: NJens Axboe <axboe@fb.com>

Retry errored data requests when re-tuning is needed and
add a flag to struct mmc_blk_request so that the retry
is only done once.
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

During kernel boot, it will try to read some logical sectors
of each block device node for the possible partition table.

But since RPMB partition is special and can not be accessed
by normal eMMC read / write CMDs, it will cause below error
messages during kernel boot:
...
 mmc0: Got data interrupt 0x00000002 even though no data operation was in progress.
 mmcblk0rpmb: error -110 transferring data, sector 0, nr 32, cmd response 0x900, card status 0xb00
 mmcblk0rpmb: retrying using single block read
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 mmcblk0rpmb: timed out sending r/w cmd command, card status 0x400900
 end_request: I/O error, dev mmcblk0rpmb, sector 0
 Buffer I/O error on device mmcblk0rpmb, logical block 0
 end_request: I/O error, dev mmcblk0rpmb, sector 8
 Buffer I/O error on device mmcblk0rpmb, logical block 1
 end_request: I/O error, dev mmcblk0rpmb, sector 16
 Buffer I/O error on device mmcblk0rpmb, logical block 2
 end_request: I/O error, dev mmcblk0rpmb, sector 24
 Buffer I/O error on device mmcblk0rpmb, logical block 3
...

This patch will discard the access request in eMMC queue if
it is RPMB partition access request. By this way, it avoids
trigger above error messages.

Fixes: 090d25fe ("mmc: core: Expose access to RPMB partition")
Signed-off-by: NYunpeng Gao <yunpeng.gao@intel.com>
Signed-off-by: NChuanxiao Dong <chuanxiao.dong@intel.com>
Tested-by: NMichael Shigorin <mike@altlinux.org>
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>

For normal request mmc_blk_issue_rq is called twice with asynchronous
transfer(cur and prev). Host's claim and release can be done in each
mmc_blk_issue_rq. However, Special request is currently excluded in
asynchronous transfer. After special request is finished, if there is
no new request, mmc_release_host won't be called in mmc_blk_issue_rq.
The problem is founded during mmc_suspend.

[<c0541124>] (__schedule+0x0/0x78c) from [<c05419e8>] (schedule+0x38/0x78)
[<c05419b0>] (schedule+0x0/0x78) from [<c03a843c>] (__mmc_claim_host+0xac/0x1b4)
[<c03a8390>] (__mmc_claim_host+0x0/0x1b4) from [<c03ac98c>] (mmc_suspend+0x28/0x9c)
[<c03ac964>] (mmc_suspend+0x0/0x9c) from [<c03aad24>] (mmc_suspend_host+0xb4/0x194)
...
Reported-by: NJohan Rudholm <jrudholm@gmail.com>
Signed-off-by: NSeungwon Jeon <tgih.jun@samsung.com>
Tested-by: NJohan Rudholm <johan.rudholm@stericsson.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

This patch supports packed write command of eMMC4.5 devices.  Several
writes can be grouped in packed command and all data of the individual
commands can be sent in a single transfer on the bus. Large amounts of
data in one transfer rather than several data of small size are
effective for eMMC write internally.  As a result, packed command help
write throughput be improved.  The following tables show the results
of packed write.

Type A:
test     none |  packed
iozone   25.8 |  31
tiotest  27.6 |  31.2
lmdd     31.2 |  35.4

Type B:
test     none |  packed
iozone   44.1 |  51.1
tiotest  47.9 |  52.5
lmdd     51.6 |  59.2

Type C:
test     none |  packed
iozone   19.5 |  32
tiotest  19.9 |  34.5
lmdd     22.8 |  40.7
Signed-off-by: NSeungwon Jeon <tgih.jun@samsung.com>
Reviewed-by: NMaya Erez <merez@codeaurora.org>
Reviewed-by: NNamjae Jeon <linkinjeon@gmail.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

When current request is running on the bus and if next request fetched
by mmcqd is NULL, mmc context (mmcqd thread) gets blocked until the
current request completes. This means that if new request comes in while
the mmcqd thread is blocked, this new request can not be prepared in
parallel to current ongoing request. This may result in delaying the new
request execution and increase it's latency.

This change allows to wake up the MMC thread on new request arrival.
Now once the MMC thread is woken up, a new request can be fetched and
prepared in parallel to the current running request which means this new
request can be started immediately after the current running request
completes.

With this change read throughput is improved by 16%.
Signed-off-by: NKonstantin Dorfman <kdorfman@codeaurora.org>
Reviewed-by: NSeungwon Jeon <tgih.jun@samsung.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

When current request is running on the bus and if next request fetched
by mmcqd is NULL, mmc context (mmcqd thread) gets blocked until the
current request completes. This means that if new request comes in while
the mmcqd thread is blocked, this new request can not be prepared in
parallel to current ongoing request. This may result in delaying the new
request execution and increase it's latency.

This change allows to wake up the MMC thread on new request arrival.
Now once the MMC thread is woken up, a new request can be fetched and
prepared in parallel to the current running request which means this new
request can be started immediately after the current running request
completes.

With this change read throughput is improved by 16%.
Signed-off-by: NKonstantin Dorfman <kdorfman@codeaurora.org>
Reviewed-by: NSeungwon Jeon <tgih.jun@samsung.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Change mmc_blk_issue_rw_rq() to become asynchronous.
The execution flow looks like this:

* The mmc-queue calls issue_rw_rq(), which sends the request
  to the host and returns back to the mmc-queue.
* The mmc-queue calls issue_rw_rq() again with a new request.
* This new request is prepared in issue_rw_rq(), then it waits for
  the active request to complete before pushing it to the host.
* When the mmc-queue is empty it will call issue_rw_rq() with a NULL
  req to finish off the active request without starting a new request.
Signed-off-by: NPer Forlin <per.forlin@linaro.org>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NVenkatraman S <svenkatr@ti.com>
Tested-by: NSourav Poddar <sourav.poddar@ti.com>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

Add an additional mmc queue request instance to make way for two active
block requests. One request may be active while the other request is
being prepared.
Signed-off-by: NPer Forlin <per.forlin@linaro.org>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NVenkatraman S <svenkatr@ti.com>
Tested-by: NSourav Poddar <sourav.poddar@ti.com>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

The way the request data is organized in the mmc queue struct, it only
allows processing of one request at a time.  This patch adds a new struct
to hold mmc queue request data such as sg list, request, blk request and
bounce buffers, and updates any functions depending on the mmc queue
struct. This prepares for using multiple active requests in one mmc queue.
Signed-off-by: NPer Forlin <per.forlin@linaro.org>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Reviewed-by: NVenkatraman S <svenkatr@ti.com>
Tested-by: NSourav Poddar <sourav.poddar@ti.com>
Tested-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NChris Ball <cjb@laptop.org>

For example, an eMMC with 2 boot partitions will have 3 threads.
The names change from:

   40 ?        00:00:00 mmcqd/0
   41 ?        00:00:00 mmcqd/0
   42 ?        00:00:00 mmcqd/0

to:

   40 ?        00:00:00 mmcqd/0
   41 ?        00:00:00 mmcqd/0boot0
   42 ?        00:00:00 mmcqd/0boot1
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Cc: Andrei Warkentin <andreiw@motorola.com>
Signed-off-by: NChris Ball <cjb@laptop.org>

Some hosts cannot do scatter/gather in hardware. Since not doing sg
is such a big performance hit, we (optionally) bounce the requests
to a simple linear buffer that we hand over to the driver.
Signed-off-by: NPierre Ossman <drzeus@drzeus.cx>

Remove dead code and unused structs from the block driver.
Signed-off-by: NPierre Ossman <drzeus@drzeus.cx>

Clean up the drivers/mmc directory by moving card and host drivers
into subdirectories.
Signed-off-by: NPierre Ossman <drzeus@drzeus.cx>

The mmc block queue functions are tailored for the mmc_block
driver, so move those functions into that module.
Signed-off-by: NPierre Ossman <drzeus@drzeus.cx>

Replace kernel_thread() with kthread_run()/kthread_stop().
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NPierre Ossman <drzeus@drzeus.cx>

Initial git repository build. I'm not bothering with the full history,
even though we have it. We can create a separate "historical" git
archive of that later if we want to, and in the meantime it's about
3.2GB when imported into git - space that would just make the early
git days unnecessarily complicated, when we don't have a lot of good
infrastructure for it.

Let it rip!