nvme-scsi.c uses several data structures and definitions that were
previously private to nvme-core.c.  Move the definitions to nvme.h,
protected by __KERNEL__.
Signed-off-by: NVishal Verma <vishal.l.verma@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

In preparation for adding nvme-scsi.c
It is preferable to retain the module name 'nvme'
Signed-off-by: NVishal Verma <vishal.l.verma@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

This adds discard support to block queues if the nvme device is capable of
deallocating blocks as indicated by the controller's optional command support.
A discard flagged bio request will submit an NVMe deallocate Data Set
Management command for the requested blocks.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The LBA Range Type feature is optional in the NVMe specification,
so we should continue with adding namespaces for controllers that do
not implement this feature.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

CONFIG_HOTPLUG is going away as an option.  As a result, the __dev*
markings need to be removed.

This change removes the use of __devinit, __devexit_p, __devinitdata,
__devinitconst, and __devexit from these drivers.

Based on patches originally written by Bill Pemberton, but redone by me
in order to handle some of the coding style issues better, by hand.

Cc: Bill Pemberton <wfp5p@virginia.edu>
Cc: Mike Miller <mike.miller@hp.com>
Cc: Chirag Kantharia <chirag.kantharia@hp.com>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Jim Paris <jim@jtan.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Matthew Wilcox <matthew.r.wilcox@intel.com>
Cc: Keith Busch <keith.busch@intel.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: NeilBrown <neilb@suse.de>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Tao Guo <Tao.Guo@emc.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

For commands that do not map a scatter list, we need to initilaize the iod's
number of sg entries (nents) to 0 and not unmap in this case.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

This data structure is defined in the NVMe specification.  It's not used
by the kernel, but is available for use by userspace software.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

nvme_get_features() was not returning the result.  Add a parameter
to return the result in (similar to nvme_set_features()) and change
all callers.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The ioctl data structure includes space for the 'result' of the admin
command to be returned; it just wasn't filled in.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If the queue has bios queued on it when it is freed, bio_endio() must be
called for them first.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

nvme_map_bio() is called after the cmdid is allocated, so we have to
free the cmdid before returning from nvme_submit_bio() if nvme_map_bio()
returned an error.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Covers the rest of the uses of pci error handler.
Signed-off-by: NStephen Hemminger <shemminger@vyatta.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

If the device is hot-unplugged while there are active commands, we should
time out the I/Os so that upper layers don't just see the I/Os disappear.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If the adapter fails initialisation, the memory allocated for the
admin queue may not be freed.  Split the memory freeing part of
nvme_free_queue() into nvme_free_queue_mem() and call it in the case of
initialisation failure.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>
Reported-by: NVishal Verma <vishal.l.verma@intel.com>

Signed-off-by: NQuoc-Son Anh <quoc-sonx.anh@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Commit 5c42ea16 used spaces instead of tabs.
Also remove the unnecessary initialisation of the 'result' variable.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

We should return here and avoid a NULL dereference.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Set the depth for IO queues to the device's maximum supported queue
entries if the requested depth exceeds the device's capabilities.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Set the max hw sectors in a namespace's request queue if the nvme device
has a max data transfer size.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The specification does not provide a use for command dword11 in the NVMe
Get Features command, but does use the NSID for some features.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The function nvme_user_admin_command does not require a namespace to
proceed.  Replace with the nvme_dev structure so that it can be called
from contexts that do not have a namespace.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

register_blkdev returns 0 when given a valid major number.
Reported-by: NRoss Zwisler <ross.zwisler@intel.com>
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Sets the request queue logical block size with the block size of the
namespace.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

This provides unified readq()/writeq() helper functions for 32-bit
drivers.

For some cases, readq/writeq without atomicity is harmful, and order of
io access has to be specified explicitly.  So in this patch, new two
header files which contain non-atomic readq/writeq are added.

 - <asm-generic/io-64-nonatomic-lo-hi.h> provides non-atomic readq/
   writeq with the order of lower address -> higher address

 - <asm-generic/io-64-nonatomic-hi-lo.h> provides non-atomic readq/
   writeq with reversed order

This allows us to remove some readq()s that were added drivers when the
default non-atomic ones were removed in commit dbee8a0a ("x86:
remove 32-bit versions of readq()/writeq()")

The drivers which need readq/writeq but can do with the non-atomic ones
must add the line:

  #include <asm-generic/io-64-nonatomic-lo-hi.h> /* or hi-lo.h */

But this will be nop in 64-bit environments, and no other #ifdefs are
required.  So I believe that this patch can solve the problem of
 1. driver-specific readq/writeq
 2. atomicity and order of io access

This patch is tested with building allyesconfig and allmodconfig as
ARCH=x86 and ARCH=i386 on top of tip/master.

Cc: Kashyap Desai <Kashyap.Desai@lsi.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Ravi Anand <ravi.anand@qlogic.com>
Cc: Vikas Chaudhary <vikas.chaudhary@qlogic.com>
Cc: Matthew Garrett <mjg@redhat.com>
Cc: Jason Uhlenkott <juhlenko@akamai.com>
Cc: James Bottomley <James.Bottomley@parallels.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Roland Dreier <roland@purestorage.com>
Cc: James Bottomley <jbottomley@parallels.com>
Cc: Alan Cox <alan@lxorguk.ukuu.org.uk>
Cc: Matthew Wilcox <matthew.r.wilcox@intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: NHitoshi Mitake <h.mitake@gmail.com>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

There's no need for drivers/block/nvme.c to include linux/version.h,
so remove the include.
Signed-off-by: NJesper Juhl <jj@chaosbits.net>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

The type of 'make_request_fn' changed in 5a7bbad2 ("block: remove
support for bio remapping from ->make_request"), but the merge of the
nvme driver didn't take that into account, and as a result the driver
would compile with a warning:

  drivers/block/nvme.c: In function 'nvme_alloc_ns':
  drivers/block/nvme.c:1336:2: warning: passing argument 2 of 'blk_queue_make_request' from incompatible pointer type [enabled by default]
  include/linux/blkdev.h:830:13: note: expected 'void (*)(struct request_queue *, struct bio *)' but argument is of type 'int (*)(struct request_queue *, struct bio *)'

It's benign, but the warning is annoying.
Reported-by: NStephen Rothwell <sfr@canb.auug.org>
Cc: Matthew Wilcox <matthew.r.wilcox@intel.com>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The number of submission & completion queues should be set by calling
Set Features, not Get Features.
Reported-by: NKwok Kong <Kwok.Kong@idt.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

QUEUE_FLAG_* are flags (other than QUEUE_FLAG_DEFAULT), so they cannot
be ORed together.  Set the queue flags using queue_flag_set_unlocked().
Reported-by: NDonald Wood <donald.e.wood@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

By using the iod->nents field (the same way other I/O paths do), we can
avoid recalculating the number of sg entries at unmap time, and make
nvme_unmap_user_pages() easier to call.

Also, use the 'write' parameter instead of assuming DMA_FROM_DEVICE.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

For user I/O and admin commands, we were forgetting to mark the end of
the SG list.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

We were always mapping as DMA_FROM_DEVICE then unmapping with
DMA_TO_DEVICE which was clearly not correct.  Follow the same pattern as
nvme_submit_io() and key off the bottom bit of the opcode to determine
whether this is a read or a write.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

IO_TIMEOUT is a little too generic and might be used by other parts of
the kernel in the future.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The new merged data structure is called nvme_iod.  This improves performance
for mid-sized I/Os (in the 16k range) since we save a memory allocation.
It is also a slightly simpler interface to use.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The queue is only needed for some rare occasions, and it's more consistent
to pass the device around.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Upcoming patches require calling get_nvmeq when we don't have a namespace.
Some callers already have the device in a local variable anyway.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Instead of encoding the handler type in the bottom two bits of the
per-completion context pointer, store the handler function as well
as the context pointer.  This gives us more flexibility and the code
is clearer.  It comes at the cost of an extra 8k of memory per queue,
but this feels like a reasonable price to pay.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The doorbell stride allows devices to spread out their doorbells instead
of packing them tightly.  This feature was added as part of ECN 003.

This patch also enables support for more than 512 queues :-)
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>