Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

QUEUE_FLAG_* are flags (other than QUEUE_FLAG_DEFAULT), so they cannot
be ORed together.  Set the queue flags using queue_flag_set_unlocked().
Reported-by: NDonald Wood <donald.e.wood@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

By using the iod->nents field (the same way other I/O paths do), we can
avoid recalculating the number of sg entries at unmap time, and make
nvme_unmap_user_pages() easier to call.

Also, use the 'write' parameter instead of assuming DMA_FROM_DEVICE.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

For user I/O and admin commands, we were forgetting to mark the end of
the SG list.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

We were always mapping as DMA_FROM_DEVICE then unmapping with
DMA_TO_DEVICE which was clearly not correct.  Follow the same pattern as
nvme_submit_io() and key off the bottom bit of the opcode to determine
whether this is a read or a write.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

IO_TIMEOUT is a little too generic and might be used by other parts of
the kernel in the future.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The new merged data structure is called nvme_iod.  This improves performance
for mid-sized I/Os (in the 16k range) since we save a memory allocation.
It is also a slightly simpler interface to use.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The queue is only needed for some rare occasions, and it's more consistent
to pass the device around.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Upcoming patches require calling get_nvmeq when we don't have a namespace.
Some callers already have the device in a local variable anyway.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Instead of encoding the handler type in the bottom two bits of the
per-completion context pointer, store the handler function as well
as the context pointer.  This gives us more flexibility and the code
is clearer.  It comes at the cost of an extra 8k of memory per queue,
but this feels like a reasonable price to pay.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The driver was still using an old definition of Identify Controller
which only came to light once we started using the 'number of namespaces'
field properly.
Reported-by: NNisheeth Bhat <nisheeth.bhat@intel.com>
Reported-by: NKhosrow Panah <Khosrow.Panah@idt.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The doorbell stride allows devices to spread out their doorbells instead
of packing them tightly.  This feature was added as part of ECN 003.

This patch also enables support for more than 512 queues :-)
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

ECN 001 documented that namespace 0 is not valid.  Sending an Identify
with CNS of 0 and Namespace of 0 is an undefined command.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The existing calculation underestimated the number of pages required
as it did not take into account the pointer at the end of each page.
The replacement calculation may overestimate the number of pages required
if the last page in the PRP List is entirely full.  By using ->npages
as a counter as we fill in the pages, we ensure that we don't try to
free a page that was never allocated.
Signed-off-by: NNisheeth Bhat <nisheeth.bhat@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Instead of open-coding calls to nvme_submit_admin_cmd, these
small wrappers are simpler to use (the patch removes 14 lines from
nvme_dev_add() for example).
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The driver was allocating 8k of memory, then freeing 4k of it.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

dma_unmap_sg() must be called with the same 'nents' passed to
dma_map_sg(), not the number returned from dma_map_sg().
Signed-off-by: NNisheeth Bhat <nisheeth.bhat@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Our SG list was constructed to always fill the entire first page, even
if that was more than the length of the I/O.  This is probably harmless,
but some IOMMUs might do something bad.

Correcting the first call to sg_set_page() made it look a lot closer to
the sg_set_page() in the loop, so fold the first call to sg_set_page()
into the loop.
Reported-by: NNisheeth Bhat <nisheeth.bhat@intel.com>
Signed-off-by: NMatthew Wilcox <willy@linux.intel.com>

Missing 'break' in the switch statement meant that we'd fall through
to the 'return -EINVAL' case.

Remove the special-purpose IDENTIFY, GET_RANGE_TYPE, DOWNLOAD_FIRMWARE
and ACTIVATE_FIRMWARE commands.  Replace them with a generic ADMIN_CMD
ioctl that can submit any admin command.

Add a new ID ioctl that returns the namespace ID of the queried device.
It corresponds to the SCSI Idlun ioctl.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If the I/O was not completed by a single NVMe command, we add the
bio to the congestion list and wake up the kthread to resubmit it.
But the kthread calls remove_wait_queue() unconditionally, which
will oops if it's not on the wait queue.  So add the kthread to
the wait queue before waking it up.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

nvme_setup_io_queues() was assuming that a NULL return from
nvme_create_queue() was an out-of-memory error.  That's not necessarily
true; the adapter might return -EIO, for example.  Change the calling
convention to return an ERR_PTR on failure instead of NULL.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

For the benefit of reviewers, add comments to a few functions describing
their calling context
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If any of the memory allocations in nvme_setup_prps fail, handle it by
modifying the passed-in data length to reflect the number of bytes we are
actually able to send.  Also allow the caller to specify the GFP flags
they need; for user-initiated commands, we can use GFP_KERNEL allocations.

The various callers are updated to handle this possibility; the main
I/O path is already prepared for this possibility (as it may happen
due to nvme_map_bio being unable to map all the segments of the I/O).
The other callers return -ENOMEM instead of doing partial I/Os.
Reported-by: NAndi Kleen <andi@firstfloor.org>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The current approach of using the namespace ID as the minor number
doesn't work when there are multiple adapters in the machine.  Rather
than statically partitioning the number of namespaces between adapters,
dynamically allocate minor numbers to namespaces as they are detected.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Previously it was being implicitly included through some other header file
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

In the kthread, walk the list of outstanding I/Os and check they've not
hit the timeout.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The trailing '_data' on the end was annoying and inconsistent.  Also, make
it actually return the data since this is needed for timing out commands.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

When an I/O completed with an error, we would call bio_endio twice
(once with -EIO and once with 0).  Found by inspection.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

THe device reports (in its capability register) how long it will take
to initialise.  If that time elapses before the ready bit becomes set,
conclude the device is broken and refuse to initialise it.  Log a nice
error message so the user knows why we did nothing.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

We need to clear the affinity mask before calling free_irq()
Reported-by: NShane Michael Matthews <shane.matthews@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The arbitration field was extended by one bit, shifting the shutdown
notification bits by one.  Also, the SQ/CQ entry size was made
configurable for future extensions.
Reported-by: NPaul Luse <paul.e.luse@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The read and write commands don't define a 'result', so there's no need
to copy it back to userspace.

Remove the ability of the ioctl to submit commands to a different
namespace; it's just asking for trouble, and the use case I have in mind
will be addressed througha  different ioctl in the future.  That removes
the need for both the block_shift and nsid arguments.

Check that the opcode is one of 'read' or 'write'.  Future opcodes may
be added in the future, but we will need a different structure definition
for them.

The nblocks field is redefined to be 0-based.  This allows the user to
request the full 65536 blocks.

Don't byteswap the reftag, apptag and appmask.  Martin Petersen tells
me these are calculated in big-endian and are transmitted to the device
in big-endian.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

NVME_IOCTL_SUBMIT_IO has a struct nvme_user_io, not a struct nvme_rw_command
as a parameter, and NVME_IOCTL_DOWNLOAD_FW is a Write, not a Read.
Reported-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Make ioctls work for 32-bit applications on 64-bit kernels.  The structures
are defined to be the same for both 32- and 64-bit applications, so
we can use the same handler for both.
Reported-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Fill in all the num_possible_cpus() entries with duplicate pointers.
This reduces the complexity of the frequently-called get_nvmeq(), as
well as avoiding a bug in it when there are fewer queues than CPUs.
Reported-by: NShane Michael Matthews <shane.matthews@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Once there are no more bios on the congestion list, we can stop waking
up the nvme kthread every time a completion happens.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>