Add a pointer to the nvme_req_info to hold a new data structure
(nvme_prps) which contains a list of the pages allocated to this
particular request for holding PRP list entries.  nvme_setup_prps()
now returns this pointer.

To allocate and free the memory used for PRP lists, we need a struct
device, so we need to pass the nvme_queue pointer to many functions
which didn't use to need it.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

For multipage BIOs, we were always using sg[0] instead of advancing
through the list.  Oops :-)
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If POISON_POINTER_DELTA isn't defined, ensure they're in page 0 which
should never be mapped.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

In the bio completion handler, check for bios on the congestion list
for this NVM queue.  Also, lock the congestion list in the make_request
function as the queue may end up being shared between multiple CPUs.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

In addition to recording the completion data for each command, record
the anticipated completion time.  Choose a timeout of 5 seconds for
normal I/Os and 60 seconds for admin I/Os.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If we're sharing a queue between multiple CPUs and we cancel a sync I/O,
we must have the queue locked to avoid corrupting the stack of the thread
that submitted the I/O.  It turns out this is the same locking that's needed
for the threaded irq handler, so share that code.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If the adapter completes a command ID that is outside the bounds of
the array, return CMD_CTX_INVALID instead of random data, and print a
message in the sync_completion handler (which is rapidly becoming the
misc completion handler :-)
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Set the context value to CMD_CTX_COMPLETED, and print a message in the
sync_completion handler if we see it.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

I have plans for other special values in sync_completion.  Plus, this
is more self-documenting, and lets us detect bogus usages.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

We're currently calling bio_endio from hard interrupt context.  This is
not a good idea for preemptible kernels as it will cause longer latencies.
Using a threaded interrupt will run the entire queue processing mechanism
(including bio_endio) in a thread, which can be preempted.  Unfortuantely,
it also adds about 7us of latency to the single-I/O case, so make it a
module parameter for the moment.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

We can't have preemption disabled when we call schedule().  Accept the
possibility that we'll get preempted, and it'll cost us some cacheline
bounces.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

If the user sends a fatal signal, sleeping in the TASK_KILLABLE state
permits the task to be aborted.  The only wrinkle is making sure that
if/when the command completes later that it doesn't upset anything.
Handle this by setting the data pointer to 0, and checking the value
isn't NULL in the sync completion path.  Eventually, bios can be cancelled
through this path too.  Note that the cmdid isn't freed to prevent reuse.

We should also abort the command in the future, but this is a good start.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Because I wasn't setting driverfs_dev, the devices were showing up under
/sys/devices/virtual/block.  Now they appear underneath the PCI device
which they belong to.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

In case the card has been left in a partially-configured state,
write 0 to the Enable bit.
Signed-off-by: NShane Michael Matthews <shane.matthews@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Calling pci_request_selected_regions() reserves these regions for our use.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Need to call dma_set_coherent_mask() to allow queues to be allocated
above 4GB.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Need to call pci_set_master() to enable device DMA
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Call pci_enable_device_mem() at initialisation and pci_disable_device
at exit.
Signed-off-by: NShane Michael Matthews <shane.matthews@intel.com>
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

It can return NULL, so handle that.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

We don't keep a list of nvme_dev any more
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Allow userspace to submit synchronous I/O like the SCSI sg interface does.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

These are generalisations of the code that was in
nvme_submit_user_admin_command().
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Factor out most of nvme_identify() into a new nvme_submit_user_admin_command()
function.  Change nvme_get_range_type() to call it and change nvme_ioctl to
realise that it's getting back all 64 ranges.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Make sure there's no left-over bits set from previous commands that used
this slot.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Generalise the code from nvme_identify() that sets PRP1 & PRP2 so that
it's usable for commands sent by nvme_submit_bio_queue().
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

DMA the result straight to userspace instead of bounce-buffering in the
kernel.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

The admin IRQ is supposed to use the pin-based (or single message MSI)
interrupt.  Accomplish this by filling in entry[0]'s vector with the
INTx irq number.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

It's called the phase bit in the current draft
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

sq_count and cq_count are always the same, so just call it 'count'.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

Two callers with an almost identical long string of arguments, and
introducing a third soon.  Time to factor out the commonalities.
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>

This driver is for devices that follow the NVM Express standard
Signed-off-by: NMatthew Wilcox <matthew.r.wilcox@intel.com>