To reduce latency when initializing WQE content, create templates for the
most common wqes. This reduces the number of operations taken to set the
content. It's not a lot of speed up, but every bit helps.

This patch updates the NVME initiator path.

[mkp: fixed typo]
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The driver is very sloppy about the WQE structure passed between routines.
The base struct type is a 64byte wqe. But in many routines they typecast and
access 128byte wqes. There were a couple of cases in the past (corrected
already) where the typecasts were incorrectly done and the 64byte buffer was
accessed as a 128 byte buffer.

Clean this up by properly declaring wqe's as 128byte wqe's and removing the
typecasts. 64byte wqes are considered a subset of the 128byte wqes.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

First Burst support was not properly indicated in NVMe PRLI.

Correct the bit position and the logic to check and set first burst support.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Commit 6e8e1c14 ("scsi: lpfc: Add WQ Full Logic for NVME Target") fails
the static checker. Checker correctly identified a missing unlock on a
return path.

Add the unlock.

Fixes: 6e8e1c14 ("scsi: lpfc: Add WQ Full Logic for NVME Target")
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Use dma_pool_zalloc() instead of dma_pool_alloc + memset
Signed-off-by: NSouptick Joarder <jrdr.linux@gmail.com>
Acked-by: NBrian King <brking@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Commit 1351e69f ("scsi: lpfc: Add push-to-adapter support to sli4")
fails compilation on some 32-bit systems as writeq() is not supported on
all architectures. Additionally, it was pointed out that as writeX()
does byteswapping if necessary for pci vs the cpu endianness, the code
was broken on BE PPC.

After discussions with Arnd Bergmann, we've resolved the issue
to the following:
  Instead of writeX(), use __raw_writeX() - which writes to io
    space while preserving byte order. To use this, the code
    was changed to use a different buffer that lpfc prepped
    via sli_pcimem_bcopy() that was set to the bytestream to
    be written.
  On platforms with __raw_writeq support, use the routine, otherwise
    use __raw_writel()

[mkp: checkpatch]

Fixes: 1351e69f ("scsi: lpfc: Add push-to-adapter support to sli4")
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

In scsi core, __scsi_queue_insert should just put request back on the
queue and retry using the same command as before. However, for blk-mq,
scsi_mq_requeue_cmd is employed here which will unprepare the
request. To align with the semantics of __scsi_queue_insert, use
blk_mq_requeue_request with kick_requeue_list == true and put the
reference of scsi_device.

Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: NJianchao Wang <jianchao.w.wang@oracle.com>
Reviewed-by: NBart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Use dma_pool_zalloc() instead of dma_pool_alloc + memset
Signed-off-by: NSouptick Joarder <jrdr.linux@gmail.com>
Acked-by: NSumit Saxena <sumit.saxena@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Reviewed-by: NScott Benesh <scott.benesh@microsemi.com>
Signed-off-by: NKevin Barnett <kevin.barnett@microsemi.com>
Signed-off-by: NDon Brace <don.brace@microsemi.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The newly added code mixes up phys_addr_t/resource_size_t with dma_addr_t
and void pointers, as seen from these compiler warning:

drivers/scsi/mpt3sas/mpt3sas_base.c: In function '_base_get_chain_phys':
drivers/scsi/mpt3sas/mpt3sas_base.c:235:21: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
  base_chain_phys  = (void *)ioc->chip_phys + MPI_FRAME_START_OFFSET +
                     ^
drivers/scsi/mpt3sas/mpt3sas_base.c: In function '_clone_sg_entries':
drivers/scsi/mpt3sas/mpt3sas_base.c:427:20: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
    sgel->Address = (dma_addr_t)dst_addr_phys;
                    ^
drivers/scsi/mpt3sas/mpt3sas_base.c:438:7: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
       (dma_addr_t)buff_ptr_phys;
       ^
drivers/scsi/mpt3sas/mpt3sas_base.c:444:10: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
          (dma_addr_t)buff_ptr_phys;

Both dma_addr_t and phys_addr_t may be wider than a pointer, so we must
avoid the conversion to pointer types. This also helps readability.

A second problem is treating MMIO addresses from a 'struct resource'
as addresses that can be used for DMA on that device. In almost all
cases, those are the same, but on some of the more obscure architectures,
PCI memory address 0 is mapped into the CPU address space at a nonzero
offset. I don't have a good fix for that, so I'm adding a comment here,
plus a WARN_ON() that triggers whenever the phys_addr_t number is
outside of the low 32-bit address space and causes a straight overflow
when assigned to the 32-bit sgel->Address.

Fixes: 182ac784 ("scsi: mpt3sas: Introduce Base function for cloning.")
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NSreekanth Reddy <Sreekanth.Reddy@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add a file for documenting SCSI sd module parameters and describe the
cache_type setting.

[mkp: tweaked text a bit]
Signed-off-by: NWeiping Zhang <zhangweiping@didichuxing.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

On 64 bit CPUs there is a memory corruption bug on probe().  It should
be a u32 pointer instead of an unsigned long pointer or we write past
the end of the setupdata[] array.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Reviewed-by: NHannes Reinecke <hare@suse.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

This patch fixes the byte order of the SGPIO api and brings it back in
sync with ledmon v0.80 and above.

[mkp: added missing SoB and fixed whitespace]
Signed-off-by: NWilfried Weissmann <wilfried.weissmann@gmx.at>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Avoid that the kernel-doc tool complains about mismatches between
kernel-doc headers and function definitions. Avoid that errors like the
following are reported when building the UFS driver with W=1:

drivers/scsi/ufs/tc-dwc-g210-pci.c:60: error: Cannot parse struct or union!
drivers/scsi/ufs/tc-dwc-g210-pltfrm.c:26: warning: cannot understand function prototype: 'struct ufs_hba_variant_ops tc_dwc_g210_20bit_pltfm_hba_vops = '
Signed-off-by: NBart Van Assche <bart.vanassche@wdc.com>
Cc: Vivek Gautam <vivek.gautam@codeaurora.org>
Cc: Stanislav Nijnikov <stanislav.nijnikov@wdc.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Avoid that the kernel-doc tool complains about a mismatch between the
kernel-doc header and the function argument list.
Signed-off-by: NBart Van Assche <bart.vanassche@wdc.com>
Cc: Damien Le Moal <damien.lemoal@wdc.com>
Cc: Hannes Reinecke <hare@suse.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Sending I/O through 32 bit descriptors to Ventura series of controller
results in IO timeout on certain conditions. This error only occurs on
systems with high I/O activity.

Changes in this patch will prevent driver from using 32 bit descriptor
and use 64 bit Descriptors
Signed-off-by: NSuganath Prabu S <suganath-prabu.subramani@broadcom.com>
Reviewed-by: NTomas Henzl <thenzl@redhat.com>
Reviewed-by: NHannes Reinecke <hare@suse.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If the posted request has an error of any type, the IOC writes
a Reply message into a host-based system reply message frame.
This functions clone it in the BAR0 mapped region.
Signed-off-by: NSuganath Prabu S <suganath-prabu.subramani@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

1) Added function _base_clone_mpi_to_sys_mem to clone
MPI request into system BAR0 mapped region.

2) Separate out MPI Endpoint IO submissions to function
_base_put_smid_mpi_ep_scsi_io.

3) MPI EP requests are submitted in two 32 bit MMIO writes.
from _base_mpi_ep_writeq.

 For 32 bit Arch,_base_writeq function is identical
to _base_mpi_ep_writeq, Removed duplicate code as suggested.
Signed-off-by: NSuganath Prabu S <suganath-prabu.subramani@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

All scsi IO's and config request's data buffer and sgl are cloned to
system memory in _clone_sg_entries before submitting it to firmware.
Signed-off-by: NSuganath Prabu S <suganath-prabu.subramani@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

For MPI Endpoint/Mcpu, driver should double buffer data buffer/SGLs.
This is normally copied from host to internal memory of IOC by DMA
engine of PCI device. Since the interface to DMA from host to mCPU is
not present for Mcpu/MPI Endpoint device, driver does double copy of
those buffers directly to the mCPU memory region via BAR0 region.

Introduced API to calculate and return BAR0 mapped host buffer's
physical and virtual address for the provided smid.
Signed-off-by: NSuganath Prabu S <suganath-prabu.subramani@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

This configures shost max sector to 128, single reply descriptor post
queue, sgl table size to 16 and 32 bit DMA for MPI Endpoint and it
supports 64K as max IO.
Signed-off-by: NSuganath Prabu S <suganath-prabu.subramani@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add device ID and flag for Andromeda/MPI Endpoint.

[mkp: typo]
Signed-off-by: NSuganath Prabu S <suganath-prabu.subramani@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

No functional changes. Just fix two wrong indentation cases in
scsi_finish_command and scsi_decide_disposition.
Signed-off-by: NJianchao Wang <jianchao.w.wang@oracle.com>
Reviewed-by: NBart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

A bugfix I did caused a build regression in some other randconfig
builds in a rare combination of options:

In file included from drivers/scsi/qedi/qedi_fw.c:16:
drivers/scsi/qedi/qedi_gbl.h:26:38: error: array type has incomplete element type 'struct qedi_debugfs_ops'
 extern const struct qedi_debugfs_ops qedi_debugfs_ops[];
                                      ^~~~~~~~~~~~~~~~

This removes the useless #ifdef around the declarations in qedi_dbg.h
to make it always build.

Fixes: 779936fa ("scsi: qedi: fix building with LTO")
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Avoid that building with W=1 causes the kernel-doc tool to complain
about function arguments that have not been documented in the libsas
kernel-doc headers. Avoid that the short description starts with a
hyphen by changing "--" into "-" in the first line of the kernel-doc
headers.
Signed-off-by: NBart Van Assche <bart.vanassche@wdc.com>
Cc: John Garry <john.garry@huawei.com>
Reviewed-by: NJohn Garry <john.garry@huawei.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Updated Copyright in files updated as part of 12.0.0.0
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Update the driver version to 12.0.0.0
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The hardware offload for NVME commands was created when the
FC-NVME standard was setting SGL Descriptor Type to SGL Data
Block Descriptor (0h) and SGL Descriptor Sub Type to Address (0h).

A late change in NVMe-over-Fabrics obsoleted these values, creating
a transport SGL descriptor type with new values to go into these
fields.

For initial hardware support, in order to be compliant to the spec,
use host-supplied cmd IU buffers instead of the adapter generated
values. Later hardware will correct this.

Add a module parameter to override this offload disablement if looking
for lowest latency. This is reasonable as nothing in FC-NVME uses
the SQE SGL values.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Newer hardware more strictly enforces buffer lenghts, causing an
mis-set value to be identified. Older hardware won't catch it.
The difference is benign on old hardware.

Set the right embedded buffer length for nvme ios.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The current driver isn't taking advantage of a performance hint whereby
the initial data buffer descriptor can be placed in the WQE as well as
the SGL.

Add the logic to detect support for the feature and to use it when
supported.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Current code is very explicit in what it allows to be downloaded.
The driver checking prevented G7 firmware download. The driver
checking is unnecessary as the device will validate what it receives.

Revise the firmware download interface checking.
Added a little debug support in case there is still a failure.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Traditional SLI4 required the driver to clear Valid bits on
EQEs and CQEs after consuming them.

The new if_type=6 hardware will cycle the value for what is
valid on each queue itteration. The driver no longer has to
touch the valid bits. This also means all the cpu cache
dirtying and perhaps flush/refill's done by the hardware
in accessing the EQ/CQ elements is eliminated.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The G7 adapter supports 64G link speeds. Add support to the driver.

In addition, a small cleanup to replace the odd bitmap logic with
a switch case.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add PCI ids for the new G7 adapter
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

New if_type=6 adapters support an additional BAR that provides
apertures to allow direct WQE to adapter push support - termed
Direct Packet Push (DPP). WQ creation differs slightly to ask for
a WQ to be DPP-ized. When submitting a WQE to a DPP WQ, it is
submitted to the host memory for the WQ normally, but is also
written by the host cpu directly to a BAR aperture.  Write buffer
coalescing in hardware is (hopefully) turned on, enabling single
pci write operation support. The doorbell is thing rung to indicate
the WQE is available and was pushed to the aperture.

This patch:
- Updates the WQ Create commands for the DPP options
- Adds the bar mapping for if_type=6 DPP bar
- Adds the WQE pushing to the DDP aperture received from WQ create
- Adds a new module parameter to disable DPP operation if desired.
  Default is enabled.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

New hardware supports a SLI-4 interface, but with a new if_type
variant of 6.

If_type=6 has a different PCI BAR map, separate EQ/CQ doorbells,
and some changes in doorbell formats.

Add the changes for the if_type into headers, adapter initialization
and control flows. Add new eq and cq handlers.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Up until now, all SLI-4 devices had the same doorbells at the same
bar locations. With newer hardware, there are now independent EQ and
CQ doorbells and the bar locations differ.

Prepare the code for new hardware by separating the eq/cq doorbell into
separate components. The components can be set based on if_type.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Up until now, an SLI-4 device had no variance in the way it handled
its EQs and CQs. With newer hardware, there are now differences in
doorbells and some differences in how entries are valid.

Prepare the code for new hardware by creating a sli4-based callout
table that can be set based on if_type.
Signed-off-by: NDick Kennedy <dick.kennedy@broadcom.com>
Signed-off-by: NJames Smart <james.smart@broadcom.com>
Reviewed-by: NJohannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Use dma_pool_zalloc() instead of dma_pool_alloc + memset
Signed-off-by: NSouptick Joarder <jrdr.linux@gmail.com>
Acked-by: NHimanshu Madhani <himanshu.madhani@cavium.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Use dma_pool_zalloc() instead of dma_pool_alloc + memset
Signed-off-by: NSouptick Joarder <jrdr.linux@gmail.com>
Acked-by: NManish Rangankar <Manish.Rangankar@cavium.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>