When the maximum send wr delivered by the user is zero, the qp does not
have a sq.

When allocating the sq db buffer to store the user sq pi pointer and map
it to the kernel mode, max_send_wr is used as the trigger condition, while
the kernel does not consider the max_send_wr trigger condition when
mapmping db. It will cause sq record doorbell map fail and create qp fail.

The failed print information as follows:

 hns3 0000:7d:00.1: Send cmd: tail - 418, opcode - 0x8504, flag - 0x0011, retval - 0x0000
 hns3 0000:7d:00.1: Send cmd: 0xe59dc000 0x00000000 0x00000000 0x00000000 0x00000116 0x0000ffff
 hns3 0000:7d:00.1: sq record doorbell map failed!
 hns3 0000:7d:00.1: Create RC QP failed

Fixes: 0425e3e6 ("RDMA/hns: Support flush cqe for hip08 in kernel space")
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

The method of hem free for SCC context is different from qp context.

In the current version, if free SCC hem during the execution of qp free,
there may be smmu error as below:

 arm-smmu-v3 arm-smmu-v3.1.auto: event 0x10 received:
 arm-smmu-v3 arm-smmu-v3.1.auto:  0x00007d0000000010
 arm-smmu-v3 arm-smmu-v3.1.auto:  0x000012000000017c
 arm-smmu-v3 arm-smmu-v3.1.auto:  0x00000000000009e0
 arm-smmu-v3 arm-smmu-v3.1.auto:  0x0000000000000000

As SCC context is still used by hardware after qp free, we can solve this
problem by removing SCC hem free from hns_roce_qp_free.

Fixes: 6a157f7d ("RDMA/hns: Add SCC context allocation support for hip08")
Signed-off-by: NYangyang Li <liyangyang20@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

Now when we have the udata passed to all the ib_xxx object creation APIs
and the additional macro 'rdma_udata_to_drv_context' to get the
ib_ucontext from ib_udata stored in uverbs_attr_bundle, we can finally
start to remove the dependency of the drivers in the
ib_xxx->uobject->context.
Signed-off-by: NShamir Rabinovitch <shamir.rabinovitch@oracle.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

The memory allocated for wrid should be initialized to zero.
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

Use the for_each_sg_dma_page iterator variant to walk the umem DMA-mapped
SGL and get the page DMA address. This avoids the extra loop to iterate
pages in the SGE when for_each_sg iterator is used.

Additionally, purge umem->page_shift usage in the driver as its only
relevant for ODP MRs. Use system page size and shift instead.
Signed-off-by: NShiraz, Saleem <shiraz.saleem@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch adds SCC context clear support for DCQCN in kernel space
driver.
Signed-off-by: NYangyang Li <liyangyang20@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch adds SCC context allocation and initialization support for
DCQCN in kernel space driver.
Signed-off-by: NYangyang Li <liyangyang20@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

When flush cqe, it needs to get the pointer of rq and sq from db address
space of user and update it into qp context by modified qp. if rq does not
exist, it will not get the value from db address space of user.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

ib_umem_get() can only be called in a method callback, which always has a
udata parameter. This allows ib_umem_get() to derive the ucontext pointer
directly from the udata without requiring the drivers to find it in some
way or another.
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>
Signed-off-by: NShamir Rabinovitch <shamir.rabinovitch@oracle.com>

In some application scenario, the user could not have receive queue when
run rdma write or read operation.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

Drivers should be using udata to determine if a method is invoked from
user space or kernel space. A pd does not necessarily say a different
objects is kernel or user.

Transforming the tests to use udata eliminates a large number of uobject
references from the drivers.
Signed-off-by: NShamir Rabinovitch <shamir.rabinovitch@oracle.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch implements the SRQ(Share Receive Queue) verbs
and update the poll cq verbs to deal with SRQ complentions.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

According to hip08 limit, the buffer size of extend sge needs to be an
integer wqe_sge_buf_page size. For example, the value of sge_shift field
of qp context is greater or equal to eight when buffer page size is 4K
size. The value of sge_shift field of qp context assigned by
hr_qp->sge.sge_cnt.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

The hip08 split two hardware version. The version id are 0x20 and 0x21
according to the PCI revison. The max size of extend sge of sq is limited
to 2M for 0x20 version and 8M for 0x21 version. It may be exceeded to 2M
according to the algorithm that compute the product of wqe count and
extend sge number of every wqe. But the product always less than 8M.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

It needs to include two special qps for every port. The hip08 have four
ports and the all reserved qp numbers are eight.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

The ll parameter is not used in ib_modify_qp_is_ok(), so remove it.
Signed-off-by: NKamal Heib <kamalheib1@gmail.com>
Reviewed-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

hns bitmap allocation functions return 0 on success and -1 on failure.
Callers of these functions wrongly used their return value as an errno,
fix that by making a proper conversion.

Fixes: a598c6f4 ("IB/hns: Simplify function of pd alloc and qp alloc")
Signed-off-by: NGal Pressman <pressmangal@gmail.com>
Acked-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

According to IB protocol, there are some cases that work requests must
return the flush error completion status through the completion queue. Due
to hardware limitation, the driver needs to assist the flush process.

This patch adds the support of flush cqe for hip08 in the cases that
needed, such as poll cqe, post send, post recv and aeqe handle.

The patch also considered the compatibility between kernel and user space.
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

Because hip06 hardware is not support for qp transition from
reset to reset state, it need to return errno when qp
transited from reset to reset. This patch fixes it.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

When used rq record db for kernel, it needs to set the rdb_en of
hr_qp to 1 and configures the dma address of record rq db of qp
context.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

This patch fixes two spelling errors.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

This patch changes the type of cqn from u32 to u64 to keep
userspace and kernel consistent, initializes resp both for
cq and qp to zeros, and also changes the condition judgment
of outlen considering future caps extension.
Suggested-by: NJason Gunthorpe <jgg@mellanox.com>
Fixes: e088a685 (hns: Support rq record doorbell for the user space)
Fixes: 9b44703d (hns: Support cq record doorbell for the user space)
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch updates to support rq record doorbell for
the kernel space.
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

This patch adds interfaces and definitions to support the rq record
doorbell for the user space.
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

The hip06 and hip08 run on a little endian ARM, it needs to
revise the annotations to indicate that the HW uses little
endian data in the various DMA buffers, and flow the necessary
swaps throughout.

The imm_data use big endian mode. The cpu_to_le32/le32_to_cpu
swaps are no-op for this, which makes the only substantive
change the handling of imm_data which is now mandatory swapped.

This also keep match with the userspace hns driver and resolve
the warning by sparse.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

The gsi qp and rc qp use the same qp context structure and the created
flow, only differentiate them by qpn and qp type.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

This patch mainly implement rq inline data feature for hip08
RoCE in kernel mode.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Reviewed-by: NLeon Romanovsky <leonro@mellanox.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

Considering the compatibility of supporting hip08's eq
process and possible changes of data structure, this patch
refactors the eq code structure of hip06.

We move all the eq process code for hip06 from hns_roce_eq.c
into hns_roce_hw_v1.c, and also for hns_roce_eq.h. With
these changes, it will be convenient to add the eq support
for later hardware version.
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Reviewed-by: NLijun Ou <oulijun@huawei.com>
Reviewed-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NJason Gunthorpe <jgg@mellanox.com>

The TRRL(Target RDMA Read/aTOMIC List) record the information
of receiving RDMA READ or ATOMIC operation in hip08. It will
be used the hardware. The driver need to assign a continuous
physical address for trrl_ba field of qp context.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NYixian Liu <liuyixian@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

This patch updates to support WQE, CQE and PBL page size configurable
feature, which includes base address page size and buffer page size.
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

This patch is implementing for posting send request and
receiving request for hip08 RoCE driver. such as
post send verbs and post recv verbs.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

This patch implements QP operations for hip08 RoCE driver and
fixes some checkpatch warning about print message in QP function.
The QP operations includes create QP, query QP, modify QP and
destroy QP.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

In hip08, the SQWQE/SGE/RQWQE and CQE have different hop num and
page size, so we need to manage the base address table of the
SQWQE/SGE/RQWQE and CQE separately.

This patch is to split CQE from MTT(SQWQE/SGE/RQWQE).
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

In order to support the scalability of the hardware version, the
features irrelevant to the hardware will be located in the hns-roce.ko,
and the hardware relevant operations will be located in hns_roce_hw_v1.ko
or hns_roce_hw_v2.ko based on the series chips.

The hip08 RoCE engine is a PCI device, hip06 RoCE engine is a platform
device. In order to support both platform device and PCI device, We
replace &hr_dev->pdev->dev with hr_dev->dev in hns-roce.ko as belows:
	Before modification:
		struct device *dev = hr_dev->dev;
	After modification:
		struct device *dev = &hr_dev->pdev->dev;

	The related structure:
	struct hns_roce_dev {
		...
		struct platform_device  *pdev;
		struct pci_dev		*pci_dev;
		struct device		*dev;
		...
	}
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

The hardware relevant definitions and operations are implemented
in hns_roce_hw_v* file. According to the diversity chips, the file
is named as hns_roce_hw_v1.c or hns_roce_hw_v2.c etc.

The general software process flow, common structures and allocated
algorithms are implemented in other files located in hns roce driver.

In order to support the scalability of the hardware version, the
common driver features are in the hns-roce.ko, and the hardware
relevant operations are in hns_roce_hw_v1.ko or hns_roce_hw_v2.ko
based on the series chips.
Signed-off-by: NLijun Ou <oulijun@huawei.com>
Signed-off-by: NShaobo Xu <xushaobo2@huawei.com>
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

drivers/infiniband/hw/hns/hns_roce_qp.c:802:9-10: WARNING: return of 0/1 in function 'hns_roce_wq_overflow' with return type bool

 Return statements in functions returning bool should use
 true/false instead of 1/0.
Generated by: scripts/coccinelle/misc/boolreturn.cocci

Fixes: 7d1b6a678e0b ("IB/hns: Support compile test for hns RoCE driver")
CC: Matan Barak <matanb@mellanox.com>
Signed-off-by: NFengguang Wu <fengguang.wu@intel.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Size of pages are held by struct ib_umem in page_size field.

It is better to store it as an exponent, because page size by nature
is always power-of-two and used as a factor, divisor or ilog2's argument.

The conversion of page_size to be page_shift allows to have portable
code and avoid following error while compiling on ARM:

  ERROR: "__aeabi_uldivmod" [drivers/infiniband/core/ib_core.ko] undefined!

CC: Selvin Xavier <selvin.xavier@broadcom.com>
CC: Steve Wise <swise@chelsio.com>
CC: Lijun Ou <oulijun@huawei.com>
CC: Shiraz Saleem <shiraz.saleem@intel.com>
CC: Adit Ranadive <aditr@vmware.com>
CC: Dennis Dalessandro <dennis.dalessandro@intel.com>
CC: Ram Amrani <Ram.Amrani@Cavium.com>
Signed-off-by: NArtemy Kovalyov <artemyko@mellanox.com>
Signed-off-by: NLeon Romanovsky <leon@kernel.org>
Acked-by: NRam Amrani <Ram.Amrani@cavium.com>
Acked-by: NShiraz Saleem <shiraz.saleem@intel.com>
Acked-by: NSelvin Xavier <selvin.xavier@broadcom.com>
Acked-by: NSelvin Xavier <selvin.xavier@broadcom.com>
Acked-by: NAdit Ranadive <aditr@vmware.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

Prepare for removal of ib_device.dma_device.
Signed-off-by: NBart Van Assche <bart.vanassche@sandisk.com>
Cc: Lijun Ou <oulijun@huawei.com>
Cc: Wei Hu(Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

This patch moves HNS vendor's specific structures to
common UAPI folder which will be visible to all consumers.
Signed-off-by: NLeon Romanovsky <leon@kernel.org>
Signed-off-by: NDoug Ledford <dledford@redhat.com>

When using CM to establish connections, qp number that was freed
just now will be rejected by ib core. To fix these problem, We
change qpn allocation to round-robin mode. We added the round-robin
mode for allocating resources using bitmap. We use round-robin mode
for qp number and non round-robing mode for other resources like
cq number, pd number etc.
Signed-off-by: NWei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: NSalil Mehta  <salil.mehta@huawei.com>
Signed-off-by: NDoug Ledford <dledford@redhat.com>