This patch does not change any functionality.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

- iser_hdr
- iser_data_buf
- iser_mem_reg
- iser_regd_buf
- iser_tx_desc
- iser_rx_desc
- iser_device
- iser_pi_context
- iser_conn
- ib_conn
- iser_comp
- iscsi_iser_task
- iser_global

While we're at it, change nit alignments in this file

This patch does not change any functionality.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Change logging: "iser:XXXX" to "iser: XXXX"
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Match to the debug level of all functions in connect/disconnect flows.
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Singal completion of every 32 scsi commands and suppress all the rest.
We don't do anything upon getting the completion so no need to "just
consume" it.  Cleanup of scsi command is done in cleanup_task callback.

Still keep dataout and control send completions as we may need to
cleanup there. This helps reducing the amount of interrupts/completions
in the IO path.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Poll in batch of 16. Since we don't want it on the stack, keep under
iser completion context (iser_comp).
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Avoid post_send counting (atomic) in the IO path just to keep track of
how many completions we need to consume.  Use a beacon post to indicate
that all prior posts completed.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

This will solve a possible condition where we might miss TX completion
(flush error) during session teardown.  Since we are using a single
CQ, we don't need to actively drain the TX CQ, instead just wait for
flush_completion (when counters reach zero) and remove iser_poll_for_flush_errors().

This patch might introduce a minor performance regression on its own,
but the next patches will enhance performance using a single CQ for RX
and TX.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

We need a way to guarentee that we don't stay in soft-IRQ context for
too long.  We might starve other pending CQ tasklets or worse lock
against application trying to issue IO on the running CPU.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Introduce iser_comp which centralizes all iser completion related
items and is referenced by iser_device and each ib_conn.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

In case iscsid was violently killed (SIGKILL) during its error
recovery stage, we may never get a connection teardown sequence for
some of the old connections.  No harm done, but when we try to unload
the module we will need to cleanup all these connections.  So we
actually may end-up here - so it's not a BUG_ON(), just give a relaxed
warning that this happened and continue with normal unload.  BUG_ON()
will cause segfault on module_exit and we don't want that.
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Previously we notified iscsi layer about the connection layer when
we consumed all of our flush errors. This was racy as there
was no guarentee that iscsi_conn wasn't terminated by then (which ends
up in an invalid memory access). In case we got a non FLUSH error
completion, we are guarenteed that iscsi_conn is still alive. We should
notify iSCSI layer with iscsi_conn_failure to initiate error handling.

While we are at it, add a nice kernel-doc style documentation.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Bailout in case a task cleanup (iscsi_iser_cleanup_task) is called
after the IB device was removed (DEVICE_REMOVAL CM event).  We also
call iscsi_conn_stop with a lock taken to prevent DEVICE_REMOVAL and
tasks cleanup from racing.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Previously we didn't need to unbind the iser_conn and iscsi_conn since
we always relied on iscsi daemon to teardown the connection and never
let it finish before we cleanup all that is needed in iser.  This is
not the case anymore (for DEVICE_REMOVAL event).  So avoid any possible
chance we cause iscsi_conn dereference after iscsi_conn was freed.

We also call iser_conn_terminate (safe to call multiple times) just
for the corner case of iscsi daemon stopping an old connection before
invoking endpoint removal (might happen if it was violently killed).

Notice we are unbinding under a lock - which is required.
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

We no longer rely on iscsi connection teardown sequence, so no need to
give a grace period and continue cleanup if it expired. Have
iser_conn_release wait for full completion before freeing iser_conn.

ib_completion:
	Guaranteed to come when:
	    - Got DISCONNECTED/ADDR_CHANGE event or
	    - iSCSI called ep_disconnect/conn_stop
	Guaranteed to finish when:
	    - Got TIMEWAIT_EXIT/DEVICE_REMOVAL event
	    - All Flush errors are consumed
	    - IB related resources are destroyed

stop_completion:
	Guaranteed to come when:
	    - iSCSI calls conn_stop
	Guaranteed to finish when:
	    - All inflight tasks were cleaned up
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

iscsi daemon is in user-space, thus we can't rely on it to be invoked
at connection teardown (if not running or does not receive CPU time).

This patch addresses the issue by re-structuring iSER connection
teardown logic and CM events handling.

The CM events will dictate the RDMA resources destruction (ib_conn)
and iser_conn is kept around as long as iscsi_conn is left around
allowing iscsi/iser callbacks to continue after RDMA transport was
destroyed.

This patch introduces a separation in logic when handling CM events:

- DISCONNECTED_HANDLER, ADDR_CHANGED
  This events indicate the start of teardown process.
  Actions:
  1. Terminate the connection: rdma_disconnect (send DREQ/DREP)
  2. Notify iSCSI of connection failure
  3. Change state to TERMINATING
  4. Poll for all flush errors to be consumed

- TIMEWAIT_EXIT, DEVICE_REMOVAL
  These events indicate the final stage of termination process and
  we can free RDMA related resources.
  Actions:
  1. Call disconnected handler (we are not guaranteed that DISCONNECTED
     event was invoked in the past)
  2. Cleanup RDMA related resources
  3. For DEVICE_REMOVAL return non-zero rc from cma_handler to
     implicitly destroy the cm_id (Can't rely on user-space, make sure
     we have forward progress)

We replace flush_completion (indicate all flushes were consumed) with
ib_completion (rdma resources were cleaned up).

The iser_conn_release_work will wait for teardown completions:

- conn_stop was completed (tasks were cleaned-up) - stop_completion
- RDMA resources were destroyed - ib_completion

And then will continue to free iser connection representation (iser_conn).
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Put all connection IB related resources release in this routine.  One
exception is the cm_id which cannot be destroyed as the routine is
protected by the state mutex.  Also move its position to avoid forward
declaration.  While at it fix qp NULL assignment.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Structure that describes the RDMA relates connection objects.  Static
member of iser_conn.

This patch does not change any functionality
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Two reasons why we choose to do this:

1. No point today calling struct iser_conn by another name ib_conn
2. In the next patches we will restructure iser control plane representation
   - struct iser_conn: connection logical representation
   - struct ib_conn: connection RDMA layout representation

This patch does not change any functionality.
Signed-off-by: NAriel Nahum <arieln@mellanox.com>
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

To catch future errors sooner.
Signed-off-by: NEric Dumazet <edumazet@google.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We had several problems here.  First, a race condition on QP1 mac
handling between mlx4_ib_update_qps and mlx4_ib_modify_qp, which is
fixed by taking the qp mutex in mlx4_ib_update_qps.

Also, qp->pri.smac_port was not updated in mlx4_ib_update_qps.

Last, in __mlx4_ib_modify_qp we did not properly handle the case where
the mac is zero, but port is non-zero.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Automatic Path Migration is not supported under RoCE. Therefore,
return a "not-supported" error if the caller attempts to set an
alternate path in a QP context.

In addition, if there are no IB ports configured, do not report
APM capability in the device flags returned by mlx4_ib_query_device.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

For native functions (non-SR-IOV), there's no reason to update
the smac_index, as QP1 is a GSI QP.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

The source MAC is needed in RoCE when building the QP1 header.

Currently, this is obtained from the source net device. However, the net
device may not yet exist, or can be destroyed in parallel to this QP1 send
operation (e.g through the VPI port change flow) so accessing it may cause
a kernel crash.

To fix this, we maintain a source MAC cache per port for the net device in
struct mlx4_ib_roce.  This cached MAC is initialized to be the default MAC
address obtained during HCA initialization via QUERY_PORT. This cached MAC
is updated via the netdev event notifier handler.

Since the cached MAC is held in an atomic64 object, we do not need locking
when accessing it.
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

When marsheling a user path to the kernel struct ib_sa_path, need
to zero smac, dmac and set the vlan id to the "no vlan" value.

Fixes: dd5f03be ("IB/core: Ethernet L2 attributes in verbs/cm structures")
Reported-by: NAleksey Senin <alekseys@mellanox.com>
Signed-off-by: NMatan Barak <matanb@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

When device is being removed (e.g during VPI port link type change
from ETH to IB), tasks for gid table changes should not be executed.

Flush the current queue of tasks and block further tasks from entering the queue.
Signed-off-by: NMoni Shoua <monis@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Chuck Lever reported the following stack trace:

    =================================
    [ INFO: inconsistent lock state ]
    3.16.0-rc2-00024-g2e78883 #17 Tainted: G            E
    ---------------------------------
    inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
    swapper/0/0 [HC0[0]:SC1[1]:HE1:SE0] takes:
    (&(&iboe->lock)->rlock){+.?...}, at: [<ffffffffa065f68b>] mlx4_ib_addr_event+0xdb/0x1a0 [mlx4_ib]
    {SOFTIRQ-ON-W} state was registered at:
     [<ffffffff810b3110>] mark_irqflags+0x110/0x170
     [<ffffffff810b4806>] __lock_acquire+0x2c6/0x5b0
     [<ffffffff810b4bd9>] lock_acquire+0xe9/0x120
     [<ffffffff815f7f6e>] _raw_spin_lock+0x3e/0x80
     [<ffffffffa0661084>] mlx4_ib_scan_netdevs+0x34/0x260 [mlx4_ib]
     [<ffffffffa06612db>] mlx4_ib_netdev_event+0x2b/0x40 [mlx4_ib]
     [<ffffffff81522219>] register_netdevice_notifier+0x99/0x1e0
     [<ffffffffa06626e3>] mlx4_ib_add+0x743/0xbc0 [mlx4_ib]
     [<ffffffffa05ec168>] mlx4_add_device+0x48/0xa0 [mlx4_core]
     [<ffffffffa05ec2c3>] mlx4_register_interface+0x73/0xb0 [mlx4_core]
     [<ffffffffa05c505e>] cm_req_handler+0x13e/0x460 [ib_cm]
     [<ffffffff810002e2>] do_one_initcall+0x112/0x1c0
     [<ffffffff810e8264>] do_init_module+0x34/0x190
     [<ffffffff810ea62f>] load_module+0x5cf/0x740
     [<ffffffff810ea939>] SyS_init_module+0x99/0xd0
     [<ffffffff815f8fd2>] system_call_fastpath+0x16/0x1b
    irq event stamp: 336142
    hardirqs last  enabled at (336142): [<ffffffff810612f5>] __local_bh_enable_ip+0xb5/0xc0
    hardirqs last disabled at (336141): [<ffffffff81061296>] __local_bh_enable_ip+0x56/0xc0
    softirqs last  enabled at (336004): [<ffffffff8106123a>] _local_bh_enable+0x4a/0x50
    softirqs last disabled at (336005): [<ffffffff810617a4>] irq_exit+0x44/0xd0

    other info that might help us debug this:
    Possible unsafe locking scenario:

          CPU0
          ----
     lock(&(&iboe->lock)->rlock);
     <Interrupt>
       lock(&(&iboe->lock)->rlock);

    *** DEADLOCK ***

The above problem was caused by the spin lock being taken both in the process
context and in a soft-irq context (in a netdev notifier handler).

The required fix is to use spin_lock/unlock_bh() instead of spin_lock/unlock
on the iboe lock.
Reported-by: NChuck Lever <chuck.lever@oracle.com>
Signed-off-by: NJack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

When a RoCE port becomes active and the netdev of the port has upper
device (e.g bond/team), GIDs derived from the upper dev should appear
in the port's RoCE GID table.
Signed-off-by: NMoni Shoua <monis@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

There's no need to reset the gid table twice and we need to do it only
for Ethernet ports. Also, no need to actively scan ndetdevs since it's
being done immediatly after we register netdev notifiers.
Signed-off-by: NMoni Shoua <monis@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

When reading the IPv6 addresses from the net-device, make sure to
avoid adding a duplicate entry to the GID table because of equality
between the default GID we generate and the default IPv6 link-local
address of the device.

Fixes: acc4fccf ("IB/mlx4: Make sure GID index 0 is always occupied")
Signed-off-by: NMoni Shoua <monis@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

When Ethernet netdev is not present for a port (e.g. when the link
layer type of the port is InfiniBand) it's possible to dereference a
null pointer when we do netdevice scanning.

To fix that, we move a section of code that needs to run only when
netdev is present to a proper if () statement.

Fixes: ad4885d2 ("IB/mlx4: Build the port IBoE GID table properly under bonding")
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NMoni Shoua <monis@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

We need to fail the bind operation if the iser connection state != UP
(started teardown) and this should be done under the state lock.
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

When failing to allocate TX CQ we already allocated RX CQ, so we need to make
sure we release it. Also, when failing to register notification to the RX CQ
we currently leak both RX and TX CQs of the current index, fix that too.
Signed-off-by: NRoi Dayan <roid@mellanox.com>
Signed-off-by: NSagi Grimberg <sagig@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

ocrdma_query_ah() does not use correct macro, and checks the wrong bit
for the validity of address handle in vector table.  Fix this.
Signed-off-by: NDevesh Sharma <devesh.sharma@emulex.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

Because of IP-based GIDs, userspace AHs must have MAC and VLAN ID
resolved separately.  Presently, user AHs are broken for ocrdma.  This
patch resolves L2 addresses while creating user AH and obtains the
right DMAC and VLAN ID before creating AH.
Signed-off-by: NDevesh Sharma <devesh.sharma@emulex.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

This patch addresses feedback from Sagi Grimberg on the rereg_mr
implementation of mlx4.  The following are fixed:

1. Set the correct pd_flags
2. Make sure we change the iova and size MR fields only after
   successful write and allocation of the MTTs.
3. Make the error checking more robust

Fixes: e630664c ("mlx4_core: Add helper functions to support MR re-registration")
Signed-off-by: NMatan Barak <matanb@mellanox.com>
Signed-off-by: NOr Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

This particular reference count is not needed with the rcu protection,
and the current code leaks a reference count, causing a hang in
qib_qp_destroy().

Cc: <stable@vger.kernel.org>
Reviewed-by: NDennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: NMike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>

There are two queries for port attributes one after another. A second
call is not needed since port_attr structure already holds the data.
Reviewed-by: NIra Weiny <ira.weiny@intel.com>
Signed-off-by: NAlex Estrin <alex.estrin@intel.com>
Signed-off-by: NRoland Dreier <roland@purestorage.com>