Checkpatch throws a warning when the argument identifier names are not
included in the function definitions.

To avoid these warnings, argument identifiers are added in the existing
function definitions.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The number of interrupts requested for user contexts are stored in the context
specific structures and utilized to manage the interrupts. For the master
contexts, this number is only used once and therefore not saved.

To prepare for future commits where the number of interrupts will be required
in more than one place, preserve the value in the master context structure.

[mkp: typo in comment]
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

As staging to support future accelerator transports, add a shim layer
such that the underlying services the cxlflash driver requires can be
conditional upon the accelerator infrastructure.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The CXL-specific work structure used to request the number of interrupts
currently resides as a nested member of both the context information and
hardware queue structures. It is used to cache values (specifically the
number of interrupts) required by the CXL layer when starting a context.

To facilitate staging that will ultimately allow the cxlflash core to
become agnostic of the underlying accelerator transport, remove these
embedded work structures.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Convert cxl-specific pointers to generic cookies to facilitate future
enhancements.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The SCSI core now zeroes the per-command private data area prior to
calling into the LLD. Replace the clearing operation that takes place
when the private command data reference is obtained with a routine that
performs common initializations. The zeroing that takes place in the
device reset path remains intact as the private command data associated
with the specified SCSI command is not guaranteed to be cleared.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The cxlflash driver supports performing a write-same16 to scrub virtual
luns when they are released by a user. To date, AFUs for adapters that
are supported by cxlflash do not have the capability to unmap as part of
the WS operation. This can lead to fragmented flash devices which results
in performance degradation.

Future AFUs can optionally support unmap write-same commands and reflects
this support via the context control register. This provides userspace
applications with direct visibility such that they need not depend on a
host API.

Detect unmap support during cxlflash initialization by reading the context
control register associated with the primary hardware queue. Update the
existing write_same16() routine to set the unmap bit in the CDB when unmap
is supported by the host.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Adopt the SISLite AFU debug capability to allow future CXL Flash
adapters the ability to better debug AFU issues. Update the SISLite
header with the changes necessary to support AFU debug operations
and create a host ioctl interface for user debug software. Also
update the cxlflash documentation to describe this new host ioctl.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Adopt the SISLite AFU LUN provisioning capability to allow future CXL
Flash adapters the ability to better manage storage. Update the SISLite
header with the changes necessary to support LUN provision operations
and create a host ioctl interface for user LUN management software. Also
update the cxlflash documentation to describe this new host ioctl.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The existing AFU capability checking infrastructure is closely tied to
the command mode capability bits. In order to support new capabilities,
refactor the existing infrastructure to be more generic.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The cxlflash driver currently lacks host management interface. Future
devices supported by cxlflash will provide a variety of host-wide
management functions. Examples include LUN provisioning, hardware debug
support, and firmware download.

In order to provide a way to manage the device, a character device will
be created during probe of each adapter. This device will support a set of
ioctls defined in the SISLite specification from which administrators can
manage the adapter.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

When the AFU is reset in an error path, pending scsi commands can be
silently dropped without completion or a formal abort. This puts the onus
on the cxlflash driver to notify mid-layer and indicating that the command
can be retried.

Once the card has been quiesced, the hardware send queue lock is acquired
to prevent any data movement while the pending commands are processed.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Currently, there is no book keeping of the pending scsi commands in the
cxlflash driver. This lack of tracking in-flight requests is too
restrictive and requires a heavy-hammer reset each time an adapter error is
encountered. Additionally, it does not allow for commands to be properly
retried.

In order to avoid this problem and to better handle error path command
cleanup, introduce a linked list for each hardware queue that tracks
pending commands.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

A context reset failure indicates the AFU is in a bad state. At present,
when such a situation occurs, no further action is taken. This leaves the
adapter in an unusable state with no recoverable actions.

To avoid this situation, context reset failures will be escalated to a host
reset operation. This will be done asynchronously to allow the acting
thread to return to the user with a failure.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Per the SISLite specification, context_reset() writes 0x1 to the LSB of the
reset register. When the AFU processes this reset request, it is expected
to clear the bit after reset is complete. The current implementation simply
checks that the entire value read back is not 1, instead of masking off the
LSB and evaluating it for a change to 0. Should the AFU manipulate other
bits during the reset (reading back a value of 0xF for example), successful
completion will be prematurely indicated given the existing logic.

Additionally, in the event that the context reset operation fails, there
does not currently exist a way to provide feedback to the initiator of the
reset. This poses a problem for the rare case that a context reset fails as
the caller will proceed on the assumption that all is well.

To remedy these issues, refactor the context reset routine to only mask off
the LSB when evaluating for success and return status to the caller. Also
update the context reset handler parameters to pass a hardware queue
reference instead of a single command to better reflect that the entire
queue associated with the context is impacted by the reset.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Currently there are separate spin locks for the two supported I/O queueing
models. This makes it difficult to serialize with paths outside the enqueue
path.

As a design simplification and to support serialization with enqueue
operations, move to only a single lock that is used for enqueueing
regardless of the queueing model.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

As an enhancement to distribute requests to multiple hardware queues, add the
infrastructure to hash a SCSI command into a particular hardware queue.
Support the following scenarios when deriving which queue to use: single
queue, tagging when SCSI-MQ enabled, and simple hash via CPU ID when SCSI-MQ
is disabled. Rather than altering the existing send API, the derived hardware
queue is stored in the AFU command where it can be used for sending a command
to the chosen hardware queue.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

As staging for supporting multiple hardware queues, add an attribute to show
and set the current number of hardware queues for the host. Support specifying
a hard limit or a CPU affinitized value. This will allow the number of
hardware queues to be tuned by a system administrator.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Introduce multiple hardware queues to improve legacy I/O path performance.
Each hardware queue is comprised of a master context and associated I/O
resources. The hardware queues are initially implemented as a static array
embedded in the AFU. This will be transitioned to a dynamic allocation in a
later series to improve the memory footprint of the driver.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

As a general cleanup, address all reasonable checkpatch warnings and
errors. These include enforcement of comment styles and including named
identifiers in function prototypes.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Validation statements to enforce assumptions about specific defines are not
being evaluated by the compiler due to the fact that they reside in a routine
that is not used. To activate them, call the routine as part of module
initialization. As an additional, related cleanup, remove the now-defunct
CXLFLASH_NUM_CMDS.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

An EEH during probe can lead to a crash as the recovery thread races with the
probe thread. To avoid this issue, introduce new states to fence out EEH
recovery until probe has completed. Also ensure the reset wait queue is
flushed during device removal to avoid orphaned threads.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Update the SISlite header to support 4 ports as outlined in the SISlite
specification. Address fallout from structure renames and refreshed
organization throughout the driver. Determine the number of ports supported by
a card from the global port selection mask register reset value.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

As staging to support FC-related updates to the SISlite specification,
introduce helper routines to obtain references to FC resources that exist
within the global map. This will allow changes to the underlying global map
structure without impacting existing code paths.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

At present, the cxlflash driver only supports hardware with two FC ports. The
code was initially designed with this assumption and is dependent on having
two FC ports - adding more ports will break logic within the driver.

To mitigate this issue, remove the existing port assumptions and transition
the code to support more than two ports. As a side effect, clarify the
interpretation of the DK_CXLFLASH_ALL_PORTS_ACTIVE flag.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Transition from a static number of FC ports to a value that is derived during
probe. For now, a static value is used but this will later be based on the
type of card being configured.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Currently, RRQ processing takes place on hardware interrupt context. This can
be a heavy burden in some environments due to the overhead encountered while
completing RRQ entries. In an effort to improve system performance, use the
IRQ polling API to schedule this processing on softirq context.

This function will be disabled by default until starting values can be
established for the hardware supported by this driver.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

As further staging to support processing the HRRQ by other means, access to
the HRRQ needs to be serialized by a disabled lock. This will allow safe
access in other non-hardware interrupt contexts. In an effort to minimize the
period where interrupts are disabled, support is added to queue up commands
harvested from the RRQ such that they can be processed with hardware
interrupts enabled. While this doesn't offer any improvement with processing
on a hardware interrupt it will help when IRQ polling is supported and the
command completions can execute on softirq context.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

When processing an AFU asynchronous interrupt, if the action results in an
operation that requires off level processing (a link reset for example),
the worker thread is scheduled. In the meantime a reset event (i.e.: EEH)
could unmap the AFU to recover. This results in an Oops when the worker
thread tries to access the AFU mapping.

[c000000f17e03b90] d000000007cd5978 cxlflash_worker_thread+0x268/0x550
[c000000f17e03c40] c00000000011883c process_one_work+0x1dc/0x680
[c000000f17e03ce0] c000000000118e80 worker_thread+0x1a0/0x520
[c000000f17e03d80] c000000000126174 kthread+0xf4/0x100
[c000000f17e03e30] c00000000000a47c ret_from_kernel_thread+0x5c/0xe0

In an effort to avoid this, a mapcount was introduced in
commit b45cdbaf ("cxlflash: Resolve oops in wait_port_offline")
but due to the race condition described above, this solution is incomplete.

In order to fully resolve this problem and to simplify things, this commit
removes the mapcount solution. Instead, the scheduled worker thread is
cancelled after interrupts have been disabled and prior to the mapping
being freed.

Fixes: b45cdbaf ("cxlflash: Resolve oops in wait_port_offline")
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The SISLite specification outlines a new queuing model to improve
over the MMIO-based IOARRIN model that exists today. This new model
uses a submission queue that exists in host memory and is shared with
the device. Each entry in the queue is an IOARCB that describes a
transfer request. When requests are submitted, IOARCBs ('current'
position tracked in host software) are populated and the submission
queue tail pointer is then updated via MMIO to make the device aware
of the requests.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Currently, when sending a SCSI command, the pointer is stored in a
reserved field of the AFU command descriptor for retrieval once the
SCSI command has completed. In order to support new descriptor formats
that make use of the reserved field, the pointer is migrated to outside
the descriptor where it can still be found during completion processing.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

As staging for supporting hardware with a different queuing mechanism,
move the send_cmd() and context_reset() routines to function pointers
that are configured when the AFU is initialized. In addition, rename
the existing routines to better reflect the queue model they support.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The original design of the cxlflash driver required AFU commands
to convey state information across multiple threads. The IOASA
"host use" byte was used to track if a command was done, errored,
or timed out. A per-command spin lock was used to serialize access
to this byte. As this is no longer required with the introduction
of completions and various refactoring over time, the spin lock,
state tracking, and associated code can be removed. To support the
simplification, the wait_resp() routine is refactored to return a
success or failure. Additionally, as the simplification to the
AFU internal command routine, explicit assignments of AFU command
fields to zero are removed as the memory is zeroed upon allocation.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

With the removal of the static private command pool, the ability to
'complete' outstanding commands was lost. While not an issue for the
commands originating outside the driver, internal AFU commands are
synchronous and therefore have a timeout associated with them. To
avoid a stale memory access, the tear down sequence needs to ensure
that there are not any active commands before proceeding. As these
internal AFU commands are rare events, the simplest way to accomplish
this is detecting the activity and waiting for it to timeout.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Clean up and remove the remaining private command pool infrastructure
that is no longer required.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Instead of using a private pool of AFU commands, use cmd_size to prime
the private pool of SCSI commands such that they are allocated with a
size large enough to contain an aligned AFU command. Use scsi_cmd_priv()
to derive the aligned/zeroed private command on queuecommand and TMF
paths. Remove cmd_checkout() as it is no longer required. The remaining
AFU private command infrastructure will be removed in a cleanup commit.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The cxlflash driver originally required a per-command 4K buffer that
hosted data passed to the AFU. When the routines that initiate AFU
and internal SCSI commands were refactored to use scsi_execute(), the
need for this buffer became obsolete. As it is no longer necessary,
the buffer is removed.
Signed-off-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

During test, a command room violation interrupt is occasionally seen
for the master context when the CXL flash devices are stressed.

After studying the code, there could be gaps in the way command room
value is being cached in cxlflash. When the cached command room is zero
the thread attempting to send becomes burdened with updating the cached
value with the actual value from the AFU. Today, this is handled with an
atomic set operation of the raw value read. Following the atomic update,
the thread proceeds to send.

This behavior is incorrect on two counts:

   - The update fails to take into account the current thread and its
     consumption of one of the hardware commands.

   - The update does not take into account other threads also atomically
     updating. Per design, a worker thread updates the cached value when a
     send thread times out. By not protecting the update with a lock, the
     cached value can be incorrectly clobbered.

To correct these issues, the update of the cached command room has been
simplified and also protected using a spin lock which is held until the
MMIO is complete. This ensures the command room is properly consumed by
the same thread. Update of cached value also takes into account the
current thread consuming a hardware command.
Signed-off-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

To read the adapter VPD, drivers can't rely on pci config APIs, as it
wouldn't work on powerVM. cxl introduced a new kernel API especially
for this, so start using it.
Co-authored-by: NChristophe Lombard <clombard@linux.vnet.ibm.com>
Signed-off-by: NFrederic Barrat <fbarrat@linux.vnet.ibm.com>
Signed-off-by: NChristophe Lombard <clombard@linux.vnet.ibm.com>
Reviewed-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Acked-by: NManoj N. Kumar <manoj@linux.vnet.ibm.com>
Signed-off-by: NMichael Ellerman <mpe@ellerman.id.au>

With the current value of cmd_per_lun at 16, the throughput
over a single adapter is limited to around 150kIOPS.

Increase the value of cmd_per_lun to 256 to improve
throughput. With this change a single adapter is able to
attain close to the maximum throughput (380kIOPS).
Also change the number of RRQ entries that can be queued.
Signed-off-by: NManoj N. Kumar <manoj@linux.vnet.ibm.com>
Acked-by: NMatthew R. Ochs <mrochs@linux.vnet.ibm.com>
Reviewed-by: NUma Krishnan <ukrishn@linux.vnet.ibm.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>